Artificial Intelligence (AI) is one of the hottest topics today. Literally, the recent progress is self-evident-say hello to GPT-3, and it will say hello. The medicine discovered by artificial intelligence is just around the corner. As policymakers tried to understand this year’s technology through centuries-old laws, the company hired more PhDs than ever before. For researchers and investors, exciting moments may not be so many for politicians and lawyers.
This year, Nathan Benaich and Ian Hogarth collaborated for the third time to prepare a report on the state of the AI business, which covers the latest research, industry, talent, and policy news. Last but not least, the author made a prediction for 2021.
In this article, I summarized the main themes and findings of the report, and then made my own views on the matter. .
Report summary
Research (slides 10-62)
The report starts from the technical side: only 15% of AI papers disclose the open source of their code, PyTorch has most of the “research market share”, and several university groups have reached the one billion parameter mark. In addition, it talks about the economic and environmental costs of large models. Even if the hardware is improving, the cost of deep learning is increasing exponentially. The current SOTA model requires millions of dollars in training costs, let alone adjustments.
In terms of applications, natural language processing (NLP) has attracted most attention this year. In addition to natural language processing, artificial intelligence is also driving the evolving paper boom in biology and medicine. In addition to these two areas, Graph Neural Networks (GNN) and Reinforcement Learning (RL) have also made breakthroughs this year.
Of course, COVID-19 has also left its mark on the AI community and is committed to almost all aspects of the disease.
Talent (slides 63-81)
While the number of papers continues to grow, so does the number of professors leaving academia to go to large technology companies, and universities are taking a hit. In order to fight back, the university will focus on specialized AI research institutes and funding programs.
The loss of international talents is even greater. This year, many scientists have emigrated from Asia to the United States for research, and most of them still stay in the United States after graduation. The United States’ reliance on foreign talent is blatant. 70% of AI researchers working in the U.S. have not received U.S. training. This translates into published results. Chinese researchers accounted for approximately 29% of NeurIPS oral reports (acceptance rate of 0.5%).
Despite the COVID, the demand for AI talent is still high, and the number of enrollment in AI courses has been growing.
Industry (Slides 82-129)
The biggest bright spot is AI-based drugs. We are close to the point where AI drugs enter the market. This goes hand in hand with the prosperity of biology/medical papers. However, one thing that is despised is regulations and procedures. The current approval method is neither for AI-discovered products, nor for AI-led products, nor for continuous improvement of the workflow.
The autonomous vehicle (AV) industry faces similar problems. Despite billions of dollars invested, the legislation for driverless cars lags far behind the self-driving cars themselves. Part of the funds will be used for internal hardware, especially customized LiDAR technology, another funds will be used for self-driving car stacks, and the rest will still be largely handmade.
At the same time, the demand for computing has inspired new computing platform providers and specialized AI hardware, such as Graphcore’s M2000, Nvidia’s DGX-A100 and Google’s TPUv4. At the same time, work on improved ML infrastructure and operations is also rapidly evolving.
Slides 113 to 129 are dedicated to industry success stories.
Politics (Slides 130–170)
This year is marked by AI’s ethical issues becoming mainstream, including but not limited to gender/racial prejudice, police and military use, facial recognition, surveillance, and counterfeiting. In particular, the military’s interest in AI technology is shocking, but not unexpected.
Conferences such as NeurIPS, ICLR, and Google have adopted new ethics, and some companies lean toward the ideals of fairness and privacy. However, there is still a long way to go to achieve real change. Chip production and IP ownership seem to be more of a concern for governments.
The political buzzword is AI nationalism: investing in the state-sovereignty issue of AI leaders and national AI policies.
Forecast (slide 172)
Benaich and Hogarth concluded the report with their 2021 forecast. They are as follows (slide 172):
1) The race to build a larger language model continues, and we saw the first 10 trillion parameter model.
2) Attention-based neural networks migrate from NLP to computer vision to achieve state-of-the-art results.
3) The AI laboratory of a large company was closed due to a change in strategy by its parent company.
4) In response to the activities of the US Department of Defense and investment in US military AI startups, in the next 12 months, a wave of AI startups focusing on Chinese and European defense raised a total of more than 100 million U.S. dollars.
5) A leading artificial intelligence drug discovery startup company (such as Recursion, Exscientia) conducts an initial public offering or is acquired at a price of more than $10B.
6) DeepMind has made major breakthroughs in structural biology and drug discovery outside of AlphaFold.
7) Facebook has made a major breakthrough in augmented reality and virtual reality through 3D computer vision.
8) NVIDIA did not complete the acquisition of Arm in the end.
Although these forecasts are for next year, some of them have become reality. Regarding (1), Microsoft announced that its DeepSpeed library already has a “trillion-parameter model” function. Although no version has been released so far, it is clear that there will be a $10 trillion model road. Regarding (2), the image value 16×16 is moving in this direction.
Regarding (6), in addition to AlphaFold, we also have…AlphaFold 2! Its latest version has the same or greater impact on biology as AlexNet’s impact on computer vision in 2012. The current media reports seem to be consistent. I firmly believe that the author will mark it as correct in 2021.
Content of report
Next, I added my opinions to the report results in the order of the slides, and made some correlations with recent events. Please keep in mind that the report was released in October. Since then, many things have happened.
Research (slides 10-62)
· Only 15% of AI papers have published their code (slide 11): I want to know the percentage of other computer science fields. In addition, not all codes are the same. In terms of code, a new architecture composed of pre-existing components is not as important as a new implementation. Novel losses or optimizer functions can be as short as embedded code snippets. All in all, I agree that AI is not as open as we think, but compared to other fields of computer science, it is still quite open.
· PyTorch will surpass TensorFlow in industry use (Slides 13, 14): Although I believe this is true, the data is misleading. Only 30% of the papers stated their framework. Many may still be constrained by TensorFlow. In addition, I found it strange that Keras data is not displayed (slide 14).
· AI competitions are very resource intensive (slides 16-24): Recently, Timnit Gebru was fired from Google because his draft paper outlined the monetary and ecological costs of training large language models. According to her paper, a Transformer with 0.2bi parameters trained on NAS will cost approximately US$1 million. GPT-3 has 175bi. Mathematics does not seem to be any good for the earth.
· This arms race will not take us anywhere (slides 16-24): I think the race for NLP breakthroughs will not bring real breakthroughs at all. GPT-3 is almost GPT-2 on steroids. Considering the Microsoft DeepSpeed mentioned above, we will continue to see models with swollen models in the media, and there will be no meaningful results in understanding through this effort.
· Can the university keep up, or can it? (Slide 22): No AI department can keep up with large-scale technologies. The university needs to play another game. Small model research may bring as much performance as possible at logarithmic cost. However, currently, the company is a leading researcher on efficient learning. For example, MobileNet/EfficientDet is Google’s and ShuffleNet is Face++.
· Transformer is very conspicuous (Slide 29): These models are based on the attention mechanism, which is a well-known power and resource consumption problem, because given a sequence of N elements, the mechanism is N². Effective attention is a hot topic, but no solution has been announced as a winner. Most of the AI costs mentioned above can be traced back to this mechanism.
· Biology is experiencing its “artificial intelligence moment” (slide 30): it does. With AlphaFold 2, we may see major breakthroughs in biology in this decade, just like we saw through AlexNet and Computer Vision in the 2010s.
· AI-based screening mammogram (slide 34): This is a controversial article. It claims to have superhuman performance, but lacks interpretability. So far, no code or data set has been released for third-party inspection and copying. This widely publicized article has aroused an enthusiastic response from researchers around the world, jointly expressing the “importance of transparency and repeatability in artificial intelligence research.” As a community, we must work hard to break AI into a stupid accuracy race Obstacles. How do doctors trust the black box algorithm?
Talent (slides 63-81)
· Brain drain (slide 64): Although this is about artificial intelligence and 2020, I can’t help but mention that this view is centered on the United States. All the universities mentioned are located in the United States. The brain is always in motion, especially from developing countries to rich countries. It just so happens that this time the university is a “poor country” and the company is a “rich country”.
· Leaving is related to a decline in entrepreneurship (slide 66): I think this connection is very poor. Companies in the market are full of talents and lack of talents. This is a bad environment for more companies. Slide 73 agrees, because most PhDs are foreigners, and foreigners are more likely to join large companies than start their own businesses.
· Contributions of Chinese-educated researchers on NeurIPS (slide 70): related to China’s plan to become an AI leader.
· Most of the top AI researchers working in the United States are not trained in the United States (slides 71-75): These slides point to a simple fact: the United States is highly dependent on foreign talent. Most students will get a PhD and stay at a technology company. Xenophobic laws are not good for the United States. but…
· Trump is not good for the United States (Slide 76): Trump tried to keep immigrants out, but found nothing, but it undoubtedly raised awareness of the United States’ dependence on foreign talents. Other countries seeking artificial intelligence advantages may Seize this opportunity to attract talent to their university.
Industry (Slides 82–129)
· AI priority drug discovery (slides 83-92): Obviously, the return on investment of drugs is faster than all investments in self-driving cars. In addition, better drugs and greater disease coverage may be more beneficial to humans than self-driving cars.
· Audiovisual companies are still in their infancy (slides 93-96): Legislation is still too early, and it is still far from the world. If the AV released today is perfect, it will be banned almost everywhere, or a driver will be required anyway.
· When even one billion dollars is not enough (slides 97-106): More will be invested, and this is still not enough. AV is a matter of time, not money. Artificial intelligence is not mature enough, and our laws are not prepared for it. Current visual research ignores that our world is continuous. We do not need to detect road signs from a single image. We need to better aggregate the results of multiple frameworks. With all due respect, the company is just dumping money on LiDAR and regulated dead ends.
· Computing progress (slides 107-111): Novel hardware is always good. However, I don’t know if the rest of the stack will remain. The problem with large data sets + large calculations is to prepare the next batch on time. The faster (larger) the calculation, the more difficult it is to obtain the training data at the required speed (+data expansion).
Politics (Slides 130–170)
· Moral hazard (slide 131): This requires special attention. The recent dismissal of Timnit Gebru highlights just how wrong the industry is in dealing with ethics. Asking large AI technologies to lead AI ethics research is like asking oil companies to lead the fight against global warming. Her dismissal shows that as long as the company does not harm its business model, they will play their part. This is not surprising. Given the close ties between American universities and corporate funds, it is difficult to expect them to participate.
· Face recognition is a major issue (slides 132-140): current laws are designed for humans. How to extend it to a system that can identify all individuals in the population? Are we eligible for an anonymous identity? To what extent? Should the company be blocked but allow law enforcement? In a sense, superheroes have similarities. How will our laws apply to Superman or The Flash? Can we really expect ordinary people’s laws to apply to superhuman abilities as well?
· What about voice and text? (Slides 132–140): Face recognition is related to our existence, but what about everything we say on the phone? Everything we say can be processed, monitored and misinterpreted. Passing the law on the face and ignoring other media will ignore the elephant in the room: everything we do is monitored.
· AI nationalism (slides 161-167): Most developed countries are awakened to AI and its threats. The hegemony of artificial intelligence can easily be transformed into military and economic domination and affect sovereignty. China is clearly already in a leading position because it has been working on AI leadership for a long time and is investing heavily in talent. As mentioned above, I think it is a matter of time for China to attract foreign talents to leave the United States. The same is true in India.
Artificial intelligence is the highest point ever. The result has never been better
上一篇: What role will the data center play in a world that supports 5G?
