AI Research Rankings 2019: Insights from NeurIPS and ICML, Leading AI Conferences

By Gleb Chuvpilo

To glean a country’s, a region’s or an institution’s contribution to an article, and to ensure they are not counted more than once, the Nature Index uses fractional count (FC), which takes into account the share of authorship on each article. The total FC available per article is 1, which is shared among all authors under the assumption that each contributed equally. For instance, an article with 10 authors means that each author receives an FC of 0.1. For authors who are affiliated with more than one institution, the author’s FC is then split equally between each institution. The total FC for an institution is calculated by summing the FC for individual affiliated authors. The process is similar for countries/regions, although complicated by the fact that some institutions have overseas labs that will be counted towards host country/region totals.

The only difference is that our Publication Index counts overseas labs towards the headquarters country/region (instead of the host country/region). This is a contentious point, but we believe that this approach better reflects the assignment of intellectual property and respective accrual of benefit to the headquarters, rather than the local lab.

Here is an example of the Publication Index calculation. If a paper has five authors — three from MIT, one from the University of Oxford, and one from Google — each author will get 1/5th of one point, or 0.2. As a result, from this paper alone, MIT will increase its Publication Index by 3*0.2=0.6 points, the University of Oxford will increase its index by 0.2, and Google will add 0.2. Since MIT is based in the United States, MIT affiliation will increase the Publication Index of the United States by 0.6. Similarly, since the University of Oxford is based in the UK, the EEA + Switzerland category will increase by 0.2. Finally, Google is a multinational corporation headquartered in the United States, therefore the United States will increase its Publication Index by an additional 0.2, for the total increase of 0.8. If an author has multiple affiliations, we split his/her fraction across each of those affiliated institutions. For instance, in the case above, if the last author listed two affiliations, Google and Stanford University (instead of just Google), both Google and Stanford University would get additional 0.2/2=0.1 points.

Finally, the reason why we thought it was fair to combine NeurIPS and ICML publications into the same dataset is that they have similar perceived prestige among top AI researchers, similar institutional participation, and similar paper acceptance rates (21.2% for NeurIPS and 22.6% for ICML).

AI Research Rankings 2019

Top 40 Global Organizations (Industry & Academia) Leading in AI Research in 2019 (with Publication Indices):

AI Research Rankings 2019—Top 40 Global Organizations (Industry & Academia) Leading in AI Research in 2019

1. Google (USA) — 167.32. Stanford University (USA) — 82.33. MIT (USA) — 69.84. Carnegie Mellon University (USA) — 67.75. UC Berkeley (USA) — 54.06. Microsoft (USA) — 51.97. University of Oxford (UK) — 37.78. Facebook (USA) — 33.19. Princeton University (USA) — 31.510. Cornell University (USA) — 30.911. Georgia Tech (USA) — 30.112. UT Austin (USA) — 29.913. University of Illinois (USA) — 29.414. Columbia University (USA) — 29.215. Tsinghua University (China) — 28.416. UCLA (USA) — 27.217. ETH (Switzerland) — 27.018. IBM (USA) — 25.819. University of Washington (USA) — 24.020. INRIA (France) — 23.221. EPFL (Switzerland) — 22.322. Peking University (China) — 21.623. University of Toronto (Canada) — 21.424. Harvard University (USA) — 19.225. Duke University (USA) — 18.726. New York University (USA) — 17.727. University of Cambridge (UK) — 15.128. KAIST (South Korea) — 14.829. Technion (Israel) — 14.630. UC San Diego (USA) — 14.631. University of Wisconsin Madison (USA) — 14.432. Amazon (USA) — 14.333. UMass Amherst (USA) — 13.834. University College London (UK) — 13.735. MILA (Canada) — 13.536. University of Southern California (USA) — 13.537. University of Pennsylvania (USA) — 13.338. Seoul National University (South Korea) — 12.739. Johns Hopkins University (USA) — 12.6

40. RIKEN (Japan) — 12.3

Top 20 Regions Leading in AI Research in 2019 (with Publication Indices):

AI Research Rankings 2019 — Top 20 Regions Leading in AI Research in 2019

1. United States — 1260.22. EEA* + Switzerland — 431.53. China — 184.54. Canada — 80.35. Japan — 49.46. South Korea — 46.87. Israel — 43.38. Australia — 27.09. India — 17.110. Singapore — 13.211. Russia — 10.612. Taiwan — 5.313. Saudi Arabia — 5.014. United Arab Emirates — 2.315. Iran — 2.216. South Africa — 1.017. Chile — 1.018. Malaysia — 0.719. Turkey — 0.6

20. New Zealand — 0.5

*Countries that belong to the EEA include Austria, Belgium, Bulgaria, Croatia, Republic of Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Sweden, UK, Iceland, Liechtenstein, and Norway (source).

Top 20 Countries Leading in AI Research in 2019 (with Publication Indices):

AI Research Rankings 2019 — Top 20 Countries Leading in AI Research in 2019

1. United States — 1260.22. China — 184.53. United Kingdom — 126.14. France — 94.35. Canada — 80.36. Germany — 64.57. Switzerland — 59.38. Japan — 49.49. South Korea — 46.810. Israel — 43.311. Australia — 27.012. India — 17.113. Netherlands — 15.314. Singapore — 13.215. Denmark — 12.216. Italy — 11.517. Sweden — 11.318. Russia — 10.619. Finland — 9.6

20. Austria — 7.4

Top 20 American Universities Leading in AI Research in 2019 (with Publication Indices):

AI Research Rankings 2019 — Top 20 American Universities Leading in AI Research in 2019

1. Stanford University — 82.32. MIT — 69.83. Carnegie Mellon University — 67.74. UC Berkeley — 54.05. Princeton University — 31.56. Cornell University — 30.97. Georgia Tech — 30.18. UT Austin — 29.99. University of Illinois — 29.410. Columbia University — 29.211. UCLA — 27.212. University of Washington — 2413. Harvard University — 19.214. Duke University — 18.715. New York University — 17.716. UC San Diego — 14.617. University of Wisconsin Madison — 14.418. UMass Amherst — 13.819. University of Southern California — 13.5

20. University of Pennsylvania — 13.3

Top 20 Global Universities Leading in AI Research in 2019 (with Publication Indices):

AI Research Rankings 2019 — Top 20 Global Universities Leading in AI Research in 2019

1. Stanford University (USA) — 82.32. MIT (USA) — 69.83. Carnegie Mellon University (USA) — 67.74. UC Berkeley (USA) — 54.05. University of Oxford (UK) — 37.76. Princeton University (USA) — 31.57. Cornell University (USA) — 30.98. Georgia Tech (USA) — 30.19. UT Austin (USA) — 29.910. University of Illinois (USA) — 29.411. Columbia University (USA) — 29.212. Tsinghua University (China) — 28.413. UCLA (USA) — 27.214. ETH (Switzerland) — 27.015. University of Washington (USA) — 24.016. INRIA (France) — 23.217. EPFL (Switzerland) — 22.318. Peking University (China) — 21.619. University of Toronto (Canada) — 21.4

20. Harvard University (USA) — 19.2

Top 20 Companies Leading in AI Research in 2019 (with Publication Indices):

AI Research Rankings 2019 — Top 20 Companies Leading in AI Research in 2019

1. Google (USA) — 167.32. Microsoft (USA) — 51.93. Facebook (USA) — 33.14. IBM (USA) — 25.85. Amazon (USA) — 14.36. Tencent (China) — 8.87. Alibaba (China) — 7.58. Bosch (Germany) — 7.29. Uber (USA) — 7.110. Intel (USA) — 6.911. Toyota (Japan) — 6.012. Yandex (Russia) — 5.813. Baidu (China) — 5.514. Nvidia (USA) — 5.215. Apple (USA) — 4.616. Salesforce (USA) — 4.217. PROWLER.io (UK) — 4.218. Criteo (France) — 3.919. Huawei (China) — 3.7

20. NEC (Japan) — 3.5

Further Analysis

Academia vs. Industry — Share of Total Publication Index:

AI Research Rankings 2019 — Academia vs. Industry

Academia — 77.8%
Industry — 22.2%

Top 150 Words in 2200 Paper Titles at NeurIPS 2019 and ICML 2019 (“word cloud”):

AI Research Rankings 2019 — Top 150 Words in 2200 Paper Titles at NeurIPS 2019 and ICML 2019

Top 30 Countries by Per Capita Publication Index (Publication Index divided by country population in millions):

AI Research Rankings 2019 — Top 30 Countries by Per Capita Publication Index

1. Switzerland — 6.972. Israel — 4.883. United States — 3.854. Singapore — 2.345. Canada — 2.176. Denmark — 2.117. United Kingdom — 1.908. Finland — 1.759. France — 1.4110. Sweden — 1.1111. Australia — 1.0812. South Korea — 0.9113. Netherlands — 0.8914. Austria — 0.8415. Germany — 0.7816. Latvia — 0.6717. Belgium — 0.4418. Estonia — 0.4419. Japan — 0.3920. Norway — 0.3221. Cyprus — 0.2822. United Arab Emirates — 0.2623. Taiwan — 0.2224. Ireland — 0.2125. Italy — 0.1926. Saudi Arabia — 0.1527. Greece — 0.1428. China — 0.1329. Czech Republic — 0.11

30. New Zealand — 0.11

Treemap of Top 40 Global Organizations Leading in AI Research (Area is Proportional to the Publication Index):

AI Research Rankings 2019 — Treemap of Top 40 Organizations Leading in AI Research

Collectively, the top 40 organizations contributed 55% of the total Publication Index, with a combined value of 1,212.3 out of 2,200 total.

Measuring Competition in AI Research (the Herfindahl Index):

The Herfindahl index (also known as Herfindahl–Hirschman Index) is a measure of the size of participants in relation to the industry and an indicator of the amount of competition among them.

Calculation:

The Herfindahl index calculation: si is the market share (percents are used as whole numbers, as in 75 instead of 0.75), and N is the number of participants.

Interpretation:

  • An H below 100 indicates a highly competitive industry.
  • An H below 1,500 indicates an unconcentrated industry.
  • An H between 1,500 to 2,500 indicates moderate concentration.
  • An H above 2,500 indicates high concentration.

For our dataset (using each organization’s share of the Total Publication Index): H=146.47, which indicates an unconcentrated industry. In other words, there are no signs of monopolization of AI research in 2019.

Discussion: Who’s Ahead in AI?

A heated debate is going on today on the state of the strategic race between the United States and China to dominate in AI. We tend to side with a more balanced perspective, but before we begin our analysis, a bit of history is in order:

  • Two major events happened in AI in 2016: in March, Google’s AlphaGo became the first computer program to beat a 9-dan Go professional, Lee Sedol, without handicaps; in October, President Obama’s administration released a strategy on future directions and considerations for AI called Preparing for the Future of Artificial Intelligence.
  • In China, these two events created a “Sputnik moment” which helped convince the Chinese government to prioritize and dramatically increase funding for artificial intelligence (see Kai-Fu Lee’s AI Superpowers).
  • In response, in July 2017 the Communist Party of China set 2030 as the deadline for an ambitious AI goal: it called for China to reach the top tier of AI economies by 2020, achieve major new breakthroughs by 2025, and become the global leader in AI by 2030. The strategy became known as the New Generation Artificial Intelligence Development Plan, and it has spurred many policies and billions of dollars of investment in research and development from ministries, provincial governments, and private companies.
  • Certain think tanks, such as CNAS, have argued that China’s AI strategy reflected the key principles from the Obama administration report — now it is China adopting them, instead of the United States.
  • This copying strategy isn’t new: to quote Peter Thiel’s Zero to One, “The Chinese have been straightforwardly copying everything that has worked in the developed world: 19th-century railroads, 20th-century air conditioning, and even entire cities. They might skip a few steps along the way — going straight to wireless without installing landlines, for instance — but they’re copying all the same.”
  • 2017 is precisely the year when we started tracking the state of AI research, so we established China’s baseline summarized in the following chart showing that the United States had an 11x lead in the total Publication Index over China:
Top 10 Countries Leading in AI in 2017: the USA had an 11x lead over China
  • In 2019 the United States has a 7x lead (USA — 1260.2, China — 184.5), so the gap is clearly narrowing. Furthermore, the analysis by the Allen Institute for Artificial Intelligence found that China has steadily increased its share of authorship of the top 10% most-cited papers: China’s share was at 26.5% in 2018, not far behind the United States at 29%.

One might say that it is not looking good for American competitiveness in AI in the next decade. However, we believe that the outcome will depend on the interplay of the advancement of three key ingredients of modern AI: algorithms, hardware, and training data, and it takes getting all three right in order to dominate the field.

We believe that the United States will have a strong lead in AI algorithms for years to come, grounded in several decades of track record in the advancement of computer science at world-class universities, such as MIT, Stanford, CMU and UC Berkeley. In addition, the openness of the likes of Google and Facebook to publishing internal research at the conferences created a thriving ecosystem (and a rotating door of sorts) for top AI researchers to move seamlessly between academia and industry (think Yann LeCun or Andrew Ng).

In addition, the United States is the home of Silicon Valley (in its original silicon-focused definition), which has been at the forefront of hardware innovation ever since the traitorous eight left Shockley Semiconductor Laboratory to found Fairchild Semiconductor in 1957. Deep Learning algorithms are extremely compute-hungry, right up there with mining for bitcoin that now consumes more power than Switzerland. We think that it will be extremely difficult for China to catch up to the United States in hardware over the next decade.

However, where the American advantage is questionable is training data, and it is by design. It is, in fact, part of a broader privacy vs. public good debate, where the United States tends to choose the former, and China—the latter. In China today AI scans faces from hundreds of millions of street cameras, reads billions of WeChat messages, and analyzes millions of health records—all following the data-as-a-public-good argument. This training data availability, combined with China’s 1.4B population, creates an enormous strategic advantage for China.

Hard-pressed to draw a conclusion, we still think that the first two factors (algorithms and hardware) will outweigh the last one (availability of data), and the United States will maintain its lead in AI for years to come.

Dataset

Please note that conferences don’t release publication data in a standard form, so our analysis ended up being quite manual (HTML parsing, Python transformations, a lot of manual name standardization, and a few unknown affiliations). If you find any bugs, please email us, and we’ll be happy to fix them. If you’d like to download and play with the dataset, it is posted here. Enjoy!