How we in Russia created the first teacher’s AI-assistant able to check school essays better that a human can
Two Russian teams became winners of the unprecedented technological contest held by the National Technological Initiative (NTI) Up Great PRO//READING, devoted to creating a teacher’s AI-assistant. It was the first time when artificial intelligence successfully managed to check the school examination essays written in English to identify different types of errors at the same level, or even a bit better, than a human teacher usually does. The Russian teams managed to successfully address the technological challenge, which will make it possible in the future to create a service capable of doing most of technical procedures on checking essays, freeing teachers from routine work. Now the developers are expected to do the same task for Russian language essays. The contest in this category is still in progress — but the winners for the categories “Structure” and “Logics” have already been selected. The winners of the English language contest and the Russian contest satellites shared the award of 130 million rubles (over $ 1.7 million), while the total amount of prize for the contest is 255 million rubles ($ 3.4 million).
More about NTI’s technological contests
NTI Up Great technological contests started in 2018 as part of the National Technological Initiative. The participants are expected to find solutions to address the most complex technological challenges which have not been yet successfully addressed anywhere in the world. The winner able to overcome the technological barrier will receive a large money prize for creating a comprehensive and reproducible solution. The organizational pattern of the contest was made in accordance with the best practices of global technological contests: XPrize, Darpa Grand Challenge etc. Up Great technological contests are operated by NTI Foundation. Co-initiators of the Up Great PRO//READING are the Skolkovo Foundation and NTI Platform. The technical partner of the contest is NTI’s Artificial Intelligence Competence Center functioning on the basis of Moscow Institute of Physics and Technology.
PRO//READING has been the first NTI Up Great technological contest where a winner was selected. The contest started in 2019 with the goal of creating an AI-assistant which would be able to help teachers find logical, factual, stylistic and other semantic errors in the school essays in Russian, literature, history, social studies and English. The aim of this contest is to promote the development of powerful AI technologies and to support the developers community in the field of machine learning. The expected result was, among others, to create an automated system for checking essays by USE experts to enable error detection at a level at least comparable to a human teacher. Technological contests for the Russian and the English part were held separately, the amount of prize is 100 million rubles for each contest ($1.3 million) It took two test cycles to select a winner for the English category. Services offered by the contest participants did a check for about 2,000 essays.
Winning teams English
The money prize of the contest was divided between two Russian teams: the first place was given to the developers from DeepPavlov — 80 million rubles (over 1 million dollars), Nanosemantics took the second place — 20 million rubles (about 267,000 dollars); The winners managed to create AI system able to address the challenges of this technological contest. The efficiency of DeepPavlov’s solutions was 107% compared to the results of the check done by a human teacher. The efficiency of Nanosemantics’ solution was 105%. AI “competed” with experienced teachers — USE experts who checked the essay manually, and the results were compared under the guidance of the competition and expert commission.
DeepPavlov’s team is known worldwide for their solutions in the field of machine-based text analysis. Team members: Denis Kuznetsov (head), Dmitry Karpov, Alexey Sorokin, Anastasia Kravtsova with the support of the scientific director of the team Mikhail Burtsev.
Nanosemantics is working on the development of speech technologies based on artificial intelligence. The team members are: Mikhail Volochich, Sergey Drozdov, Pavel Shestakov, Alexander Muravyov, Vladimir Dneprovsky, Daria Arakelova, Alexey Loginov, Alexander Kislinsky, Pavel Sukhachev.
“In the future, it may be a cloud service, with the access for teachers either via a special client, or via a web-page. For sure, it will need some fine-tuning to meet the application-specific requirements, and it will need integration with user systems. This means that from the stage of contest testing to the stage of finished product, there will be a certain way to go, but we hope that our service will really help teachers speed up checking essays, reduce the amount of routine, and will also be useful for analytical purposes,” says Denis Kuznetsov, head of DeepPavlov, the winning team.
“When the government grants money not for R&D, but for the result, this drastically changes the situation with overcoming the technological barriers. And, of course, I am very glad that winning teams have grown up in the ecosystem of the National Technology Initiative,” says Dmitry Peskov, Special Representative of the President of the Russian Federation for Digital and Technological Development, CEO of the NTI Platform.
Russian language
As for the main task of the contest, or, to be more precise, creating a solution in Russian (this also includes other humanities disciplines — literature, history and social studies), the barrier has not been overcome and the participants are waiting for the next round. The contest will last until the winner is selected, or will be finished in October 2022 without selecting a final winner. However, the winners have been selected for special categories (so-called ‘satellites’): Structure and Logic. The contest categories are intended to further stimulate participants to develop effective solutions that partially solve the global task of the main contest in the Russian language. In this case, the prize could be granted to the teams developing a solution that allows identifying logical errors in the text, as well as identifying semantic errors in compositions or essays and finding a connection between the blocks. According to the contest rules the same team cannot pick up two cash prizes for the categories, so the awards were distributed as follows: in the “Structure” category the prize was given to Nanosemantics — 12 million rubles ($ 160 thousand), while the team of the Russian Chemical Technology University (RCTU) named after D.I. Mendeleev received 3 million rubles ($ 40 thousand); in the “Logic” category, the prize was received by Antiplagiat team (12 million rubles) and First Try (3 million rubles);
The total amount of prizes for satellite contests for the two languages is over $ 700 thousand.
“Whether the Deep Learning system can work with semantics as effectively as a human can is a challenge that Russian developers have tried to address,” says Kirill Kaem, Senior Vice President for Innovation at the Skolkovo Foundation.
“It is great that in the English language category the participants have overcome this barrier. But the Russian language is not so simple. I think that the semantic structure of the Russian language, perhaps, will not be hacked by the Deep Learning system for some time yet,” he believes.
Previously, the NTI Up Great technological contests “Winter City” (competitions for unmanned vehicles used in the conditions of the Russian winter) and “First Element” (creation of power plants on hydrogen fuel cells for small aircraft) have already been held in Russia. However, no winners were selected who had been able to 100% solve the tasks offered in these contests. Breakthrough solutions were offered only for certain categories, which means that the technological barriers were overcome only partially.
In addition, such contests as AI’M DOCTOR (developing an AI assistant to a doctor) and Aerologistics (creation of an unmanned aircraft for cargo transportation) have been announced. For more detailed information about the NTI Up Great contests see our website https://upgreat.one/
