How to generate ideas in Machine Learning?
Little story about https://www.infornopolitan.xyz
A few months ago, I faced with a problem, my Machine Learning model simply did not work. Trying to understand how I can extend my architecture, various names of other methods and their components, surfaced in my memory. At some moment I realized that my knowledge is very limited, and ideas are poor. I know a couple of dozen models and this is a very small part of the total amount that can be very useful to me.
The first thought was: what if I will know and understand at least all the papers from the last NeurIPS conference? It seemed to me that if I will understand more models, my qualities as a researcher would certainly increase. This idea strongly motivated me to start meticulously studying articles from the conference.
Google Sheets - create and edit spreadsheets online, for free.
Create a new spreadsheet and edit with others at the same time -- from your computer, phone or tablet. Get stuff done…
It is rather difficult to structure such type of information, and it is necessary to write down dependencies and connections between the methods. I had no wish to represent dependencies as a table or a list, I wanted something more natural. A little thought, I realized that to have a strictly fixed graph, with edges between models and their components is interesting.
For example GAN consist of Generator(GEN), Discriminator (DIS), Adversarial Autoencoder (AAE) based on Autoencoder (AE) and DIS, . Every component is a separated node in the graph, so for AAE we will have an edge to AE and DIS.
Step by step, I analyzed papers, writing out what methods they consist of, in what subject area they are applied, on what data they were tested, and tried to present it in the graph form. For people like me, who have not been at conferences, it’s incredibly difficult to sort out ~1000 articles. After a couple of weeks of working, I realized how fragmented the Machine Learning community is. There are a lot of models being developed, and often research papers on a new conference do not inherit most of the improvements proposed a year ago…
Naturally, I did research and tried to find analogs to what I was doing. There are many amazing services that allow you to monitor the emergence of new articles in the field. But all these methods were aimed primarily at simplifying the acquisition of knowledge, but not for help to create new ideas. Creativity is more important than the experience, and tools that can help you to be more creative should be an integral part of the research process.
We have tools that make it easy to conduct experiments, run and evaluate models, but we do not have methods that allow us to generate and evaluate ideas quickly. In Machine learning, and Artificial Intelligence research as a whole, fresh and elegant ideas play a key role.
Other fields of science cannot boast of tools for generating ideas too, but I am absolutely sure that Artificial Intelligence and other areas that solve difficult problems, needs an extraordinary approach for generating ideas. After thinking about this, the graph, which was previously developed as something individual, become a part of the big idea of developing tools for Machine Intelligence research in terms of ideas generation.
The way I analyzed these articles is very far from ideal. In just a few months, I only took out about ~250 articles from the last NeurIPS conference and ~250 other papers on which new are based. Most of the areas were completely unfamiliar to me, to understand them took several days. Sometimes I could not find the right description for some methods, and the abstract of the article was used for this.
Based on this, the second logical step was to create an opportunity for authors to add their own methods and make changes to existing ones in the graph. After all, no one except the authors of the article knows how to disassemble and describe their method in the best way.
In the future, with community support, we can level up BACKRONYM and built the system which will recommend us how to extend research. For this scenario, we plan to give to users the ability to visualize their own knowledge, publicly or privately, irrespective of the main graph. For example, user can build a graph of all methods which he knows or even graph of one model components, and the system will recommend him the most useful paper or method.
This how we do now, there are 10 columns in table:
1) Paper title
2) Link to paper
3) Names of authors
4) Release date
5) Place of publication
6) Method name
7) Subject area (Using For)
8) Acronym of method name
9) A brief description of the method
10) The list of methods on which this method is based (list of acronyms which are available in table).
Each article can be one single method, for example, Autoencoder (AE), or separately, Autoencoder (AE) -> Encoder (ENCDR), Decoder (DCDR), That is, one model can consist of several elements that can be used separately.
Now the simples way to use the graph for idea improvement it skips connections. For example, you use CNN in your model, and in a graph, you can see another method that inherited from CNN. You can make little research and try just to replace your CNN with the advanced version.
Nevertheless, I believe that the main thing that the graph can give is associations. I was excited to know how GAN idea occurred to Ian Goodfellow. Talking on Artificial Intelligence podcast by Lex Fridman, Ian said that it was motivated by Boltzmann machine “positive” and “negative” training phases. This is a great example when a model inherits the properties of another not directly but very abstractly.
I would very much like to see this type of connections in this graph, and it depends solely on authors of papers, will or not they share what was motivated them to create such type of model to solve some problem. I think that the story not only about what methods were used specifically in their architecture but also ideologically on which concepts and knowledge the solution was formed, can allow us to be much more resourceful.
It seems to me that even today this graph take a place to be because it probably allows someone to get associations that may help to create some new method or extend current. Today, it’s just a graph, but tomorrow we will have the tools to objectively get machine recommendations and automatically evaluate the impact the proposed idea on the way to General Artificial Intelligence.
This project is community-driven. We want to make it better and motivate more peoples to add their models to the graph. More accurate information about methods and better visualization technologies can really make it a very useful tool.
As I said great help for this project improvement will be if you will add some methods to the graph: https://www.infornopolitan.xyz/add-research
Additional to 3D graph we have a discussion forum on our website, to discuss all ideas that were created by BACKRONYM, and related to Machine Intelligence https://www.infornopolitan.xyz/discussion
Also, many projects are under development now and your contribution will increase the quality of these products as well. https://www.infornopolitan.xyz/backronym
 Generative Adversarial Nets https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf
 Adversarial Autoencoders https://arxiv.org/abs/1511.05644
 Autoencoder https://www.aaai.org/Papers/AAAI/1987/AAAI87-050.pdf
 Convolutional Neural Network http://yann.le.../lecun-99.pdf
 Ian Goodfellow: Artificial Intelligence podcast at MIT https://www.youtube.com/watch?v=Z6rxFNMGdn0