Generative Modeling for Design

Generative Modeling Applications: Generative modeling is truly incredible, it’s the subfield of machine learning that seems the most intriguing to me because there’s a certain mystery to it. If we think about what machine learning can do, discriminative models are pretty cool. There are lots of useful applications for being able to detect or classify something. But generative models are a dark art. These models learn the distribution of a dataset in it’s entirety, then we aim to use them to create entirely new data points. These data points could be image, music, videos, text, anything! And these generative models are generally hard to train as well. Generative Adversarial Networks are known to have a loss function that looks like this:

That’s wild! Clearly it’s hard to ‘tame’ these networks to learn what we need them to properly. For the GAN specific case, there are tons of little tricks to make sure the cost function is diminished in a smooth fashion. I’ve made lots of videos on GANs, if you’re interested in how they work, here are afew interesting blog posts. But more to the point — once we’ve decided what we want to generate, we have to find the associated dataset. Do we want to generate faces?

Maybe fashion styles?

Perhaps even an entire university? Let me tell you an idea that I haven’t yet talked about publicly in-depth. I could someday automate my own job if you think about it! What if you could enter a topic you’d like to learn about, and have an app generate an entire detailed curriculum for you to learn that topic? First, it would build a knowledge graph of all related topics. It would then learn how to create an in-order sequential curriculum from that knowledge graph, judging each topic on it’s difficulty. Keyword recognition and semantic search help here. Once it’s learned the necessary representations, we could use those to seed a series of generative models. These models could generate text, images, quizzes, and yes, even instructor videos using it’s knowledge. Chatbots are themselves generative text models, we could use those as additional teaching aids for students that are 24/7 and hyper-personalized.

So, back to the topic of this course, how do we make money with this technology? Generative models are computationally expensive and need the computational power to be able to perform their necessary inference. Thus, it’s best to place them on a web server. Then, whether you have a mobile app or a web app, you’ll be able to send and retrieve data from the model. Unless (!) we’re using a mobile-optimized model, say using CoreML for iOS or Tensorflow Lite for Android. You can also use those directly from flutter. This repository has some good examples of deploying popular text models like GPT-2 and BERT to web apps or mobile apps. In short, the steps to build a generative model for an app are as follows

1. Decide what you want to generate
2. Find a relevant dataset. See this https://ai.google/tools/datasets/ or https://toolbox.google.com/datasetsearch
3. Train a generative model on that dataset. Usually a GAN will do the trick, else try a VAE . OpenAI has an impressive list of them here
4. Use that trained weights file inside of a web framework like Django or Flask to create an API.
5. Build a full-on web app, or use that API from a Flutter app! I hope you’re starting to piece together how to think about this! If you focused solely on how GANs work or on how VAEs work, you’d be lost for weeks in theory while losing sight of the bigger picture! It’s best to decide our high level goals first, develop a path, then dive into theory with a dedicated, larger purpose.