6 Machine Learning Ideas for the Modern Deep Learning Practioner
These protocols will go on to define the future. Make sure you master them
To help me understand you fill out this survey (anonymous)
Machine Learning evolves at a crazy pace. There are tons of new ideas and procedures being invented. Just two years ago, multi-modal training (training ML models with input from text, images, sounds, and videos) was a very niche idea. Now different publications from Google AI and Deepmind have got this thought on everybody's mind.
As I’ve gone through tons of research, technical publications, and interactions with experts, I found there to be many recurring ideas that show up a lot in the high-value solutions being developed. In this article, I will share those ideas with you, so that you can be armed with some of the most impactful/powerful ideas in ML.
1: Encoder-Decoder Pairs
This is one of those ideas that often gets overlooked when we discuss amazing Machine Learning Solutions. On the surface, this is a very simple concept to grasp. Encoders take your input and encode it into a latent space. The Decoder takes vectors from the latent space and transforms them back. This makes them a natural fit for Language processing tasks, where they have found great success.
As you read through the large-scale ML solutions such as Facebook’s Language Transcoder, Language Translation, Text Reconstruction, and Large Language Models we spot Encoder-Decoder pairs being used in the processing. However, their utility doesn’t end there.
This can be used in a variety of ways in Computer Vision. Adversarial Learning, Reconstruction, Image Storage, and Generation are some notable examples. It also plays a crucial role in DALL-E. We take the text input and encode it into the latent space. Then we can take a decoder that decodes the latent vectors into an image. This is how we are able to generate images from text descriptions. Facebook AI’s Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors.
Their effectiveness boils down to their simplicity. The concept behind taking input vectors and representing them in a latent space is a deceptively powerful concept. It can be applied to tie related concepts from different domains, giving them unmatched versatility. The aforementioned publications from Deepmind and Google AI hint at how multi-modal training might be the key to AGI. If that is the case, Encoder-Decoder pairs will play an important role. I’ll be covering this in more detail, so make sure you connect with me using the links at the end of this article.
2: Attention
With the prominence of Large Language Models and transformers, we can now say that the attention mechanism has been revolutionary. The attention mechanism in Transformers allows them to identify the parts of a sentence that are important. Attention allows Transformers to filter out the noise and capture relationships between words that are even far apart.
Everyone already knows this in the context of NLP. What I didn’t know is that this holds true even for CV. The attention mechanism allows Transformers to keep a “global view of the image” allowing them to extract features very different from ConvNets. Remember, CNNs use kernels to extract features, which restricts means that they find the local features. Attention allows Transformers to bypass this.
The above picture is taken from the very interesting, Do Vision Transformers See Like Convolutional Neural Networks? It’s interesting enough that I will do a breakdown of this paper later. The important aspect is the following quote, also from the paper.
…demonstrating that access to more global information also leads to quantitatively different features than computed by the local receptive fields in the lower layers of the ResNet
3: Random Forest
Now I know some of you are very confused seeing this included here. Random Forests are the classic Machine Learning model. They’re right up there with Logistic Regression, Naive Bayes, and KNNs as your first models. So why are they mentioned in the list of modern techniques?
RF classifiers and regressors are as OG as they come. True. But the RF idea has evolved into all kinds of other techniques. Random Forests can be used in tasks effectively involving:
- Outlier Detection
- Feature Engineering
- Feature Importances, Downsampling, and Selection
- Data Imputation
RFs are amazing because their versatility makes them powerful. They are robust to outliers, can tackle missing values, and are perfect for messy datasets. In my article, How to handle missing environmental data, I demonstrate how these properties are perfect for handling a diverse range of datasets.
4: Randomness
If you’re one of my original followers, you know that I’m a huge fan of introducing noise and randomness to your Machine Learning training. I’ve been stressing it for a while now, back when it wasn’t as mainstream. These days, all the cool kids are doing it.
Randomness can be the game changer when you want to create models that can generalize over many distributions. Along with this noisy training, has potential benefits for adversarial training. This article goes into depth about Using Randomness Effectively in Deep Learning.
5: Convolutional Neural Networks
This is a classic, but it is mentioned here for a good reason. CNNs are simple, intuitive, and can have great performance. While they are specialized in Computer Vision tasks, they are the kings in that domain. I don’t need to sit here and talk about how amazing CNNs are. That would be a waste of everyone’s time.
What is often overlooked when we discuss ConvNets is their way of automatically extracting features using Kernels. We know that features will make or break a model’s performance. As extremely large-scale Machine Learning becomes more common, the importance of having good preprocessing will become a differentiating factor. Mastering the feature extraction methodologies of CNNs will serve you very well.
6: GANs
The GAN philosophy has seen a bit of a resurrection recently. More accurately, we have seen some very game-changing technologies implementing the GAN architecture. This has brought the idea back into “popular” discourse. The idea of training two models with opposing objectives by pitting them against each other leads to phenomenal results.
This extends far beyond Traditional GANs. GANs are based on the combination of Discriminative and Generative Learners. We can combine these two techniques in a variety of ways to result in some exceptional results. We can implement Evolutionary Learners in these cases, which will allow you to attain some exceptional results, such as with Project Geneva.
That’s it for this article. Naturally, mastery over Machine Learning is crucial to truly leverage these powerful ideas. This article gives you a step-by-step plan to develop proficiency in Machine Learning using FREE resources. Unlike the other boot camps/courses, this plan will help you develop your foundational skills and set yourself up for long-term success in the field.
For Machine Learning a base in Software Engineering, Math, and Computer Science is crucial. It will help you conceptualize, build, and optimize your ML. My daily newsletter, Coding Interviews Made Simple covers topics in Algorithm Design, Math, Recent Events in Tech, Software Engineering, and much more to make you a better developer. I am currently running a 20% discount for a WHOLE YEAR, so make sure to check it out.
I created Coding Interviews Made Simple using new techniques discovered through tutoring multiple people into top tech firms. The newsletter is designed to help you succeed, saving you from hours wasted on the Leetcode grind. I have a 100% satisfaction policy, so you can try it out at no risk to you. You can read the FAQs and find out more here
Feel free to reach out if you have any interesting jobs/projects/ideas for me as well. Always happy to hear you out.
For monetary support of my work following are my Venmo and Paypal. Any amount is appreciated and helps a lot. Donations unlock exclusive content such as paper analysis, special code, consultations, and specific coaching:
Venmo: https://account.venmo.com/u/FNU-Devansh
Paypal: paypal.me/ISeeThings
Reach out to me
Use the links below to check out my other content, learn more about tutoring, or just to say hi. Also, check out the free Robinhood referral link. We both get a free stock (you don’t have to put any money), and there is no risk to you. So not using it is just losing free money.
Check out my other articles on Medium. : https://rb.gy/zn1aiu
My YouTube: https://rb.gy/88iwdd
Reach out to me on LinkedIn. Let’s connect: https://rb.gy/m5ok2y
My Instagram: https://rb.gy/gmvuy9
My Twitter: https://twitter.com/Machine01776819
If you’re preparing for coding/technical interviews: https://codinginterviewsmadesimple.substack.com/
Get a free stock on Robinhood: https://join.robinhood.com/fnud75
