@ Promoted
Published in

@ Promoted

Free and Open Source Personalization ML for Y Combinator Companies

If your product has search, feed, or recommendations, and you’d like to use machine learning for personalization, you can get the best algorithms hosted by top companies for free using your Y Combinator credits. Most open-source implementations of these algorithms are at https://recbole.io/ . Otherwise, use AWS Personalize using your YC AWS credits.

If you are not a Y Combinator company, you can get a deal from AWS, Microsoft, or Google. Even paying full price, these standard solutions are dramatically less expensive and of higher quality than anything 99% of startups can build in-house. Any in-house solution should be tested against these standard solutions to evaluate true engineering ROI.

The state-of-the-art in “low data” recommendations are item sequence predictors using HRNN-like neural networks. “Low data” means “clicks and purchases only.” These are not what “top companies” like Facebook or Pinterest use in production for ads, feed, or search ranking. Big companies use “big data” solutions. However, the spirit of sequence-prediction-as-personalization will be included somewhere in the recommendation system, either as part of a complex NN architecture or as a feature transformation consumed by other models.

If you’ve outgrown AWS Personalize, then consider investing in measurement and data infrastructure and then ML OPs before attempting complex machine learning. Anything optimized must be measured correctly first, and it must continuously operate without failures. This is doubly challenging if your product is evolving and there is a commercial incentive to game your system as in e-commerce and especially ads. Always test your model against a smoothed, average-engagement-rate prediction, and keep this simple prediction available at runtime to handle ML failures.

A typical AWS Personalize system diagram

Top Personalization Solutions

AWS Personalize: The standard. We’ve used AWS Personalize as a baseline collaborative filtering system. Anything more complex that we train must outperform AWS Personalize in offline evaluation before we productionize it. AWS Personalize can cost a few thousand per month, which you can pay with your $100k Y Combinator credits. We don’t see AWS Personalize outperforming a basic popularity metric unless there is both a large diversity of items and most users are repeat users. Median latency is about 20–40ms, but we regularly see >100ms latencies, so you’ll need a fallback for timeouts.

Google AI & Google Remote Config Personalization: Google has a fantastic basic solution for anything ML you may need. The “Remote Config Personalization” for Firebase covers most growth engineering work of A/B testing single attributes. Again, you can pay for these using your Google YC credits of up to $100k.

Microsoft Personalizer: I haven’t used this, but it seems to be AWS Personalize for Azure. Microsoft doesn’t have a YC Deal, but they have a $150k credits program for startups.

ByteDance (TikTok): Used in Asia. Youlong Cheng is the tech lead and is excited to meet more teams in the USA. They don’t have a YC deal yet, but they’re looking into it.

Notable Y Company “Recommend” add-ons

These appear to be user event sequence predictors. Amplitude integrates with your existing Amplitude analytics and Algolia integrates with your existing Algolia search. If you already use one of these two products, the “recommend” add-on will get you most of the value of adding something like AWS Personalize yourself.

--

--

--

http://promoted.ai  —  Online Marketplace Engineering and ML

Recommended from Medium

Flex vs Grid (CSS)

Functional Patterns Part 1

GroupWise to Office 365 Migration

Creator Interview With David Thorpe

Creator Interview with David Thorpe

AWS vs Parler. How I see it?

(B.O.O.K.$) SAFe 4.5

Kubernetes Trends at KubeCon 2017

10 Beginners-Guide To Learning JavaScript, Node, Python, and Android App Development

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Andrew Yates

Andrew Yates

Marketplace design and engineering @ Promoted.ai

More from Medium

Crane: The lifeblood of PrimeHub

Computing the driver acceptance probability or … the DRAC of Beat

Scaling our dependency graph

VertexAI’s Feature Store for dummies