How to Spend ¾ of Your Yearly Budget in 3 Weeks: A PySpark Cautionary Tale

This is a cautionary tale of how I spent a large amount of money running PySpark clusters and what I discovered about deploying a recommendation engine at scale. I’ll describe how I finally chased down an intermittent error and lessons learned along the way including why you can’t always write PySpark code like Python code and how important it is to check third party library integrations.

Originally published at https://shoprunnerblog.wordpress.com on March 4, 2019.