ETL with standalone Spark containers for ingesting small files
At Mic, we have high volumes of data streaming into our ingestion pipeline from various sources. Much of our data is generated from user interactions on the website (article views, video plays, etc.), but some come from…