Collection And Processing Of Statistical Data. Part 2

The Ubex project continues to develop the platform as newer modules required for its proper functioning are being added and developed.

As we have already described in our previous posts, the JS counter hosted on the webmaster’s site sends data about visits and visitors to the fail-safe cluster of the Pixel Collector servers hosted in the Amazon AWS cloud.

What follows is that this data is enriched on the Pixel Collector servers with service headers and transferred to the Kinesis Data Streams streaming service.

One of the tasks that we face is to update the data on the statistics of advertising campaigns (the remaining budget, impressions, clicks, etc.) as much as possible. To solve this problem, we have implemented the Lamda architecture using Apache Spark.

In fact, we have two data streams, the operational data and the statistical data.

The first half of our architecture components using the Spark Streaming technology reads incoming data from Kinesis Streaming and updates our internal statistics in near real-time mode. This was made possible thanks to the extremely low latency between the receipt of data in the Kinesis Data Streams and their readability, as well as through refraining from processing data in batches in favor of streaming data reading.

The second half launches the scheduled Spark SQL tasks that re-process the accumulated raw data, perform additional cleaning and processing with them, and update the data in the database.

The logical question that might arise is why we need such complexities in the first place. In fact, the first stream contains the operational data. We use it to take into account ad displays and withdraw funds from the advertiser’s account in real time, otherwise there will be artificial increases in ad impressions and we will remain in debt to our partners by RTB.

The second stream contains slow reports. Once every 15 minutes, we recheck the statistics, count the common values ​​and write down the aggregated data into the database, then display it in the Personal Account of the webmaster and the advertiser.

The Ubex project continues to develop and apply a variety of advanced techniques and services to ensure that the platform functions flawlessly. Though the infrastructure is not yet entirely ready for release, great progress has been made and the development process is on track with the stated roadmap. Stay tuned for more news from Ubex.