Loja Integrada & Pagali Switch To DoubleCloud To Make Running ClickHouse® Easier

Pagali switched to a managed ClickHouse, as running ClickHouse themselves was too time consuming, pulling valuable staffing resources from more important projects.

Highlights

• Pagali currently processes 45% of all Loja Integrada’s transactions, needing somewhere efficient and lightning fast to store and analyze all those transactions.
• Once finished, Pagali’s instance was half the size they expected, due to it being 6x more performative, meaning they now pay around 50% less compared to their previous RDS Aurora db (Postgres).

“Made the entire hosting, maintenance and synchronization process uncomplicated. We are very happy with the solution and will extend it to other operations”.

Udlei Nati, CTO, Pagali

Who Are Loja Integrada & Pagali

DoubleCloud recently had the pleasure of speaking with Udlei Nati, original founder and CTO of Loja Integrada, an e-commerce platform based in Brazil.

Loja Integrada, through their business unit Pagali, had recently started using DoubleCloud’s managed ClickHouse platform so we took the opportunity to catch up with them, learn a bit more about the organization and how they were finding DoubleCloud.

Loja Integrada was founded over ten years ago and is Brazil’s largest e-commerce platform, supporting and empowering a community of small businesses and sole traders to sell their products online. It’s a platform that’s constantly growing and updating, seeing a huge surge of sellers and buyers during the pandemic. After Udlei sold the company, he stayed on for an additional two years in his role as CTO, before taking some well-deserved time off to travel the globe for a year, touring the UK, the US and many other destinations before settling down in Germany, and then finally, Portugal.
Around two years ago however, the current board of Loja Integrada approached him about resuming his role as CTO as the company had massively grown in the interim. Working closely with the Loja Integrada board, Udlei set up Pagali, a brand new business unit of Loja Integrada to complement their fast growing platform.

Pagali is a payment gateway similar to PayPal that has already been rolled out to nearly half of the stores on Loja Integrada’s platform, as it makes the payment process much more efficient and profitable for everyone involved.

Pagali, currently employs Udlei and seven other people, including a front-end developer, two back-end developer, a designer and three more and currently processes about 45% of all of Loja Integrada’s transactions.

Solution The Challenge:

Due to the speed and scale of Loja Integrada’s growth and the huge adoption of Pagali across the platform, a huge amount of data had been (and still is) being generated incredibly quickly. Whilst that’s a nice problem to have, due to the current size of the team, it means everything that Pagali does has to be easy to maintain. To keep their momentum going, there’s no time for convoluted, complex solutions that require an inordinate amount of time just to maintain… they need that time to innovate and grow.

Instead, they’ve opted for a more component led design approach, with lots of small solutions that can all operate independently, but when combined create one large solution… almost like Lego bricks. That also involves a lot of open-source tech to avoid vendor lock-in, for everything from the payment authentication protocols through to the emails they send.

PostgreSQL was originally used to handle their data but after the first six months, Pagali had quickly outgrown it and started looking around for a different solution.
Unfortunately they didn’t have time to do a deep analysis of dozens of different platforms so ended up making the switch to Amazon Aurora, a PostgreSQL alternative made directly by the AWS team.

Everything was working well until the end of last year (2022 in case you’re reading this case study five years from now) when Pagali needed to make their dashboards a lot more complex to better represent the amount of data they were now handling and do a deeper analysis of what was going on within the business.

Although Loja Integrada has a BI team that could have handled that for Pagali, they wanted a solution that was unique to them and could grow and scale as they did, without having to rely on outside resources… in essence a solution that fit in with their component led design ethos.
Redshift was briefly considered by the team but was dismissed as an idea as it wouldn’t have been fast enough for what they needed. AWS Glue, AWS Athena and even Parquet Files were all also considered, or perhaps a combination of all three to build a data lake within S3, but that would have involved far too many moving pieces, that would require a large amount of knowledge and time to make sure it was fully optimized.
It also wouldn’t have been the easiest to maintain if (when) something went wrong.
And that’s when the team came across ClickHouse® and DoubleCloud’s managed ClickHouse solution.

The Solution & Outcome:

The team at Pagali had heard about ClickHouse before but hadn’t used it themselves so began testing and found (like many before them) that it was fast… in fact much faster than they expected. The decision had been made; ClickHouse was perfect for their needs and they’d use that. However, they also quickly realized that it wasn’t as simple as they’d hoped to sync all their data within ClickHouse. The goal at Pagali is to always aim for self-managed solutions — They’ve the skills and talent to do it themselves, but why waste time when they could be innovating and growing instead?
That’s when they came across DoubleCloud.

They quickly signed up for our free trial, with everything working perfectly during the installation process, finding our Transfer service simple to use. However, due to their specific needs, there were some minor issues here and there when they started syncing their data so, to save time, they started looking for other solutions in parallel, just in case DoubleCloud didn’t work out.
Pagali’s worry was that DoubleCloud may not have been able to solve all their requirements, however as it turns out, there was no parallel solution required, with them singing DoubleCloud’s praises.

New problem cropping up? Solved!
New requirement added? Solved!
Could we do this…? Solved!

In fact, the Pagali team felt a great deal of synergy with DoubleCloud, both being start-up’s, doing everything fast and efficiently and building a reputation for great customer service whilst also building out our respective platforms.

Once all of Pagali’s data had been synced, it was time to move onto stage two of the project. Moving all the dashboards and reports, connecting them to ClickHouse and getting impressed by the performance.

In fact, Pagali mentioned that once everything was completed, the instance they’re using was half the size they expected and 6 x more performative, meaning they now pay around 50% less compared to their previous RDS Aurora db (Postgres).