Revisiting NoSQL
Storytime
I’ve been working with NoSQL databases almost since the beginning of my developer career. For the most part, I was busy making apps for iOS, many of which required a few server-side utilities like user accounts or a simple data storage. As I certainly wasn’t capable of developing a full blown backend service back then, I resolved to Firebase as often as possible.
That’s how I met your mother a NoSQL Database. At that time, Firebase was mostly about their Realtime Database, which had a structure similar to a big JSON file. It had no schema, no data types, no relationships, etc. 😰
It was simple, but also, it was everything I needed at that moment. And you could even subscribe to write events to make the app reflect the data, yeah, just like the name implies, in real-time. I was amazed. Maybe the thought of storing all the user data without a “rigid” structure scared me a little bit. But it was the best I could do.
Then Firebase evolved and introduced their next iteration of that concept — Firestore Database, which I happily jumped onto with my next project. And oh boy, was the Firestore a much more refined solution. It was adapting a document database system (just like MongoDB).
They introduced all the basic, and some not so basic (like latitude and longitude as one) datatypes, clear separation between documents and collections of documents. You could also store a reference to another object in the database. I felt like I’ve been finally using a database and not a JavaScript Object, stored in the cloud. Developer Experience got improved quite a bit.
SQL Apprentice
As I said, developer experience was improving and growing. My experience was growing. And with that experience, someday, I became ready to learn my first backend framework — Django. And with Django came the PostgreSQL, a “real” database. That’s how I thought of it at the beginning. Django became my go-to solution for each future project that required the things I’d get Firebase applied to before.
I learned a bit about SQL, I got more confident with database design and data normalization, and even wrote some complex queries. Ok, those queries were not actually written with SQL and using a Django built-in ORM instead. That’s something supposed to help you work with the database. At first, I assumed that it’s ORM’s fault and it’s inefficiency.
That’s why I left it as it was because I thought I couldn’t do much about it. After some time, I realized how badly I misused the ORM and started diving deeper into the relational database world, so I can come back and fix that mess one day.
Since then, I’ve been working with Postgres for quite some time and treated it as a standard solution when working with a server-side database for my project. Just for the sake of clarification, it didn’t matter if it was Postgres per se; what I meant is I was using a RELATIONAL database, and I was getting better at it.
Stepping out
Not a long time ago, I got back to the project with the complex queries I mentioned before. Let’s call it “Project X” for now. Those queries were living there for about a year without me checking on them.
I tried to apply some of my new knowledge and experience to optimize them, as with time and the database growing, those queries only got slower. Some of them now took about 15 seconds to execute. I knew I had to do something about that. And I did; I managed to rewrite them and get the execution time well under 2 seconds. It’s not that I’m so good at it, I just did a terrible job writing them at first.
Getting under 2 seconds still wasn’t perfect for me, but I could settle on this kind of improvement. The resulting query was much cleaner, and I was quite happy with the solution, but there was one thing still bugging me. The underlying data model was something I knew should’ve been improved. But I didn’t know how.
In terms of SQL principles and database design rules, there wasn’t much to complain about. Probably a senior database engineer would’ve come up with a much better model. But it wasn’t bad.
Aha moment
Last weekend, I was listening to a podcast about JavaScript on my way to a grocery store. It was a new podcast for me, so I went with the latest episode, which happened to be about MongoDB. It grabbed my attention pretty fast as I got some serious Firebase flashbacks Hey, I think I’ve seen that somewhere!
Then I went for the other two episodes about MongoDB, with this cool guy Joe Karlsson, who did a fantastic job describing some more use cases and current capabilities of this database.
What I was most intrigued about were some completely new (to me) ways to model data in the database. I spent the whole afternoon reading about structuring data with NoSQL document databases and design patterns.
Looks like the NoSQL landscape, with MongoDB’s help, grew a lot throughout the last few years, when I was away, being content with my “proper” SQL database. They certainly caught up with the “older players” in terms of functionality. It looks like now both worlds are trying to implement the other’s best parts in their offerings, thus getting closer and closer to each other.
Now into practice
I’m the one to get excited quickly after reading a lot about a particular tech. And this time, it was no different. But hey! I’m aware of that, and I try to take that into account. See, I’m a responsible developer! Keeping that in mind, I decided to see how I would design the database for my “Project X” if I were to start right now.
After a bit of sketching on a piece of paper, I saw that using the things I learned about NoSQL databases, I could reflect my underlying data in a slightly more understandable way. Probably could make it easier to query too. I don’t want to say that it’d be certainly better in every way, but it might have some serious advantages over the current solution when applied to “Project X.”
I think I learned something more than just a way to model data…
Such wisdom
I hope you did too. I hope you not only learned that when starting a new project, you should not go blindly for a NoSQL database, nor a SQL one, but that you should maybe apply this mindset to any other tech stack decision you will make.
Being aware of the frameworks or libraries and knowing why you are using them, might prevent you from over-engineering the product. Maybe a more straightforward solution will do the job equally well, and you don’t need that ton of dependencies.
Of course, every time try to examine what your needs are and what your strengths are. For the most part, it’s probably best to stay within reach of your current skills. The whole process is going to be much more smooth. But do not hesitate to explore and see if there is a better fit for you. Maybe it’s something that you or your team can adapt over time to benefit from a better project structure in the future.
Originally published at https://albertpawlowski.com.
