Salut! Hi everyone, I’m back, this time with a new post with an important matter, communication data standardization, I was witness of an incredible mess created by a Jr developer on our backend, he spent two days just creating models and make them fit in both sides, frontend and backend and adding validations and all this data issues we can have, it was a task with a story point of three. After two days, he still needed to create and implement the REST endpoint, do you know what was worse? The models didn’t fit the models in other microservices hahaha, so, I think it’s important for us to understand and take care of this side, even when it’s not coding coding, and we as devs, don’t like things that are not related with coding, let’s do it quick to not get you bored.
Communication Data
What do I mean by communication data? Well, data is everywhere, it could be a color, name, object, shape or anything else, it is important because, any system works with data, user data, metrics data, images, machine datas and others, and systems needs to communicate, frontend needs to talk with backend of course, backend may needs to get data from an external server and more, so, when I said communication data I mean, data that allow systems to understand each other, our microservice for orders needs to have the same user structure than our account microservice, if not, some functionalities could fail or lack of relevant information, having the right information could save life, imagine a hospital where the machine that analyze blood doesn’t has the data needed by the machine to analyze surgery’s candidates, that could bring a huge mess, data modeling is as important as the code.
Standardize Data
Now that we talked a bit about mismatched data, you may understand why we need to standardize it, but first, let’s have a deeper talk about the examples above. It is true that mismatched data could be a problem, but also, mismatched data it’s not a bad thing, I know, I know, I’m saying opposite things but let me explain:
// Account microservice user model
class User {
private UUID id;
private String firstName;
private String lastName;
private String email;
private String password;
...
}// Order microservice user model
class User {
private UUID id;
private String firstName;
private String lastName;
private String email;
...
}Do you see the difference? And is that an issue? haha in case you didn’t see it, account microservice has a password field in the user model and order microservice don’t and it’s okay, we don’t need the user’s password in our order microservice, why should we needed for? Standardize is not a copy and paste, it’s a structure model to communicate what we need, but it will be a problem if I remove the firstName from our order microservice class or if I rename it to userFirstName, then the mismatch data will be an issue.
Designing and Development
Now that I explained why data standardize it’s important, let’s talk about the real world, how to actually do it…
Designing
This is the worse part haha, I don’t mean that is bad, I just mean that it’s the most difficult thing, and not just because data, it is also complicated for systems, architecture and everything, why? Because we don’t have anything clear, there’s not following steps, or clear instructions, there are abstract concepts, ideas, data and requirements, so, it is a complex job.
I won’t talk about the designing process in deep but I’ll suggest the following steps (they’re not a bible of design):
- Investigation: the first thing you need to do it’s understand what you’re about to do, make the questions, investigate, read, do all the research you need to understand the problem so you can see a possible solution
- Extraction: now that you posses the information, you need to extract (depending on your criteria) the relevant data for your needs, here, it will depend entirely on your criteria, try to ask yourself any details, do we need the user’s name? do we need user’s IP? and also try to put yourself in client’s shoes, see what kind of data you need to bring the user what they’re looking for, “When user does click here, we should run a process to create a certificate, what kind of data do we need for that? ask yourself everything and take the best choice you can
- Modeling: you have the extracted data right? well, now you’ll need a model, and I’m not talking about classes or interfaces or any coding item, but, you need to see your data interacting each other, how could you do it? by creating diagrams, illustrations or any other tools that will help you to see the interaction and the behavior in a simulation of the real process
- PoC: Unless you’re so sure about your results, I’ll recommend to do PoCs (Prove of Concept), here you’ll create a small piece of the system you want to develop (just what is necessary to probe your model is correct) and once you confirm it works for you, you’re ready to code
Development
Proves of concepts are one thing but the real application challenge it’s coding, why? because it will be a production development, depending on your project the size could be huge, and it will happen, you’ll take an eternity just modeling data and validating, that’s why smart people created tools to help us with that, I’ll talk about 3 of them:
Json Schema
This is a tool to parse json structure to actually models, this json structure has validations like, min length, regex, required fields, and more, you can use it to agile the process of creating classes and standardized the data across the microservices, just need one file that will compile everywhere.
GraphQL
GraphQL has a similar feature but this will depend on your requirements, not all the companies use GraphQL, but in case you do, you can create classes by graphql files
ChatGPT
A popular tool that you can use to improve your speed (if you’re allow to use it), this AI can generate json schemas and graphql queries based on the text you send it to it.
This are the tools (there are a lot more) you could use to improve your development process and avoid being like my Jr dev friend that spent 2 days just modeling data for a two story points task.
That was it for me, I’ll post something about Json Schema and GraphQL for modeling examples if I think it’s needed but, they’re pretty simple to understand so, I’m not sure to do it…
Thank you for reading
And see you in the next
