Don’t overlook the cloud storage cost — With the current state of the economic situation, it’s more important than ever to maximize our cash on hand and develop a series of cost optimization strategies. …

Effectively communicate with data users — The status page is not a new thing in the software engineering field. It is essential for many platforms in today’s fast-moving world to transparently communicate up-to-date incidents, scheduled maintenance, and real-time service status with users. …

An important step before choosing the database — In database systems, data can be stored in two ways: row-oriented data stores and column-oriented data stores. By changing the way data is stored on the hard drive of the computer, we can make a big impact on the database’s performance. The choice also determines whether it is optimized for…

Don’t be limited by your job description — In the past five years of my career, I became a full-time Software Engineer and a full-time Data Engineer. …

Spend money in need — One of the cloud’s benefits is its unlimited scalability. Data storage, computation power, and networking are all can be scaled in most cloud computing infrastructures like GCP and AWS. However, scalability can cause cloud waste — overprovisioning resources, nonoptimal data storage, too many idle resources, etc. Companies could waste up…

Key metrics to keep your stakeholders happy — For today’s data teams, the demand for real-time, accurate data has never been higher, but data reliability never gets full attention. Data is growing quickly every day, they come from more sources and formats than ever before. To keep up with it, data teams invest a lot in managing scalable…

Deep dive into the import system — Reusability is one of the key metrics to measure the quality of the code. It is the extent to which code can be used in different programs with minimal change. In Python, we use import to import code from a module. But have you ever been curious about how import…

Cost and resource management in BigQuery — Cost and resource control is a significant issue for Data Engineers when working with cloud-based big data stacks. Without this knowledge, you can easily end up with an extremely-high cloud service bill or receive complaints from the organization about the slow environment. …

Less is more? — The term one-liner comes from comedy where a joke is delivered in a single line. A good one-liner is said to be meaningful and concise. This concept also exists in programming. Python one-liners are short programs that can perform powerful operations. …