…lly spawn into existence as projects get started, teams drift and inevitably as acquisitions occur. It’s the task of the business intelligence (now data engineering) teams to solve these issues with methodologies that enforces consensus, like Master Data Management (MDM), data integration, and an ambitious data warehousing programs. Nowadays, at modern fast pace companies, the silo problem quickly grows out of proportion, where you…
…ess man can you be if you don’t speak good english? While generations of technologies age and fade, SQL is still standing strong as the lingua franca of data. A data engineer should be able to express any degree of complexity in SQL using techniques like “correlated subqueries” and window functions. SQL/DML/DDL primitives are simple enough that it should hold no secrets to a data engineer. Beyond the declarative nature of SQL, she/he should be able to read and
understand database execution plans, and have an understanding of what
all the steps are, how indices work, the different join algorithm and the
distributed dimension within the plan.