How to become a Data Engineer

What a Data Engineer does, and what they don't

A Data Engineer builds and keeps alive the paths data travels: pulling it from wherever it is born, landing it in a shape that can be queried, and making sure all of that happens again tomorrow without anyone pushing it by hand.

What they don't do, however much job ads blur it:

• Not training models. That's the Data Scientist or the ML Engineer. You hand them data that's ready.
• Not building the dashboard. That's the analyst. You make sure the table behind it exists, is fresh, and doesn't lie.
• Not running the servers. That's platform or infrastructure — though in a small team it lands on you anyway.

The practical difference: analysts get asked what the data says; data engineers get asked why yesterday's number changed and why the 3 a.m. job didn't run. If that second question makes you curious rather than annoyed, this is your role.

What you don't need to start

Every item on this list stops somebody every day:

• A master's degree. The portfolio outweighs the credential in almost every hiring process for this role.
• Spark in month one. Distributed processing solves a problem you don't have yet. When you have it, you'll learn it.
• The cloud before the fundamentals. Managed services rename themselves every couple of years. SQL and data modelling don't.
• Twenty tools. Nobody uses them all. A real team runs four or five, and they differ between companies. What transfers is judgement, not the brand name.

There is also no honest number of months. It depends on how many real hours a week you put in and how much code you write — not on how many courses you open.

How this path is laid out

Three levels. Each one has what to learn, a project that closes it, and how to know you have it: a criterion you can check, not a feeling.

If you want the role's skill map instead of the sequence, that's the 5 core skills of a Data Engineer, with a self-assessment. This guide is the order; that page is the map.

Level 1 — Answer a question using data you didn't create

This is the floor. Without it, everything above is cargo cult.

What to learn

• SQL, properly. Not SELECT *: joins, aggregations, window functions, and above all modelling — why a table is split into three, and when that helps you or gets in your way.
• A real database. PostgreSQL is the honest standard to learn on. DuckDB lets you practise analytics without running a server.
• Enough Python. Read a file, call an API, transform, write. No advanced object-oriented design required.
• The command line and Git. If your code isn't versioned it isn't code, it's a draft.

The project that closes it

Take a public dataset you actually care about — city budgets, prices, weather, transit — load it into Postgres with your own script, and answer five questions that cannot be answered from a single table. Put the five queries in a repo with a README explaining where the data came from and what you cleaned.

How to know you have it

You can look at an unfamiliar eight-table schema and, within half an hour, say which is the fact table, which are the reference tables, and which join is going to duplicate rows.

Level 2 — Make the data arrive on its own

This is where the craft starts: the difference between a script you ran once and a process someone relies on.

What to learn

• Versioned transformation with dbt. The highest-return entry point of this level: it turns loose SQL into a project with dependencies, tests and docs.
• Ingestion. Airbyte for connectors not worth writing; your own Python when the source is odd.
• Orchestration. Airflow is what you'll find in most job ads; Dagster models the data you produce rather than only the tasks, and tends to click faster. Learn one well before looking at the other.
• Formats and storage. Why Parquet and not CSV, and what changes when the file lives in object storage such as MinIO.
• Docker. Not for production yet — so your project runs the same on someone else's machine.

The project that closes it

A daily pipeline that wakes up by itself: pulls from a public API, lands Parquet, transforms with dbt with at least three data-quality tests, and fails loudly when the source changes. Orchestrated, and containerised.

How to know you have it

You break the source on purpose — rename a column — and your pipeline stops with an error that says what happened, instead of quietly writing bad data.

Level 3 — Make it hold up and be trustworthy

This one isn't studied in the abstract: you learn it when volume or the team demands it. Reach for it when you have the problem, not before.

What to learn

• Scale. Spark when the data doesn't fit on one machine; Kafka when it has to arrive now and not tomorrow.
• Lakehouse. Iceberg or Delta Lake over object storage, queried with Trino. It's the architecture that shows up most in serious postings today.
• Quality as code. Great Expectations: expectations about the data get declared and tested, like any other code.
• Governance and catalogue. OpenMetadata, so someone who isn't you can find out where a table came from without asking you.

The project that closes it

A minimal lakehouse: Parquet on MinIO, an Iceberg table on top, queries through Trino, lineage visible in the catalogue. It doesn't have to be big — it has to be complete end to end.

How to know you have it

You can explain to someone, without diagrams, why you chose an open table format over a traditional warehouse, and in which case that decision would have been wrong.

The portfolio: what convinces and what doesn't

What doesn't: twenty repos finishing the same tutorial, notebooks with no README, course certificates.

What does:

• Two projects, finished. One from level 2 and one from level 3 beat ten half-done.
• A README that explains decisions. Why that tool, what you rejected, what broke and how you fixed it. It's the only thing that shows judgement, and judgement is what gets hired.
• Data you can talk about. A dataset from your city or your previous industry gets you a better interview than a generic one.
• It runs. docker compose up and it works. If you have to explain how to install it, nobody will try it.

The search: how to read a job ad

Job titles lie often. Read the technology list instead:

• SQL, an orchestrator and a warehouse → an actual Data Engineer role.
• Power BI or Tableau first → an analyst under a different name.
• Kubernetes, Terraform and on-call → platform or data DevOps.
• All eleven tools → either they don't know what they need, or you'd be the only person in the area. Ask in the interview how many people are on the data team. The answer tells you more than the whole ad.

What technical interviews usually ask: SQL with window functions, modelling (design the schema for this case), and one open design question — how would you load this every day. That last one isn't fishing for a tool: it wants you to mention what happens when it fails, when you backfill, and how you find out.

External resources

Curated, free unless noted, and with the source visible:

• Data Engineering Zoomcamp (DataTalksClub) — a free, complete course with a capstone. The most solid starting point if you want structure.
• Data Engineering Wiki — community-maintained resources, organised by topic.
• The Data Engineering Cookbook (Andreas Kretz) — concept map and interview questions.
• SQLBolt and Select Star SQL — SQL from zero, in the browser.
• dbt documentation — the official source; the tool's own tutorial beats most courses about it.
• Start Data Engineering — articles about design decisions, not syntax.
• Designing Data-Intensive Applications (Martin Kleppmann, paid) — the book that separates people who use the tools from people who understand why they exist. Not for level 1.

Every tool named in this guide has its own entry in the kiosk: what it is, what it's for, when not to use it, and a link to its official documentation.