62 Data Engineer interview questions with answers (2026)

All Data Engineer interview questions

How do Data Lake and Data Warehouse differ? When should you use each?

What is Change Data Capture (CDC)? How does it work and what are its use cases?

What is Kafka Connect? What problem does it solve in data pipelines?

What is Schema Registry in the Kafka ecosystem? Why use Avro/Protobuf instead of JSON?

At-least-once vs at-most-once vs exactly-once in Kafka: differences and how to achieve each?

What is Amazon Redshift? When should you use Redshift instead of RDS? Explain columnar storage and distribution styles.

What are normalization and denormalization, and what is the trade-off?

What is the difference between a view and a materialized view?

How do OLTP and OLAP differ?

What is a window function and how does it differ from GROUP BY?

What is the difference between a star schema and a snowflake schema (dimensional modeling)?

Classic SQL exercise: how do you find the second-highest salary / top-N per group?

Kafka Streams vs Apache Flink — when do you pick which?

What is the logical execution order of a SELECT statement, and why does it differ from the written order?

How do you compute a running total with a window function, and how do `ROWS` and `RANGE` differ in the frame clause?

A 500M-row log table is only queried for the last 30 days — how would you partition it by time? What does partition pruning require?

What does a Data Engineer do? How does the role differ from a Data Analyst and a Data Scientist?

What is big data? How do you tell that a problem is actually a big data problem?

Why does a data lake usually store Parquet files instead of CSV? When does Avro fit better?

Design a pipeline that moves order data from a production MySQL database into a data warehouse for a daily revenue report. How would you do it?

What are the steps of ETL? Describe a basic ETL process.

How do ETL and ELT differ? When should you choose each one?

An ETL job fails halfway and the rerun duplicates data in the target table. How do you design the pipeline so reruns and backfills are safe?

What is a lakehouse? What does it add over an old-style data lake (files on HDFS/S3)?

What is the medallion architecture (bronze, silver, gold)? Why split data into several layers like this?

Report queries on the warehouse (BigQuery, Redshift, Snowflake) are slow and expensive. How do you optimise them?

What is a surrogate key? Why does a dimension table not simply use the natural key from the source system?

What is the grain of a fact table? What types of fact tables exist and when do you use each?

What is a Slowly Changing Dimension (SCD)? How do SCD type 1 and type 2 differ?

A staging table has duplicate records because the source resends the same event several times. How do you deduplicate it with SQL before loading the main table?

How does an incremental load differ from a full load? How do you write an incremental load query using a watermark?

How do you write SQL that updates a dimension as SCD type 2 when new source data arrives?

What are the components of the Apache Spark architecture? What do the driver and executors do when a job runs?

How do transformations and actions differ in Spark? Why does Spark use lazy evaluation?

How do narrow and wide transformations differ in Spark? Why is the shuffle the most expensive step?

How do Hadoop MapReduce and Apache Spark differ? Why is Spark usually faster?

What is data skew in Spark? How do you detect and handle it?

How do batch processing and stream processing differ? When should you pick each?

When syncing data from an application database to a warehouse, how do query-based CDC (scanning by updated_at) and log-based CDC differ? Which do you choose?

In stream processing, how does event time differ from processing time? How do you handle late data with watermarks?

What is Apache Airflow used for? Explain DAGs, tasks and operators.

In Airflow, what do you use to make a DAG wait until something happens (a file lands on S3, another DAG finishes)? How do sensors work?

In Airflow, how do the data interval, logical date, catchup and retries work? How do you re-run data for past dates (backfill)?

What kinds of data quality checks go into a pipeline? Where should they be placed?

What is a data contract? What problem does it solve between the backend team and the data team?

A daily Spark job used to take 20 minutes and now takes 2 hours. How do you debug it?

How do RDD, DataFrame and Dataset differ in Spark? Which API should a new job use?

How do `cache()` and `persist()` differ in Spark? When should you cache a DataFrame?

How do `repartition()` and `coalesce()` differ in Spark? Which one should you use before writing files?

How do pandas and PySpark differ? When should a Data Engineer pick each one?

You need to process a 20GB CSV file with Python on a machine with only 8GB of RAM. How do you do it?

How does Spark choose a join strategy? How do broadcast hash join and sort-merge join differ?

What is the "small files problem" on a data lake? Why does it slow jobs down and how do you fix it?

Write SQL to find users who logged in on at least 3 consecutive days (the gaps and islands problem).

Given a clickstream table (user_id, event_time), write SQL that splits events into sessions, where a session ends after more than 30 minutes of inactivity.

How do Kimball, Inmon and Data Vault differ when designing a data warehouse? When do you pick each?

How do Delta Lake, Apache Iceberg and Apache Hudi differ? Which table format would you choose for a lakehouse, and on what criteria?

What are data lineage and a data catalog? How does Unity Catalog differ from the Hive Metastore?

How do you test a data pipeline before and after it goes to production?

Design a flow that moves events from Kafka into a data lake with Spark Structured Streaming so that no data is lost or duplicated when the job restarts.

A pipeline carries personal data (phone numbers, national ID numbers, emails). How do you design it to comply with personal data protection law, including when a user asks for their data to be deleted?

Design a pipeline that collects and analyses clickstream data (page views, clicks) for an e-commerce website producing about 10TB of events per day.

All Data Engineer interview questions

How do Data Lake and Data Warehouse differ? When should you use each?
What is Change Data Capture (CDC)? How does it work and what are its use cases?
What is Kafka Connect? What problem does it solve in data pipelines?
What is Schema Registry in the Kafka ecosystem? Why use Avro/Protobuf instead of JSON?
At-least-once vs at-most-once vs exactly-once in Kafka: differences and how to achieve each?
What is Amazon Redshift? When should you use Redshift instead of RDS? Explain columnar storage and distribution styles.
What are normalization and denormalization, and what is the trade-off?
What is the difference between a view and a materialized view?
How do OLTP and OLAP differ?
What is a window function and how does it differ from GROUP BY?
What is the difference between a star schema and a snowflake schema (dimensional modeling)?
Classic SQL exercise: how do you find the second-highest salary / top-N per group?
Kafka Streams vs Apache Flink — when do you pick which?
What is the logical execution order of a SELECT statement, and why does it differ from the written order?
How do you compute a running total with a window function, and how do `ROWS` and `RANGE` differ in the frame clause?
A 500M-row log table is only queried for the last 30 days — how would you partition it by time? What does partition pruning require?
What does a Data Engineer do? How does the role differ from a Data Analyst and a Data Scientist?
What is big data? How do you tell that a problem is actually a big data problem?
Why does a data lake usually store Parquet files instead of CSV? When does Avro fit better?
Design a pipeline that moves order data from a production MySQL database into a data warehouse for a daily revenue report. How would you do it?
What are the steps of ETL? Describe a basic ETL process.
How do ETL and ELT differ? When should you choose each one?
An ETL job fails halfway and the rerun duplicates data in the target table. How do you design the pipeline so reruns and backfills are safe?
What is a lakehouse? What does it add over an old-style data lake (files on HDFS/S3)?
What is the medallion architecture (bronze, silver, gold)? Why split data into several layers like this?
Report queries on the warehouse (BigQuery, Redshift, Snowflake) are slow and expensive. How do you optimise them?
What is a surrogate key? Why does a dimension table not simply use the natural key from the source system?
What is the grain of a fact table? What types of fact tables exist and when do you use each?
What is a Slowly Changing Dimension (SCD)? How do SCD type 1 and type 2 differ?
A staging table has duplicate records because the source resends the same event several times. How do you deduplicate it with SQL before loading the main table?
How does an incremental load differ from a full load? How do you write an incremental load query using a watermark?
How do you write SQL that updates a dimension as SCD type 2 when new source data arrives?
What are the components of the Apache Spark architecture? What do the driver and executors do when a job runs?
How do transformations and actions differ in Spark? Why does Spark use lazy evaluation?
How do narrow and wide transformations differ in Spark? Why is the shuffle the most expensive step?
How do Hadoop MapReduce and Apache Spark differ? Why is Spark usually faster?
What is data skew in Spark? How do you detect and handle it?
How do batch processing and stream processing differ? When should you pick each?
When syncing data from an application database to a warehouse, how do query-based CDC (scanning by updated_at) and log-based CDC differ? Which do you choose?
In stream processing, how does event time differ from processing time? How do you handle late data with watermarks?
What is Apache Airflow used for? Explain DAGs, tasks and operators.
In Airflow, what do you use to make a DAG wait until something happens (a file lands on S3, another DAG finishes)? How do sensors work?
In Airflow, how do the data interval, logical date, catchup and retries work? How do you re-run data for past dates (backfill)?
What kinds of data quality checks go into a pipeline? Where should they be placed?
What is a data contract? What problem does it solve between the backend team and the data team?
A daily Spark job used to take 20 minutes and now takes 2 hours. How do you debug it?
How do RDD, DataFrame and Dataset differ in Spark? Which API should a new job use?
How do `cache()` and `persist()` differ in Spark? When should you cache a DataFrame?
How do `repartition()` and `coalesce()` differ in Spark? Which one should you use before writing files?
How do pandas and PySpark differ? When should a Data Engineer pick each one?
You need to process a 20GB CSV file with Python on a machine with only 8GB of RAM. How do you do it?
How does Spark choose a join strategy? How do broadcast hash join and sort-merge join differ?
What is the "small files problem" on a data lake? Why does it slow jobs down and how do you fix it?
Write SQL to find users who logged in on at least 3 consecutive days (the gaps and islands problem).
Given a clickstream table (user_id, event_time), write SQL that splits events into sessions, where a session ends after more than 30 minutes of inactivity.
How do Kimball, Inmon and Data Vault differ when designing a data warehouse? When do you pick each?
How do Delta Lake, Apache Iceberg and Apache Hudi differ? Which table format would you choose for a lakehouse, and on what criteria?
What are data lineage and a data catalog? How does Unity Catalog differ from the Hive Metastore?
How do you test a data pipeline before and after it goes to production?
Design a flow that moves events from Kafka into a data lake with Spark Structured Streaming so that no data is lost or duplicated when the job restarts.
A pipeline carries personal data (phone numbers, national ID numbers, emails). How do you design it to comply with personal data protection law, including when a user asks for their data to be deleted?
Design a pipeline that collects and analyses clickstream data (page views, clicks) for an e-commerce website producing about 10TB of events per day.