Data Pipelines.

ETL, streaming, and data infrastructure. Covers 12 sub-skills.

• Data Pipelines · 1 of 3

  ETL & ELT

  Modern warehouse-centric pipelines push raw rows to the warehouse first and then run dbt-style SQL transforms on top. Which acronym names that order?

  AELTBETLCLTEDTEL

• Data Pipelines · 2 of 3

  Stream Processing

  What is backpressure in stream processing?

  AA rebalance that moves partitions off a consumer that has fallen behindBBuffering that absorbs a burst until the consumers have caught up againCA dead letter path for the events a consumer was unable to processDFlow control that slows the producers when consumers fall behind

• Data Pipelines · 3 of 3

  Batch Processing

  What is stage versus task boundary in Spark?

  AStage at partition; task at shuffle boundaryBStage at each action; task at each transformationCStage per executor; task per input file splitDStage at shuffle; task at partition boundary