Accueil›Blog›Test technique Airflow

Guide recrutement data

Test technique Airflow : ce qu'on evalue en entretien Data Engineer

Airflow orchestre des milliers de pipelines data. Concevoir un DAG robuste n'est pas la meme chose que lire un DAG existant.

Data Builder·Juin 2025·7 min de lecture·Data Engineer

Sommaire

Comprendre un DAG
Operateurs
Dependances
Production
TaskFlow API
Grille

Airflow orchestre des milliers de pipelines data en production. Concevoir un DAG robuste n'est pas la meme chose que lire un DAG existant.

1Comprendre un DAG

Question discriminante

Qu'est-ce qu'un DAG dans Airflow ? Pourquoi doit-il etre acyclique ?

with DAG(dag_id="pipeline", schedule="0 6 * * *",
         start_date=datetime(2025,1,1), catchup=False) as dag:
    task_a >> task_b >> [task_c, task_d] >> task_e

start_date et catchup — comportement du backfill automatique
DAG runs vs task instances

2Operateurs et hooks

Question discriminante

Difference entre un operateur et un hook ?

PythonOperator / BashOperator — les plus courants
Sensors — attendent une condition (fichier, table, temps)
Providers — packages AWS, GCP, dbt, Spark

3Dependances et scheduling

Question discriminante

Comment une tache attend-elle la completion d'un DAG externe ?

ExternalTaskSensor — poll regulier, consomme un slot
TriggerDagRunOperator — plus efficace mais couplage fort
XComs — passer des donnees entre taches

4Production : retry et alertes

Question discriminante

Un DAG critique echoue a 3h du matin. Qu'avez-vous mis en place ?

on_failure_callback — alerte Slack ou PagerDuty
SLA Miss — alerte si une tache depasse son temps maximum
Pools et queues — limiter la concurrence

5TaskFlow API et Dynamic tasks

Question discriminante Senior

Qu'est-ce que la TaskFlow API ? En quoi simplifie-t-elle les DAGs ?

@task decorator — syntaxe Python native Airflow 2.0+
Dynamic task mapping — taches dynamiques selon les donnees
Dataset-driven scheduling — declencher quand un dataset change

6Grille par niveau

Niveau	Maitrise attendue	Signal GO	NO-GO
Junior	Lire un DAG, PythonOperator, scheduling cron	Comprend catchup et backfill	Confond DAG et task
Confirme	Sensors, XComs, Variables, retry, providers	A configure des alertes, ExternalTaskSensor	N'a jamais configure de retry
Senior	TaskFlow API, dynamic tasks, SLAs, monitoring	Monitoring complet en prod, SLAs configures	Ne connait pas la TaskFlow API
Lead	Architecture multi-DAGs, dataset scheduling	A defini les standards de DAGs equipe	Ne peut pas expliquer les compromis de couplage

1DAG: structure and best practices

Key question

What is a DAG in Airflow? What are the rules for writing a well-designed DAG?

DAG (Directed Acyclic Graph) - task graph with dependencies. No cycles allowed.
No heavy code at parse time - imports, DB connections and API calls must be inside tasks, not at the DAG level.
Idempotency - a task re-run on the same date must produce the same result. Never use datetime.now() in tasks, use context['ds']
Atomic tasks - each task does one thing. No monolithic tasks that extract + transform + load in one step
catchup=False - by default. If catchup=True and start_date=2020, Airflow will create thousands of backfill runs on first deploy

2Main operators

Key question

What operators do you use most? What is the difference between PythonOperator and BashOperator?

PythonOperator - execute a Python function. The most common. With TaskFlow API, replaced by @task
BashOperator - execute a bash command. For shell scripts, dbt commands, CLI tools
KubernetesPodOperator - run a Docker container on Kubernetes. Perfect isolation, custom dependencies per task
BranchPythonOperator - conditional branching: return the task_id to execute next
Sensors - wait for an external condition (file in S3, REST API, time). Poke mode vs reschedule mode

3Dependencies and scheduling

Key question

How do you manage dependencies between tasks and between DAGs?

ExternalTaskSensor - wait for a task in another DAG to succeed. Avoids tight coupling between DAGs
TriggerDagRunOperator - explicitly trigger another DAG. More controllable than ExternalTaskSensor
Datasets (Airflow 2.4+) - data-driven scheduling: a DAG triggers when a dataset is updated by another DAG

4Production and monitoring

Key question

How do you manage Airflow in production? What monitoring do you put in place?

SLA Miss - alert if a task exceeds its expected duration. Configure sla= on each critical task
Callbacks - on_failure_callback, on_retry_callback for Slack/PagerDuty alerts
Zombie tasks - tasks stuck in running state without a heartbeat. The scheduler detects and restarts them
DB cleanup - the Airflow DB grows indefinitely without maintenance. Run airflow db clean regularly
Prometheus + Grafana - expose StatsD metrics: task duration, success/failure rate, scheduler lag

5Level assessment grid

Level	Mastery	GO signal	NO-GO
Junior	DAG structure, basic operators, CRON scheduling	Writes a functional DAG with PythonOperator, knows catchup	Has never written a DAG, confuses Airflow and Cron
Confirmed	TaskFlow API, XCom, Sensors, ExternalTaskSensor	Uses TaskFlow, manages inter-DAG dependencies, configures failure callbacks	Does not know what XCom is, has never used a Sensor
Senior	Executors, performance, CI/CD, production monitoring	Has deployed Airflow in production, managed performance issues, implemented CI/CD on DAGs	Cannot explain the difference between CeleryExecutor and KubernetesExecutor

Test technique Airflow : ce qu'on evalue en entretien Data Engineer

1Comprendre un DAG

2Operateurs et hooks

3Dependances et scheduling

4Production : retry et alertes

5TaskFlow API et Dynamic tasks

6Grille par niveau

1DAG: structure and best practices

2Main operators

3Dependencies and scheduling

4Production and monitoring

5Level assessment grid

Vous recrutez un profil data ?