Accueil›Blog›Test technique Prefect : orchestration moderne vs Airflow

Guide recrutement data

Test technique Prefect : orchestration moderne vs Airflow

Prefect est une alternative moderne à Airflow avec une meilleure developer experience. En entretien, on évalue la capacité à choisir entre les orchestrateurs et à architecturer des pipelines robustes.

Data Builder·Juin 2025·6 min de lecture·Data Engineer

Sommaire

Pourquoi Prefect
Flows et tasks
Deployments et work pools
Gestion des erreurs
Airflow vs Prefect vs Dagster
Monitoring et observabilité
Grille

1Pourquoi Prefect est différent d'Airflow

Question discriminante

Quelle est la principale différence architecturale entre Prefect et Airflow ?

Airflow — DAG défini statiquement en Python, scheduler central qui planifie, workers qui exécutent. DAG doit être parseable sans erreur au chargement, même sans exécution
Prefect — flows définis en Python pur (décorations @flow/@task). L'exécution est découplée du scheduling. Prefect Cloud stocke les métadonnées, les workers exécutent localement
Avantages Prefect — moins de boilerplate, retry natif par task, state management avancé, testable localement sans infrastructure, inputs/outputs typés
Avantages Airflow — écosystème mature (400+ providers), plus répandu en entreprise, meilleure intégration avec les outils existants
Dagster — asset-centric (les données, pas les tâches). Idéal pour les équipes qui veulent raisonner en termes de data products

# Test unitaire d'un flow Prefect (impossible avec Airflow sans mocking lourd)
from prefect import flow, task

@task
def extract(url: str) -> list:
    return requests.get(url).json()

@flow
def pipeline(url: str = "https://api.example.com"):
    data = extract(url)
    return data

# Test simple en local - aucune infrastructure nécessaire
if __name__ == "__main__":
    result = pipeline("https://api.example.com/orders")
    print(result)

2Flows et tasks : la syntaxe Prefect

Question discriminante

Comment convertissez-vous un script Python existant en flow Prefect ?

from prefect import flow, task
from prefect.tasks import task_input_hash
from datetime import timedelta

@task(
    retries=3,
    retry_delay_seconds=60,
    cache_key_fn=task_input_hash,
    cache_expiration=timedelta(hours=1),
    tags=["extraction", "api"]
)
def extract_data(source_url: str) -> list:
    response = requests.get(source_url, timeout=30)
    response.raise_for_status()
    return response.json()

@task(log_prints=True)
def transform(data: list) -> pd.DataFrame:
    df = pd.DataFrame(data)
    df = df[df['amount'] > 0]
    print(f"{len(df)} lignes après filtrage")
    return df

@task
def load(df: pd.DataFrame, table: str) -> int:
    df.to_sql(table, engine, if_exists='append', index=False)
    return len(df)

@flow(name='pipeline-ventes', log_prints=True)
def pipeline(source: str = 'https://api.example.com/orders'):
    data = extract_data(source)
    df = transform(data)
    n = load(df, 'orders')
    print(f'{n} lignes chargées')
    return n

Cache natif — task_input_hash met en cache le résultat selon les inputs. Si les inputs n'ont pas changé, la task est skippée
Tags — organiser et filtrer les runs dans l'UI. Utile pour les alertes par équipe
Subflows — un flow peut appeler un autre flow. Permet la composition de pipelines complexes

3Deployments et work pools

Question discriminante

Qu'est-ce qu'un Deployment Prefect ? Comment planifier l'exécution ?

from prefect.deployments import Deployment
from prefect.server.schemas.schedules import CronSchedule

# Créer un déploiement avec CRON
deployment = Deployment.build_from_flow(
    flow=pipeline,
    name='pipeline-ventes-quotidien',
    schedule=CronSchedule(cron='0 6 * * *', timezone='Europe/Paris'),
    work_pool_name='mon-pool-gcp',
    parameters={'source': 'https://api.example.com/orders'},
    tags=['production', 'ventes']
)
deployment.apply()

# Types de work pools :
# Process      → exécution locale (dev/test)
# Docker       → conteneur Docker isolé
# Kubernetes   → pod K8s avec autoscaling
# Cloud Run    → GCP serverless (facturation à l'usage)
# ECS          → AWS serverless

Work pool — définit l'infrastructure d'exécution. Un même flow peut être déployé sur plusieurs pools (dev, staging, prod)
Worker — process qui interroge le pool et exécute les flows. Tourne en local ou dans un cluster
Prefect Cloud vs self-hosted — Prefect Cloud (SaaS) gère le serveur. Self-hosted : déployer prefect server soi-même

4Gestion des erreurs et retry

Question discriminante

Comment gérez-vous les erreurs dans Prefect ? En quoi est-ce plus puissant qu'Airflow ?

from prefect.tasks import exponential_backoff

@task(
    retries=3,
    retry_delay_seconds=exponential_backoff(backoff_factor=10),  # 10s, 100s, 1000s
    retry_jitter_factor=0.5  # évite les retry storms
)
def call_external_api(url: str) -> dict:
    ...

# State handlers : réagir aux changements d'état
from prefect import flow
from prefect.states import Failed

def notify_on_failure(flow, flow_run, state):
    if isinstance(state, Failed):
        send_slack_alert(f"Flow {flow_run.name} a échoué : {state.message}")

@flow(on_failure=[notify_on_failure])
def pipeline():
    ...

Exponential backoff — évite de surcharger une API en panne en espaçant exponentiellement les retries
State handlers — déclencher des actions (Slack, PagerDuty, email) selon l'état du flow ou de la task
Resume depuis un checkpoint — reprendre un flow échoué depuis la dernière task réussie sans tout relancer
Automations Prefect Cloud — règles événementielles : si flow échoue 3 fois → créer un incident Opsgenie

5Airflow vs Prefect vs Dagster

Question discriminante

Comment choisissez-vous votre orchestrateur selon le contexte ?

	Airflow	Prefect	Dagster
Paradigme	DAG-centric (tâches)	Flow-centric (Python pur)	Asset-centric (données)
Courbe apprentissage	Élevée	Faible	Moyenne
Écosystème	Mature, 400+ providers	Croissant	Croissant
Developer experience	Moyen	Excellent	Excellent
Data assets	Non natif	Partiel	Natif
Test local	Difficile	Très facile	Facile
Idéal pour	Équipes existantes, beaucoup de providers	Nouveaux projets Python-first	Teams data-asset oriented

6Monitoring et observabilité

Question discriminante

Comment monitorez-vous vos flows Prefect en production ?

Prefect Cloud dashboard — vue centralisée de tous les runs, filtrage par flow/tag/état, logs centralisés avec recherche full-text
Automations — déclencher des actions (Slack, email, webhook) quand un flow échoue, dépasse son SLA ou réussit trop rarement
Work pool monitoring — surveiller la santé des workers (queue depth, dernière activité)
Intégration Prometheus/Grafana — exporter les métriques Prefect (runs, latence, taux d'échec) via un exporter dédié
Alertes SLA — configurer des alertes si un flow tarde plus de N minutes après son heure planifiée

Work pool - definit l infrastructure d execution. Un meme flow peut etre deploye sur plusieurs pools (dev, staging, prod)
Automations Prefect Cloud - regles evenementielles : si flow echoue 3 fois -> creer un incident Opsgenie. Remplace les scripts de monitoring manuels
State handlers - declencher des actions (Slack, PagerDuty, email) selon l etat du flow ou de la task. on_failure, on_completion, on_cancellation
SLA monitoring - alertes si un flow tarde plus de N minutes apres son heure planifiee. Configurable par deployment
prefect serve vs deployment - prefect serve : lancer directement depuis Python (dev, pas de work pool). Deployment : config persistante avec scheduling CRON (prod)

7Grille par niveau

Niveau	Maîtrise	Signal GO	NO-GO
Confirmé	Flows et tasks, déploiements, retry basique, scheduling CRON	A transformé un script Python en flow Prefect, sait déployer avec un CRON et un work pool	N'a jamais utilisé d'orchestrateur, confond Prefect et Airflow
Senior	Work pools, state handlers, automations, comparaison orchestrateurs	Justifie le choix d'orchestrateur selon le contexte, a configuré des automations et alertes SLA	Ne sait pas expliquer la différence architecturale entre Airflow et Prefect

1Why Prefect is different from Airflow

Discriminating question

What is the main architectural difference between Prefect and Airflow?

Airflow — DAG statically defined in Python, central scheduler that plans, workers that execute. DAG must be parseable without error at load time, even without execution
Prefect — flows defined in pure Python (@flow/@task decorators). Execution is decoupled from scheduling. Prefect Cloud stores metadata, workers execute locally
Prefect advantages — less boilerplate, native retry per task, advanced state management, locally testable without infrastructure, typed inputs/outputs
Airflow advantages — mature ecosystem (400+ providers), more widespread in enterprise, better integration with existing tools
Dagster — asset-centric (the data, not the tasks). Ideal for teams that want to reason in terms of data products

# Unit test of a Prefect flow (impossible with Airflow without heavy mocking)
from prefect import flow, task

@task
def extract(url: str) -> list:
    return requests.get(url).json()

@flow
def pipeline(url: str = "https://api.example.com"):
    data = extract(url)
    return data

# Simple local test - no infrastructure needed
if __name__ == "__main__":
    result = pipeline("https://api.example.com/orders")
    print(result)

2Flows and tasks: Prefect syntax

Discriminating question

How do you convert an existing Python script into a Prefect flow?

from prefect import flow, task
from prefect.tasks import task_input_hash
from datetime import timedelta

@task(
    retries=3,
    retry_delay_seconds=60,
    cache_key_fn=task_input_hash,
    cache_expiration=timedelta(hours=1),
    tags=["extraction", "api"]
)
def extract_data(source_url: str) -> list:
    response = requests.get(source_url, timeout=30)
    response.raise_for_status()
    return response.json()

@task(log_prints=True)
def transform(data: list) -> pd.DataFrame:
    df = pd.DataFrame(data)
    df = df[df['amount'] > 0]
    print(f"{len(df)} rows after filtering")
    return df

@task
def load(df: pd.DataFrame, table: str) -> int:
    df.to_sql(table, engine, if_exists='append', index=False)
    return len(df)

@flow(name='sales-pipeline', log_prints=True)
def pipeline(source: str = 'https://api.example.com/orders'):
    data = extract_data(source)
    df = transform(data)
    n = load(df, 'orders')
    print(f'{n} rows loaded')
    return n

Native cache — task_input_hash caches the result based on inputs. If inputs haven't changed, the task is skipped
Tags — organize and filter runs in the UI. Useful for team-based alerts
Subflows — a flow can call another flow. Allows composition of complex pipelines

3Deployments and work pools

Discriminating question

What is a Prefect Deployment? How do you schedule execution?

from prefect.deployments import Deployment
from prefect.server.schemas.schedules import CronSchedule

# Create a deployment with CRON
deployment = Deployment.build_from_flow(
    flow=pipeline,
    name='daily-sales-pipeline',
    schedule=CronSchedule(cron='0 6 * * *', timezone='Europe/Paris'),
    work_pool_name='my-gcp-pool',
    parameters={'source': 'https://api.example.com/orders'},
    tags=['production', 'sales']
)
deployment.apply()

# Work pool types:
# Process      → local execution (dev/test)
# Docker       → isolated Docker container
# Kubernetes   → K8s pod with autoscaling
# Cloud Run    → GCP serverless (pay-per-use)
# ECS          → AWS serverless

Work pool — defines the execution infrastructure. The same flow can be deployed on multiple pools (dev, staging, prod)
Worker — process that polls the pool and executes flows. Runs locally or in a cluster
Prefect Cloud vs self-hosted — Prefect Cloud (SaaS) manages the server. Self-hosted: deploy prefect server yourself

4Error handling and retry

Discriminating question

How do you handle errors in Prefect? Why is it more powerful than Airflow?

from prefect.tasks import exponential_backoff

@task(
    retries=3,
    retry_delay_seconds=exponential_backoff(backoff_factor=10),  # 10s, 100s, 1000s
    retry_jitter_factor=0.5  # avoids retry storms
)
def call_external_api(url: str) -> dict:
    ...

# State handlers: react to state changes
from prefect import flow
from prefect.states import Failed

def notify_on_failure(flow, flow_run, state):
    if isinstance(state, Failed):
        send_slack_alert(f"Flow {flow_run.name} failed: {state.message}")

@flow(on_failure=[notify_on_failure])
def pipeline():
    ...

Exponential backoff — avoids overloading a failing API by exponentially spacing out retries
State handlers — trigger actions (Slack, PagerDuty, email) based on flow or task state
Resume from checkpoint — restart a failed flow from the last successful task without rerunning everything
Prefect Cloud automations — event-driven rules: if flow fails 3 times → create an Opsgenie incident

5Airflow vs Prefect vs Dagster

Discriminating question

How do you choose your orchestrator based on context?

	Airflow	Prefect	Dagster
Paradigm	DAG-centric (tasks)	Flow-centric (pure Python)	Asset-centric (data)
Learning curve	High	Low	Medium
Ecosystem	Mature, 400+ providers	Growing	Growing
Developer experience	Average	Excellent	Excellent
Data assets	Not native	Partial	Native
Local testing	Difficult	Very easy	Easy
Ideal for	Existing teams, many providers	New Python-first projects	Data-asset oriented teams

6Monitoring and observability

Discriminating question

How do you monitor your Prefect flows in production?

Prefect Cloud dashboard — centralized view of all runs, filtering by flow/tag/state, centralized logs with full-text search
Automations — trigger actions (Slack, email, webhook) when a flow fails, exceeds its SLA, or succeeds too rarely
Work pool monitoring — monitor worker health (queue depth, last activity)
Prometheus/Grafana integration — export Prefect metrics (runs, latency, failure rate) via a dedicated exporter
SLA alerts — configure alerts if a flow is more than N minutes late after its scheduled time

Work pool - defines the execution infrastructure. The same flow can be deployed on multiple pools (dev, staging, prod)
Prefect Cloud automations - event-driven rules: if flow fails 3 times -> create an Opsgenie incident. Replaces manual monitoring scripts
State handlers - trigger actions (Slack, PagerDuty, email) based on flow or task state. on_failure, on_completion, on_cancellation
SLA monitoring - alerts if a flow is more than N minutes late after its scheduled time. Configurable per deployment
prefect serve vs deployment - prefect serve: launch directly from Python (dev, no work pool). Deployment: persistent config with CRON scheduling (prod)

7Level grid

Level	Mastery	GO signal	NO-GO
Mid-level	Flows and tasks, deployments, basic retry, CRON scheduling	Has converted a Python script into a Prefect flow, knows how to deploy with a CRON and a work pool	Has never used an orchestrator, confuses Prefect and Airflow
Senior	Work pools, state handlers, automations, orchestrator comparison	Justifies orchestrator choice based on context, has configured automations and SLA alerts	Cannot explain the architectural difference between Airflow and Prefect

Vous recrutez un Data Engineer ?

Premier entretien gratuit. Rapport GO/NO-GO sous 48h.

Tester gratuitement Reserver un appel