Accueil›Blog›Test technique Azure Data Factory : pipelines, triggers, DataFlow

Guide recrutement data

Test technique Azure Data Factory : pipelines, triggers, DataFlow

Azure Data Factory est l orchestrateur ETL de référence dans l écosystème Microsoft. En entretien, on évalue la capacité à concevoir des pipelines robustes et à les intégrer dans une architecture Azure.

Data Builder·Juin 2025·6 min de lecture·Data Engineer

Sommaire

Concepts fondamentaux ADF
Pipelines et activités
Linked Services et Datasets
Triggers
Mapping Data Flow
ADF vs Synapse vs Fabric
Grille

1Concepts fondamentaux ADF

Question discriminante

Quels sont les 4 composants principaux d Azure Data Factory ?

Pipelines — workflows d activités, équivalent d un DAG Airflow
Activities — chaque étape : Copy Data, Data Flow, Stored Procedure, Web Activity, Databricks Notebook
Linked Services — connexions aux sources et destinations (SQL Server, Blob Storage, REST API, Snowflake...)
Datasets — représentation d une structure de données dans un Linked Service (table SQL, fichier CSV...)
Integration Runtime — moteur d exécution : Azure IR (cloud), Self-hosted IR (on-premise), SSIS IR

2Pipelines : orchestration des activités

Question discriminante

Comment gérez-vous les dépendances entre activités dans ADF ? Et les erreurs ?

// Structure d un pipeline ADF typique (JSON ARM template)
{
  'name': 'pipeline_ingestion_ventes',
  'activities': [
    {
      'name': 'CopyFromSource',
      'type': 'Copy',
      'dependsOn': [],
      'inputs': [{'referenceName': 'SourceDataset'}],
      'outputs': [{'referenceName': 'StagingDataset'}]
    },
    {
      'name': 'TransformData',
      'type': 'DatabricksNotebook',
      'dependsOn': [{
        'activity': 'CopyFromSource',
        'dependencyConditions': ['Succeeded']  // ou Failed, Skipped, Completed
      }]
    },
    {
      'name': 'OnFailure_Alert',
      'type': 'WebActivity',
      'dependsOn': [{
        'activity': 'TransformData',
        'dependencyConditions': ['Failed']
      }]
    }
  ]
}

dependencyConditions — Succeeded, Failed, Skipped, Completed. Permet de créer des branches conditionnelles
Try-Catch — activer une alerte en cas d échec via WebActivity (webhook Slack/Teams)

3Linked Services et paramétrage

Question discriminante

Comment évitez-vous de dupliquer vos Linked Services pour DEV/PROD ?

Parameters — paramétrer les Linked Services avec des variables d environnement
Azure Key Vault — stocker les credentials dans Key Vault, ADF y accède via Managed Identity. Jamais de mot de passe en dur
Global Parameters — paramètres partagés par tous les pipelines (environnement, date de référence)
Managed Identity — ADF s authentifie sur les services Azure sans stocker de credentials

4Triggers : déclencher les pipelines

Question discriminante

Quels types de triggers ADF connaissez-vous ? Dans quel cas utilisez-vous chacun ?

Trigger	Déclenchement	Cas d usage
Schedule	CRON planifié	Rafraîchissement quotidien, rapports hebdomadaires
Tumbling Window	Fenêtres fixes, backfill possible	Traitement par tranches horaires, avec retry et dépendances
Event-based	Arrivée d un fichier dans Blob Storage	Ingestion déclenchée par l arrivée de fichiers partenaires
Manual	Déclenchement manuel ou via API REST	Tests, reprises, triggers depuis une autre application

5Mapping Data Flow : transformation visuelle

Question discriminante

Quand utilisez-vous Mapping Data Flow plutôt qu une activité Databricks ?

Mapping Data Flow — transformation de données en mode no-code/low-code dans ADF. S exécute sur Spark managé
Cas d usage — transformations standard (filtres, jointures, pivots, aggrégations) sans expertise Spark
Limites — moins flexible qu un notebook Databricks, plus cher à l exécution que du SQL natif
Databricks Activity — préférer pour la logique complexe, le ML, ou quand l équipe maîtrise Python/Spark
Stored Procedure — préférer pour les transformations SQL natives dans Azure SQL ou Synapse

6ADF vs Azure Synapse vs Microsoft Fabric

Question discriminante

Comment positionnez-vous ADF, Synapse et Fabric ?

Service	Rôle	Idéal pour
Azure Data Factory	Orchestration ETL/ELT pure	Ingestion et mouvement de données, intégration d outils tiers
Azure Synapse	Analytics + intégration + Spark	SQL Analytics + pipelines Spark dans une interface unifiée
Microsoft Fabric	Plateforme unifiée Microsoft	Nouvelle stack end-to-end (Power BI + Synapse + ADF) sur OneLake

Tendance 2025 : Microsoft pousse vers Fabric comme plateforme unifiée. ADF reste pertinent pour les intégrations avec des systèmes non-Microsoft.

# Paramétrage d'un pipeline ADF (éviter les duplications)
# pipeline_copy_table.json
{
  "parameters": {
    "source_table": {"type": "String"},
    "start_date": {"type": "String", "defaultValue": "@utcnow()"},
    "container": {"type": "String", "defaultValue": "raw"}
  },
  "activities": [{
    "name": "CopyData",
    "type": "Copy",
    "typeProperties": {
      "source": {
        "type": "SqlServerSource",
        "sqlReaderQuery": "SELECT * FROM @{pipeline().parameters.source_table} WHERE updated_at >= '@{pipeline().parameters.start_date}'"
      },
      "sink": {
        "type": "ParquetSink",
        "storeSettings": {"type": "AzureBlobFSWriteSettings"}
      }
    }
  }]
}

Integration Runtime self-hosted — pour les sources on-premise (SQL Server datacenter). Installer l'IR sur un serveur Windows avec accès réseau à la source
CI/CD ADF — connecter ADF à Git Azure DevOps. Développer sur feature branch, merger vers main déclenche le déploiement automatique via ARM templates
Data Flows — transformations visuelles Spark-backed dans ADF. Éviter pour la logique complexe : préférer Databricks ou dbt pour la maintenabilité
Triggers — Schedule, Tumbling Window (rattrapages possibles), Event-based (déclenchement sur création de fichier dans ADLS)
Cost control — les activités Lookup et ForEach avec beaucoup d'itérations coûtent cher. Préférer les Copy Activities en batch plutôt que ligne par ligne

# Pipeline ADF parametrable
# pipeline_copy_table.json
{
  "parameters": {
    "source_table": {"type": "String"},
    "start_date": {"type": "String"},
    "container": {"type": "String", "defaultValue": "raw"}
  },
  "activities": [{
    "name": "CopyData",
    "type": "Copy",
    "typeProperties": {
      "source": {
        "type": "SqlServerSource",
        "sqlReaderQuery": "SELECT * FROM @{pipeline().parameters.source_table} WHERE updated_at >= '@{pipeline().parameters.start_date}'"
      },
      "sink": {"type": "ParquetSink"}
    }
  }]
}

Integration Runtime self-hosted - pour les sources on-premise (SQL Server datacenter). Installer l IR sur un serveur Windows avec acces reseau a la source
CI/CD ADF - connecter ADF a Git Azure DevOps. Merger vers main declenche le deploiement automatique via ARM templates
Triggers Event-based - declencher un pipeline sur creation de fichier dans ADLS Gen2. Remplace le polling par des evenements
Data Flows - transformations visuelles Spark-backed dans ADF. Pour la logique complexe, preferer Databricks ou dbt pour la maintenabilite
Cost control - les activites Lookup et ForEach avec beaucoup d iterations coutent cher. Preferer les Copy Activities en batch

7Grille par niveau

Niveau	Maitrise	Signal GO	NO-GO
Confirmé	Pipelines, Copy Activity, Schedule Trigger, Key Vault	A déployé un pipeline Copy + transformation, utilise Key Vault	Stocke les mots de passe en dur dans les Linked Services
Senior	Tumbling Window, Event Trigger, Mapping Data Flow, ARM templates	A configuré un Event Trigger, déploie via ARM/CI-CD, justifie ADF vs Databricks	Ne sait pas la différence entre Schedule et Tumbling Window

1ADF Core Concepts

Discriminating question

What are the 4 main components of Azure Data Factory?

Pipelines — activity workflows, equivalent to an Airflow DAG
Activities — each step: Copy Data, Data Flow, Stored Procedure, Web Activity, Databricks Notebook
Linked Services — connections to sources and destinations (SQL Server, Blob Storage, REST API, Snowflake...)
Datasets — representation of a data structure in a Linked Service (SQL table, CSV file...)
Integration Runtime — execution engine: Azure IR (cloud), Self-hosted IR (on-premise), SSIS IR

2Pipelines: activity orchestration

Discriminating question

How do you manage dependencies between activities in ADF? And errors?

// Structure d un pipeline ADF typique (JSON ARM template)
{
  'name': 'pipeline_ingestion_ventes',
  'activities': [
    {
      'name': 'CopyFromSource',
      'type': 'Copy',
      'dependsOn': [],
      'inputs': [{'referenceName': 'SourceDataset'}],
      'outputs': [{'referenceName': 'StagingDataset'}]
    },
    {
      'name': 'TransformData',
      'type': 'DatabricksNotebook',
      'dependsOn': [{
        'activity': 'CopyFromSource',
        'dependencyConditions': ['Succeeded']  // ou Failed, Skipped, Completed
      }]
    },
    {
      'name': 'OnFailure_Alert',
      'type': 'WebActivity',
      'dependsOn': [{
        'activity': 'TransformData',
        'dependencyConditions': ['Failed']
      }]
    }
  ]
}

dependencyConditions — Succeeded, Failed, Skipped, Completed. Allows creating conditional branches
Try-Catch — trigger an alert on failure via WebActivity (Slack/Teams webhook)

3Linked Services and parameterization

Discriminating question

How do you avoid duplicating your Linked Services for DEV/PROD?

Parameters — parameterize Linked Services with environment variables
Azure Key Vault — store credentials in Key Vault, ADF accesses them via Managed Identity. Never hardcode passwords
Global Parameters — parameters shared across all pipelines (environment, reference date)
Managed Identity — ADF authenticates to Azure services without storing credentials

4Triggers: launching pipelines

Discriminating question

What types of ADF triggers do you know? In which case do you use each one?

Trigger	Activation	Use case
Schedule	Scheduled CRON	Daily refresh, weekly reports
Tumbling Window	Fixed windows, backfill possible	Processing by hourly slices, with retry and dependencies
Event-based	File arrival in Blob Storage	Ingestion triggered by arrival of partner files
Manual	Manual trigger or via REST API	Tests, reruns, triggers from another application

5Mapping Data Flow: visual transformation

Discriminating question

When do you use Mapping Data Flow rather than a Databricks activity?

Mapping Data Flow — data transformation in no-code/low-code mode in ADF. Runs on managed Spark
Use cases — standard transformations (filters, joins, pivots, aggregations) without Spark expertise
Limitations — less flexible than a Databricks notebook, more expensive to run than native SQL
Databricks Activity — prefer for complex logic, ML, or when the team is proficient in Python/Spark
Stored Procedure — prefer for native SQL transformations in Azure SQL or Synapse

6ADF vs Azure Synapse vs Microsoft Fabric

Discriminating question

How do you position ADF, Synapse, and Fabric?

Service	Role	Ideal for
Azure Data Factory	Pure ETL/ELT orchestration	Data ingestion and movement, third-party tool integration
Azure Synapse	Analytics + integration + Spark	SQL Analytics + Spark pipelines in a unified interface
Microsoft Fabric	Unified Microsoft platform	New end-to-end stack (Power BI + Synapse + ADF) on OneLake

2025 trend: Microsoft is pushing toward Fabric as a unified platform. ADF remains relevant for integrations with non-Microsoft systems.

# Paramétrage d'un pipeline ADF (éviter les duplications)
# pipeline_copy_table.json
{
  "parameters": {
    "source_table": {"type": "String"},
    "start_date": {"type": "String", "defaultValue": "@utcnow()"},
    "container": {"type": "String", "defaultValue": "raw"}
  },
  "activities": [{
    "name": "CopyData",
    "type": "Copy",
    "typeProperties": {
      "source": {
        "type": "SqlServerSource",
        "sqlReaderQuery": "SELECT * FROM @{pipeline().parameters.source_table} WHERE updated_at >= '@{pipeline().parameters.start_date}'"
      },
      "sink": {
        "type": "ParquetSink",
        "storeSettings": {"type": "AzureBlobFSWriteSettings"}
      }
    }
  }]
}

Self-hosted Integration Runtime — for on-premise sources (datacenter SQL Server). Install the IR on a Windows server with network access to the source
ADF CI/CD — connect ADF to Azure DevOps Git. Developing on a feature branch, merging to main triggers automatic deployment via ARM templates
Data Flows — Spark-backed visual transformations in ADF. Avoid for complex logic: prefer Databricks or dbt for maintainability
Triggers — Schedule, Tumbling Window (backfills possible), Event-based (triggered on file creation in ADLS)
Cost control — Lookup and ForEach activities with many iterations are expensive. Prefer batch Copy Activities rather than row by row

# Pipeline ADF parametrable
# pipeline_copy_table.json
{
  "parameters": {
    "source_table": {"type": "String"},
    "start_date": {"type": "String"},
    "container": {"type": "String", "defaultValue": "raw"}
  },
  "activities": [{
    "name": "CopyData",
    "type": "Copy",
    "typeProperties": {
      "source": {
        "type": "SqlServerSource",
        "sqlReaderQuery": "SELECT * FROM @{pipeline().parameters.source_table} WHERE updated_at >= '@{pipeline().parameters.start_date}'"
      },
      "sink": {"type": "ParquetSink"}
    }
  }]
}

Self-hosted Integration Runtime - for on-premise sources (datacenter SQL Server). Install the IR on a Windows server with network access to the source
ADF CI/CD - connect ADF to Azure DevOps Git. Merging to main triggers automatic deployment via ARM templates
Event-based Triggers - trigger a pipeline on file creation in ADLS Gen2. Replaces polling with events
Data Flows - Spark-backed visual transformations in ADF. For complex logic, prefer Databricks or dbt for maintainability
Cost control - Lookup and ForEach activities with many iterations are expensive. Prefer batch Copy Activities

7Level grid

Level	Proficiency	GO signal	NO-GO
Confirmed	Pipelines, Copy Activity, Schedule Trigger, Key Vault	Has deployed a Copy + transformation pipeline, uses Key Vault	Hardcodes passwords in Linked Services
Senior	Tumbling Window, Event Trigger, Mapping Data Flow, ARM templates	Has configured an Event Trigger, deploys via ARM/CI-CD, justifies ADF vs Databricks	Does not know the difference between Schedule and Tumbling Window

Vous recrutez un Data Engineer Azure ?

Premier entretien gratuit. Rapport GO/NO-GO sous 48h.

Tester gratuitement Reserver un appel