Accueil›Blog›Test technique GitHub Actions pour la data : CI/CD pipelines data

Guide recrutement data

Test technique GitHub Actions pour la data : CI/CD pipelines data

GitHub Actions est devenu le standard CI/CD pour les projets data. En entretien Senior, on evalue la capacite a automatiser le cycle de vie complet d un pipeline data.

Data Builder·Juin 2025·6 min de lecture·Data Engineer · Analytics Engineer

Sommaire

Concepts GitHub Actions
CI/CD pour dbt
Tests de qualite automatises
Build et push Docker
Gestion des secrets
Matrix strategy
Grille

1Concepts fondamentaux

Question discriminante

Quelle est la difference entre un job et une step dans GitHub Actions ? Et entre on push et on pull_request ?

# Structure d un workflow GitHub Actions data
name: Data Pipeline CI

on:
  pull_request:
    branches: [main]
    paths:
      - 'dbt/**'
      - 'tests/**'
  push:
    branches: [main]

jobs:
  dbt_test:          # un job = un runner independant
    runs-on: ubuntu-latest
    steps:           # steps = etapes sequentielles dans un job
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install dbt-bigquery great-expectations

Jobs paralleles — les jobs s executent en parallele par defaut. Utiliser needs: pour les dependances
on: pull_request — declenche sur chaque PR. Idéal pour les tests
on: push main — declenche au merge. Idéal pour le deploiement

2CI/CD pour dbt : slim CI

Question discriminante

Qu est-ce que le slim CI dbt ? Pourquoi est-il important ?

jobs:
  dbt_ci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: dbt deps
        run: dbt deps
        
      # Slim CI : ne tester que les modeles modifies
      - name: dbt build (slim)
        run: |
          dbt build \
            --select state:modified+ \
            --defer \
            --state ./prod_artifacts \
            --target ci
        env:
          DBT_PROFILES_DIR: .
          BIGQUERY_KEYFILE: ${{ secrets.GCP_SA_KEY }}
          
      # Upload les artefacts pour le prochain run
      - uses: actions/upload-artifact@v4
        with:
          name: dbt-artifacts
          path: target/

Slim CI — ne rebuild et ne teste que les modeles modifies et leurs descendants. Economise 80%+ du temps de CI
--defer — utiliser les relations de production pour les modeles non modifies
--state — comparer avec les artefacts du dernier run de production

3Tests qualite automatises

Question discriminante

Comment integrez-vous Great Expectations dans un pipeline CI ?

  - name: Great Expectations validation
    run: |
      great_expectations checkpoint run my_checkpoint
    env:
      GE_CLOUD_ACCESS_TOKEN: ${{ secrets.GE_TOKEN }}

  - name: Commentaire PR avec resultats
    uses: actions/github-script@v7
    if: failure()
    with:
      script: |
        github.rest.issues.createComment({
          issue_number: context.issue.number,
          body: '❌ Data quality checks failed. See logs for details.'
        })

4Build Docker et push vers registry

Question discriminante

Comment buildez-vous et publiez-vous une image Docker d API ML dans un pipeline CI ?

  build-push:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: docker/login-action@v3
        with:
          registry: europe-west1-docker.pkg.dev
          username: _json_key
          password: ${{ secrets.GCP_SA_KEY }}
          
      - uses: docker/build-push-action@v5
        with:
          push: ${{ github.ref == 'refs/heads/main' }}
          tags: europe-west1-docker.pkg.dev/mon-projet/api/scoring:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

5Gestion des secrets

Question discriminante

Comment gerez-vous les credentials dans GitHub Actions sans les exposer ?

GitHub Secrets — stocker dans Settings > Secrets. Jamais en clair dans le workflow
OIDC — Workload Identity Federation : GitHub Actions s authentifie directement sur GCP/AWS sans stocker de cles JSON. Meilleure pratique 2025
Environment secrets — secrets specifiques a un environnement (production vs staging)
Jamais dans les logs — GitHub masque automatiquement les secrets dans les logs

6Matrix strategy : tester sur plusieurs versions

jobs:
  test:
    strategy:
      matrix:
        python-version: ['3.10', '3.11', '3.12']
        dbt-adapter: ['dbt-bigquery', 'dbt-snowflake']
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - run: pip install ${{ matrix.dbt-adapter }}

# Workflow complet: lint + test + deploy
name: Data Pipeline CI
on: [push, pull_request]

jobs:
  quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Python lint
        run: |
          pip install ruff
          ruff check src/
      - name: Unit tests
        run: pytest tests/ -v --cov=src
      - name: Integration test (staging)
        if: github.ref == 'refs/heads/main'
        run: python src/pipeline.py --env staging
        env:
          DB_PASSWORD: ${{ secrets.STAGING_DB_PASSWORD }}
      - name: Deploy prod
        if: github.ref == 'refs/heads/main' && success()
        run: python deploy.py --env production
        env:
          DB_PASSWORD: ${{ secrets.PROD_DB_PASSWORD }}

# .github/workflows/data_ci.yml
name: Data Pipeline CI/CD
on:
  push: {branches: [main]}
  pull_request: {branches: [main]}

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v4
        with: {python-version: '3.11'}
      - uses: actions/cache@v3
        with:
          path: ~/.cache/pip
          key: pip-${{ hashFiles('requirements.txt') }}
      - run: pip install -r requirements.txt
      - run: ruff check src/
      - run: pytest tests/ --cov=src --cov-report=xml

  dbt-check:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install dbt-snowflake
      - run: dbt deps --project-dir dbt/
      - run: dbt compile --project-dir dbt/ --profiles-dir dbt/
      - run: dbt test --select state:modified+ --defer --state ./prod-artifacts
        env:
          SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
          SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}

  deploy:
    needs: dbt-check
    if: github.ref == 'refs/heads/main'
    environment: production
    runs-on: ubuntu-latest
    steps:
      - run: python deploy.py --env production
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

Cache pip - actions/cache sur pip reduit le temps de CI de 3min a 30s. Cle de cache basee sur le hash du fichier requirements.txt
Secrets GitHub - Settings -> Secrets -> Actions. Jamais de credentials en dur dans le YAML. Utiliser des environments pour les secrets de prod avec protection
Matrix builds - tester sur Python 3.10, 3.11, 3.12 en parallele avec strategy.matrix. Decouvrir les incompatibilites de version avant la prod
GitHub Environments - configurer une approbation manuelle obligatoire avant le deploy en production. Tracabilite complete de qui a approuve quoi
Self-hosted runners - executer les workflows sur vos propres machines (acces VPN, GPU, ressources puissantes). Utile pour les jobs de training ML ou d acces aux donnees sensibles

7Grille par niveau

Niveau	Maitrise	Signal GO	NO-GO
Confirme	Workflows basiques, tests automatises, secrets	A configure un workflow dbt CI avec tests, gere les secrets	Lance les tests manuellement, ne sait pas ce qu est un workflow
Senior	Slim CI dbt, OIDC, matrix, deploiement Docker	A mis en place le slim CI dbt, utilise OIDC au lieu des cles JSON	Stocke les cles GCP en clair, ne connait pas le slim CI

1Core Concepts

Discriminating question

What is the difference between a job and a step in GitHub Actions? And between on push and on pull_request?

# Structure d un workflow GitHub Actions data
name: Data Pipeline CI

on:
  pull_request:
    branches: [main]
    paths:
      - 'dbt/**'
      - 'tests/**'
  push:
    branches: [main]

jobs:
  dbt_test:          # un job = un runner independant
    runs-on: ubuntu-latest
    steps:           # steps = etapes sequentielles dans un job
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install dbt-bigquery great-expectations

Parallel jobs — jobs run in parallel by default. Use needs: for dependencies
on: pull_request — triggered on each PR. Ideal for testing
on: push main — triggered on merge. Ideal for deployment

2CI/CD for dbt: slim CI

Discriminating question

What is dbt slim CI? Why is it important?

jobs:
  dbt_ci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: dbt deps
        run: dbt deps
        
      # Slim CI : ne tester que les modeles modifies
      - name: dbt build (slim)
        run: |
          dbt build \
            --select state:modified+ \
            --defer \
            --state ./prod_artifacts \
            --target ci
        env:
          DBT_PROFILES_DIR: .
          BIGQUERY_KEYFILE: ${{ secrets.GCP_SA_KEY }}
          
      # Upload les artefacts pour le prochain run
      - uses: actions/upload-artifact@v4
        with:
          name: dbt-artifacts
          path: target/

Slim CI — only rebuilds and tests modified models and their descendants. Saves 80%+ of CI time
--defer — use production relations for unmodified models
--state — compare with artifacts from the last production run

3Automated quality tests

Discriminating question

How do you integrate Great Expectations into a CI pipeline?

  - name: Great Expectations validation
    run: |
      great_expectations checkpoint run my_checkpoint
    env:
      GE_CLOUD_ACCESS_TOKEN: ${{ secrets.GE_TOKEN }}

  - name: PR comment with results
    uses: actions/github-script@v7
    if: failure()
    with:
      script: |
        github.rest.issues.createComment({
          issue_number: context.issue.number,
          body: '❌ Data quality checks failed. See logs for details.'
        })

4Docker build and push to registry

Discriminating question

How do you build and publish an ML API Docker image in a CI pipeline?

  build-push:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: docker/login-action@v3
        with:
          registry: europe-west1-docker.pkg.dev
          username: _json_key
          password: ${{ secrets.GCP_SA_KEY }}
          
      - uses: docker/build-push-action@v5
        with:
          push: ${{ github.ref == 'refs/heads/main' }}
          tags: europe-west1-docker.pkg.dev/mon-projet/api/scoring:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

5Secrets management

Discriminating question

How do you manage credentials in GitHub Actions without exposing them?

GitHub Secrets — store in Settings > Secrets. Never in plaintext in the workflow
OIDC — Workload Identity Federation: GitHub Actions authenticates directly with GCP/AWS without storing JSON keys. Best practice 2025
Environment secrets — secrets specific to an environment (production vs staging)
Never in logs — GitHub automatically masks secrets in logs

6Matrix strategy: testing across multiple versions

jobs:
  test:
    strategy:
      matrix:
        python-version: ['3.10', '3.11', '3.12']
        dbt-adapter: ['dbt-bigquery', 'dbt-snowflake']
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - run: pip install ${{ matrix.dbt-adapter }}

# Workflow complet: lint + test + deploy
name: Data Pipeline CI
on: [push, pull_request]

jobs:
  quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Python lint
        run: |
          pip install ruff
          ruff check src/
      - name: Unit tests
        run: pytest tests/ -v --cov=src
      - name: Integration test (staging)
        if: github.ref == 'refs/heads/main'
        run: python src/pipeline.py --env staging
        env:
          DB_PASSWORD: ${{ secrets.STAGING_DB_PASSWORD }}
      - name: Deploy prod
        if: github.ref == 'refs/heads/main' && success()
        run: python deploy.py --env production
        env:
          DB_PASSWORD: ${{ secrets.PROD_DB_PASSWORD }}

# .github/workflows/data_ci.yml
name: Data Pipeline CI/CD
on:
  push: {branches: [main]}
  pull_request: {branches: [main]}

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v4
        with: {python-version: '3.11'}
      - uses: actions/cache@v3
        with:
          path: ~/.cache/pip
          key: pip-${{ hashFiles('requirements.txt') }}
      - run: pip install -r requirements.txt
      - run: ruff check src/
      - run: pytest tests/ --cov=src --cov-report=xml

  dbt-check:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install dbt-snowflake
      - run: dbt deps --project-dir dbt/
      - run: dbt compile --project-dir dbt/ --profiles-dir dbt/
      - run: dbt test --select state:modified+ --defer --state ./prod-artifacts
        env:
          SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
          SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}

  deploy:
    needs: dbt-check
    if: github.ref == 'refs/heads/main'
    environment: production
    runs-on: ubuntu-latest
    steps:
      - run: python deploy.py --env production
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

pip cache - actions/cache on pip reduces CI time from 3min to 30s. Cache key based on requirements.txt file hash
GitHub Secrets - Settings -> Secrets -> Actions. Never hardcode credentials in YAML. Use environments for prod secrets with protection
Matrix builds - test on Python 3.10, 3.11, 3.12 in parallel with strategy.matrix. Discover version incompatibilities before prod
GitHub Environments - configure mandatory manual approval before deploying to production. Full traceability of who approved what
Self-hosted runners - run workflows on your own machines (VPN access, GPU, powerful resources). Useful for ML training jobs or sensitive data access

7Level grid

Level	Mastery	GO signal	NO-GO
Confirmed	Basic workflows, automated tests, secrets	Has configured a dbt CI workflow with tests, managed secrets	Runs tests manually, does not know what a workflow is
Senior	dbt slim CI, OIDC, matrix, Docker deployment	Has set up dbt slim CI, uses OIDC instead of JSON keys	Stores GCP keys in plaintext, does not know slim CI

Vous recrutez un Data Engineer ou Analytics Engineer ?

Premier entretien gratuit. Rapport GO/NO-GO sous 48h.

Tester gratuitement Reserver un appel