Accueil›Blog›Test technique pandas profiling et diagnostic qualité des données

Guide recrutement data

Test technique pandas profiling et diagnostic qualité des données

Le diagnostic de qualité des données est systématiquement la première étape d un projet data. En entretien, on évalue la rigueur de cette phase souvent négligée.

Data Builder·Juin 2025·6 min de lecture·Data Analyst · Data Scientist

Sommaire

Méthode de diagnostic
YData Profiling
Analyser les valeurs manquantes
Détecter les outliers
Corrélations et redondances
Produire un rapport de qualité
Grille

1Méthode de diagnostic qualité

Question discriminante

Quelle est votre démarche systématique quand vous découvrez un nouveau dataset ?

import pandas as pd
import numpy as np

# Étape 1 : vue d ensemble
print(df.shape)          # dimensions
print(df.dtypes)         # types de colonnes
print(df.head())
print(df.describe())     # statistiques descriptives

# Étape 2 : valeurs manquantes
missing = df.isnull().sum()
missing_pct = (missing / len(df) * 100).sort_values(ascending=False)
print(missing_pct[missing_pct > 0])

# Étape 3 : doublons
print(f'Doublons complets: {df.duplicated().sum()}')
print(f'Doublons sur clé: {df.duplicated(subset=["order_id"]).sum()}')

# Étape 4 : distributions
for col in df.select_dtypes(include='number').columns:
    print(f'{col}: min={df[col].min():.2f}, max={df[col].max():.2f}, '
          f'mean={df[col].mean():.2f}, median={df[col].median():.2f}')

Ordre recommandé — 1. dimensions/types, 2. valeurs manquantes, 3. doublons, 4. distributions, 5. corrélations, 6. anomalies
Documenter — noter chaque anomalie découverte avec son impact potentiel

2YData Profiling : rapport automatique

Question discriminante

Comment utilisez-vous YData Profiling pour accélérer le diagnostic ?

from ydata_profiling import ProfileReport

# Rapport complet (attention : lent sur >100k lignes)
profile = ProfileReport(
    df,
    title='Diagnostic qualité - Commandes 2024',
    explorative=True
)
profile.to_file('diagnostic_qualite.html')

# Rapport minimal pour les gros volumes
profile_minimal = ProfileReport(
    df,
    minimal=True,  # désactive les corrélations coûteuses
    title='Diagnostic rapide'
)

# Ce que le rapport contient automatiquement :
# - Distribution de chaque variable
# - Valeurs manquantes et patterns
# - Corrélations (Pearson, Spearman, Cramér V)
# - Alertes automatiques (skewness, constantes, haute cardinalité)
# - Interactions entre variables

3Analyser les valeurs manquantes

Question discriminante

Comment distinguez-vous MCAR, MAR et MNAR ? Pourquoi est-ce important ?

import missingno as msno
import matplotlib.pyplot as plt

# Visualiser les patterns de valeurs manquantes
msno.matrix(df)           # pattern visuel
msno.heatmap(df)          # corrélation entre colonnes manquantes
msno.dendrogram(df)       # clustering des patterns

# Test de Little (MCAR vs non-MCAR)
# Si p-value < 0.05 : les données ne sont PAS MCAR
from scipy import stats

# Vérifier si les manquants dépendent d autres colonnes
df['income_missing'] = df['income'].isnull().astype(int)
print(df.groupby('income_missing')['age'].describe())
# Si l âge diffère selon income_missing -> MAR (pas MCAR)

MCAR — Missing Completely At Random. Le manquant est aléatoire. Imputation simple acceptable
MAR — Missing At Random. Le manquant dépend d autres variables observées. Imputation par modèle recommandée
MNAR — Missing Not At Random. Le manquant dépend de la valeur elle-même. Problème sérieux, souvent non corrigible

4Détecter les outliers

Question discriminante

Quelles méthodes utilisez-vous pour détecter les valeurs aberrantes ?

import numpy as np
from scipy import stats

# Méthode 1 : IQR (robuste)
def detect_outliers_iqr(series):
    Q1 = series.quantile(0.25)
    Q3 = series.quantile(0.75)
    IQR = Q3 - Q1
    lower = Q1 - 1.5 * IQR
    upper = Q3 + 1.5 * IQR
    return series[(series < lower) | (series > upper)]

# Méthode 2 : Z-score (si distribution normale)
z_scores = np.abs(stats.zscore(df['amount']))
outliers_z = df[z_scores > 3]

# Méthode 3 : Isolation Forest (multivariée)
from sklearn.ensemble import IsolationForest
iso = IsolationForest(contamination=0.01, random_state=42)
df['is_outlier'] = iso.fit_predict(df[numeric_cols])
# -1 = outlier, 1 = normal

IQR — robuste, non paramétrique. À préférer pour les distributions asymétriques
Z-score — sensible aux outliers eux-mêmes. Utiliser seulement sur des distributions proches de la normale
Isolation Forest — détection multivariée. Trouve des outliers invisibles sur une seule dimension

5Corrélations et redondances

Question discriminante

Comment identifiez-vous les features redondantes dans un dataset ?

import seaborn as sns

# Corrélations linéaires (Pearson)
corr_matrix = df.select_dtypes('number').corr()

# Visualiser avec heatmap
sns.heatmap(corr_matrix, annot=True, fmt='.2f', cmap='coolwarm')

# Identifier les paires très corrélées
upper = corr_matrix.where(
    np.triu(np.ones(corr_matrix.shape), k=1).astype(bool)
)
high_corr = [(c, r, upper.loc[r, c])
             for c in upper.columns
             for r in upper.index
             if abs(upper.loc[r, c]) > 0.95]
print('Features très corrélées (>0.95):', high_corr)

# Variance Inflation Factor (VIF) pour la multicolinéarité
from statsmodels.stats.outliers_influence import variance_inflation_factor
vif = pd.DataFrame()
vif['VIF'] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
vif['feature'] = X.columns
# VIF > 10 : multicolinéarité problématique

6Produire un rapport de qualité actionnable

Question discriminante

Qu est-ce qu un rapport de qualité de données actionnable ?

Structure recommandée — résumé exécutif (nombre de problèmes critiques), puis détail par dimension de qualité
Priorisation — distinguer bloquant (ne peut pas utiliser la donnée), majeur (biais possible), mineur (à surveiller)
Recommandations concrètes — pour chaque problème : action recommandée, responsable, délai
Format — notebook Jupyter exporté en HTML, ou rapport YData Profiling, ou dashboard Metabase

import pandas as pd
from ydata_profiling import ProfileReport

df = pd.read_csv('orders.csv')

# Rapport complet de qualite
profile = ProfileReport(df, title='Orders Quality Report', explorative=True)
profile.to_file('quality_report.html')

# Comparaison deux periodes (drift detection)
profile_week1 = ProfileReport(df_week1, title='Week 1')
profile_week2 = ProfileReport(df_week2, title='Week 2')
comparison = profile_week1.compare(profile_week2)
comparison.to_file('drift_comparison.html')

# Verifications manuelles cles
print(df.dtypes)
print(df.isnull().sum() / len(df))  # taux de nullite par colonne
print(df.describe())  # stats descriptives
print(df.duplicated().sum())  # doublons

ydata-profiling - remplace pandas-profiling (abandonne). Genere automatiquement un rapport HTML : distributions, correlations, valeurs manquantes, outliers, doublons
Correlations dangereuses - une correlation > 0.95 entre deux features = potentielle colinearite. A identifier avant d envoyer dans un modele ML
Profiling en CI - generer le profil sur un echantillon a chaque run dbt. Detecter les changements de distribution avant qu ils n atteignent les dashboards
Great Expectations integration - exporter les expectations depuis le profil ydata-profiling. Transformer les observations en regles de validation automatiques
Sampling strategique - sur > 10M lignes, profiler un echantillon stratifie (par date, region) plutot que les premieres lignes. Evite les biais de position

7Grille par niveau

Niveau	Maitrise	Signal GO	NO-GO
Junior	df.describe(), valeurs manquantes, doublons	Fait une exploration systématique avant de modéliser	Commence à modéliser sans explorer les données
Confirmé	YData Profiling, IQR outliers, corrélations	Utilise YData Profiling, distingue MCAR/MAR/MNAR, détecte les outliers	Ne sait pas ce que sont MCAR et MAR
Senior	Isolation Forest, VIF, rapport de qualité structuré	Utilise des méthodes multivariées, produit un rapport avec recommandations	N a jamais produit de rapport de qualité de données

1Quality diagnostic method

Discriminating question

What is your systematic approach when you discover a new dataset?

import pandas as pd
import numpy as np

# Step 1: overview
print(df.shape)          # dimensions
print(df.dtypes)         # column types
print(df.head())
print(df.describe())     # descriptive statistics

# Step 2: missing values
missing = df.isnull().sum()
missing_pct = (missing / len(df) * 100).sort_values(ascending=False)
print(missing_pct[missing_pct > 0])

# Step 3: duplicates
print(f'Full duplicates: {df.duplicated().sum()}')
print(f'Duplicates on key: {df.duplicated(subset=["order_id"]).sum()}')

# Step 4: distributions
for col in df.select_dtypes(include='number').columns:
    print(f'{col}: min={df[col].min():.2f}, max={df[col].max():.2f}, '
          f'mean={df[col].mean():.2f}, median={df[col].median():.2f}')

Recommended order — 1. dimensions/types, 2. missing values, 3. duplicates, 4. distributions, 5. correlations, 6. anomalies
Document — record each anomaly discovered with its potential impact

2YData Profiling: automatic report

Discriminating question

How do you use YData Profiling to speed up the diagnostic?

from ydata_profiling import ProfileReport

# Full report (warning: slow on >100k rows)
profile = ProfileReport(
    df,
    title='Quality diagnostic - Orders 2024',
    explorative=True
)
profile.to_file('quality_diagnostic.html')

# Minimal report for large volumes
profile_minimal = ProfileReport(
    df,
    minimal=True,  # disables expensive correlations
    title='Quick diagnostic'
)

# What the report automatically contains:
# - Distribution of each variable
# - Missing values and patterns
# - Correlations (Pearson, Spearman, Cramér V)
# - Automatic alerts (skewness, constants, high cardinality)
# - Interactions between variables

3Analyzing missing values

Discriminating question

How do you distinguish MCAR, MAR and MNAR? Why does it matter?

import missingno as msno
import matplotlib.pyplot as plt

# Visualize missing value patterns
msno.matrix(df)           # visual pattern
msno.heatmap(df)          # correlation between missing columns
msno.dendrogram(df)       # clustering of patterns

# Little's test (MCAR vs non-MCAR)
# If p-value < 0.05: data is NOT MCAR
from scipy import stats

# Check if missingness depends on other columns
df['income_missing'] = df['income'].isnull().astype(int)
print(df.groupby('income_missing')['age'].describe())
# If age differs based on income_missing -> MAR (not MCAR)

MCAR — Missing Completely At Random. Missingness is random. Simple imputation acceptable
MAR — Missing At Random. Missingness depends on other observed variables. Model-based imputation recommended
MNAR — Missing Not At Random. Missingness depends on the value itself. Serious problem, often uncorrectable

4Detecting outliers

Discriminating question

What methods do you use to detect outliers?

import numpy as np
from scipy import stats

# Method 1: IQR (robust)
def detect_outliers_iqr(series):
    Q1 = series.quantile(0.25)
    Q3 = series.quantile(0.75)
    IQR = Q3 - Q1
    lower = Q1 - 1.5 * IQR
    upper = Q3 + 1.5 * IQR
    return series[(series < lower) | (series > upper)]

# Method 2: Z-score (if normal distribution)
z_scores = np.abs(stats.zscore(df['amount']))
outliers_z = df[z_scores > 3]

# Method 3: Isolation Forest (multivariate)
from sklearn.ensemble import IsolationForest
iso = IsolationForest(contamination=0.01, random_state=42)
df['is_outlier'] = iso.fit_predict(df[numeric_cols])
# -1 = outlier, 1 = normal

IQR — robust, non-parametric. Preferred for skewed distributions
Z-score — sensitive to outliers themselves. Use only on distributions close to normal
Isolation Forest — multivariate detection. Finds outliers invisible on a single dimension

5Correlations and redundancies

Discriminating question

How do you identify redundant features in a dataset?

import seaborn as sns

# Linear correlations (Pearson)
corr_matrix = df.select_dtypes('number').corr()

# Visualize with heatmap
sns.heatmap(corr_matrix, annot=True, fmt='.2f', cmap='coolwarm')

# Identify highly correlated pairs
upper = corr_matrix.where(
    np.triu(np.ones(corr_matrix.shape), k=1).astype(bool)
)
high_corr = [(c, r, upper.loc[r, c])
             for c in upper.columns
             for r in upper.index
             if abs(upper.loc[r, c]) > 0.95]
print('Highly correlated features (>0.95):', high_corr)

# Variance Inflation Factor (VIF) for multicollinearity
from statsmodels.stats.outliers_influence import variance_inflation_factor
vif = pd.DataFrame()
vif['VIF'] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
vif['feature'] = X.columns
# VIF > 10: problematic multicollinearity

6Producing an actionable quality report

Discriminating question

What is an actionable data quality report?

Recommended structure — executive summary (number of critical issues), then detail by quality dimension
Prioritization — distinguish blocking (data cannot be used), major (possible bias), minor (to monitor)
Concrete recommendations — for each issue: recommended action, owner, deadline
Format — Jupyter notebook exported to HTML, or YData Profiling report, or Metabase dashboard

import pandas as pd
from ydata_profiling import ProfileReport

df = pd.read_csv('orders.csv')

# Full quality report
profile = ProfileReport(df, title='Orders Quality Report', explorative=True)
profile.to_file('quality_report.html')

# Two-period comparison (drift detection)
profile_week1 = ProfileReport(df_week1, title='Week 1')
profile_week2 = ProfileReport(df_week2, title='Week 2')
comparison = profile_week1.compare(profile_week2)
comparison.to_file('drift_comparison.html')

# Key manual checks
print(df.dtypes)
print(df.isnull().sum() / len(df))  # null rate per column
print(df.describe())  # descriptive stats
print(df.duplicated().sum())  # duplicates

ydata-profiling - replaces pandas-profiling (deprecated). Automatically generates an HTML report: distributions, correlations, missing values, outliers, duplicates
Dangerous correlations - a correlation > 0.95 between two features = potential collinearity. Must be identified before feeding into an ML model
Profiling in CI - generate the profile on a sample at each dbt run. Detect distribution changes before they reach dashboards
Great Expectations integration - export expectations from the ydata-profiling profile. Turn observations into automatic validation rules
Strategic sampling - on > 10M rows, profile a stratified sample (by date, region) rather than the first rows. Avoids position bias

7Level grid

Level	Mastery	GO signal	NO-GO
Junior	df.describe(), missing values, duplicates	Performs systematic exploration before modeling	Starts modeling without exploring the data
Mid-level	YData Profiling, IQR outliers, correlations	Uses YData Profiling, distinguishes MCAR/MAR/MNAR, detects outliers	Does not know what MCAR and MAR are
Senior	Isolation Forest, VIF, structured quality report	Uses multivariate methods, produces a report with recommendations	Has never produced a data quality report

Vous recrutez un Data Analyst ou Data Scientist ?

Premier entretien gratuit. Rapport GO/NO-GO sous 48h.

Tester gratuitement Reserver un appel