Accueil›Blog›Test technique Apache Flink : stream processing avancé

Guide recrutement data

Test technique Apache Flink : stream processing avancé

Flink est la référence pour le stream processing à faible latence. En entretien Senior, on évalue la compréhension de l event time, du stateful processing et des garanties de livraison.

Data Builder·Juin 2025·6 min de lecture·Data Engineer

Sommaire

Flink vs Spark Streaming
Event time vs processing time
Stateful processing
Checkpointing et fault tolerance
Flink SQL
Cas d usage
Grille

1Flink vs Spark Structured Streaming

Question discriminante

Dans quel cas choisissez-vous Flink plutôt que Spark Streaming ?

	Spark Streaming	Apache Flink
Modèle	Micro-batch (100ms minimum)	True streaming (event by event)
Latence	100ms - 1s	1-10ms
State management	Limité	Très riche (RocksDB)
Exactement-une-fois	Possible mais complexe	Natif
Idéal pour	Équipes Spark, batch + streaming unified	Latence sub-seconde, état complexe, fraude temps réel

2Event time vs processing time

Question discriminante

Quelle est la différence entre event time et processing time dans Flink ?

from pyflink.datastream import StreamExecutionEnvironment
from pyflink.common.time import Time
from pyflink.datastream.window import TumblingEventTimeWindows

env = StreamExecutionEnvironment.get_execution_environment()

# Processing time : l heure système quand l event est traité
# Simple mais incohérent si les events arrivent en retard
stream.window(TumblingProcessingTimeWindows.of(Time.minutes(5)))

# Event time : l heure DANS l event (quand il s est passé)
# Cohérent, mais nécessite des watermarks pour les retardataires
stream \
    .assign_timestamps_and_watermarks(
        WatermarkStrategy
            .for_bounded_out_of_orderness(Duration.of_seconds(10))
            .with_timestamp_assigner(MyTimestampAssigner())
    ) \
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))

Event time recommandé — pour les analytics. Les résultats sont déterministes même si les events arrivent en retard
Processing time — pour les cas où la latence est critique et la cohérence moins importante

3Stateful processing : l avantage de Flink

Question discriminante

Qu est-ce que le stateful processing dans Flink ? Donnez un exemple concret.

from pyflink.datastream import KeyedProcessFunction
from pyflink.datastream.state import ValueStateDescriptor

# Détecter les sessions utilisateur avec état
class SessionDetector(KeyedProcessFunction):
    def open(self, runtime_context):
        # État persistant par clé (user_id)
        self.last_event_time = runtime_context.get_state(
            ValueStateDescriptor('last_event', Types.LONG())
        )
        self.session_start = runtime_context.get_state(
            ValueStateDescriptor('session_start', Types.LONG())
        )

    def process_element(self, event, ctx):
        current_time = event.timestamp
        last_time = self.last_event_time.value()
        
        if last_time is None or (current_time - last_time) > 30 * 60 * 1000:
            # Nouvelle session (> 30 minutes d inactivité)
            self.session_start.update(current_time)
        
        self.last_event_time.update(current_time)
        yield event

4Checkpointing et fault tolerance

Question discriminante

Comment Flink garantit-il exactly-once en cas de panne ?

Checkpointing — Flink sauvegarde périodiquement l état de tous les opérateurs. En cas de panne, reprend depuis le dernier checkpoint
Distributed snapshots — algorithme de Chandy-Lamport : snapshot cohérent sans arrêter le traitement
Exactly-once — combinaison du checkpointing Flink + transactions sur les sinks (Kafka transactions, JDBC transactions)
Configuration — env.enable_checkpointing(60000) : checkpoint toutes les 60 secondes
Backend — RocksDB pour les gros états (spillover sur disque), HashMapStateBackend pour les petits états (tout en RAM)

5Flink SQL : streaming avec SQL

Question discriminante

Comment écrivez-vous un job Flink avec SQL plutôt que l API Java/Python ?

from pyflink.table import EnvironmentSettings, TableEnvironment

env_settings = EnvironmentSettings.in_streaming_mode()
tbl_env = TableEnvironment.create(env_settings)

# Définir la source Kafka
tbl_env.execute_sql("""
    CREATE TABLE orders (
        order_id STRING,
        user_id STRING,
        amount DOUBLE,
        event_time TIMESTAMP(3),
        WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
    ) WITH (
        'connector' = 'kafka',
        'topic' = 'orders',
        'properties.bootstrap.servers' = 'kafka:9092',
        'format' = 'json'
    )
""")

# Agrégation sur une fenêtre temporelle
tbl_env.execute_sql("""
    SELECT
        TUMBLE_START(event_time, INTERVAL '5' MINUTE) AS window_start,
        user_id,
        SUM(amount) AS total_amount
    FROM orders
    GROUP BY TUMBLE(event_time, INTERVAL '5' MINUTE), user_id
""")

6Cas d usage Flink en production

Question discriminante

Dans quels contextes d entreprise voit-on Flink déployé ?

Détection de fraude temps réel — analyser chaque transaction en quelques millisecondes, croiser avec l historique du compte
Alertes métier immédiates — alerter quand le stock d un produit tombe sous un seuil, en temps réel
ETL streaming — transformer les events Kafka et charger dans BigQuery/Snowflake en quasi-temps-réel
Feature engineering temps réel — calculer des features (moyenne mobile, count récent) pour le serving ML

from pyflink.table import StreamTableEnvironment, EnvironmentSettings

settings = EnvironmentSettings.new_instance().in_streaming_mode().build()
t_env = StreamTableEnvironment.create(environment_settings=settings)

t_env.execute_sql("""
    CREATE TABLE orders (
        order_id STRING,
        amount DOUBLE,
        customer_id STRING,
        event_time TIMESTAMP(3),
        WATERMARK FOR event_time AS event_time - INTERVAL '10' SECOND
    ) WITH (
        'connector' = 'kafka',
        'topic' = 'orders',
        'properties.bootstrap.servers' = 'kafka:9092',
        'format' = 'json'
    )
""")

result = t_env.sql_query("""
    SELECT customer_id,
        TUMBLE_START(event_time, INTERVAL '5' MINUTE) AS window_start,
        SUM(amount) AS total_5min
    FROM orders
    GROUP BY customer_id, TUMBLE(event_time, INTERVAL '5' MINUTE)
""")

Flink vs Spark Streaming - Flink : vrai streaming evenement par evenement, latence sub-seconde. Spark : micro-batch (latence secondes), meilleur ecosysteme Python
Exactly-once avec checkpointing - Flink checkpointe l etat dans S3/GCS. En cas de panne, reprise exacte sans doublons
Flink SQL - pipelines streaming en SQL. Confluent, Amazon MSF utilisent Flink SQL comme interface principale
CEP (Complex Event Processing) - detecter des patterns dans des sequences. Ex : 3 transactions > 1000 euros en 5 minutes sur le meme compte
Managed Flink - Amazon MSF, Confluent Cloud, ou self-hosted Kubernetes avec l operator Flink

Flink vs Spark Streaming - Flink : vrai streaming evenement par evenement, latence sub-seconde. Spark Structured Streaming : micro-batch (latence secondes), meilleur ecosysteme Python
Exactly-once avec checkpointing - Flink checkpointe l etat periodiquement dans S3/GCS. En cas de panne, reprise exacte sans doublons
CEP (Complex Event Processing) - detecter des patterns dans des sequences d evenements : 3 transactions > 1000 euros en 5 minutes pour le meme compte
Managed Flink - Amazon MSF, Confluent Cloud, ou self-hosted Kubernetes avec l operator Flink
Flink SQL - definir des pipelines streaming en SQL. Confluence, Amazon MSF et Immeuble utilisent Flink SQL comme interface principale

7Grille par niveau

Niveau	Maitrise	Signal GO	NO-GO
Confirmé	Event time vs processing time, watermarks, Flink SQL	Explique la différence event/processing time, a écrit du Flink SQL	Ne sait pas la différence entre Flink et Spark Streaming
Senior	Stateful processing, checkpointing, exactly-once	A implémenté du stateful processing, comprend le checkpointing	Ne sait pas ce qu est le stateful processing

1Flink vs Spark Structured Streaming

Discriminating question

In what case do you choose Flink over Spark Streaming?

	Spark Streaming	Apache Flink
Model	Micro-batch (100ms minimum)	True streaming (event by event)
Latency	100ms - 1s	1-10ms
State management	Limited	Very rich (RocksDB)
Exactly-once	Possible but complex	Native
Ideal for	Spark teams, batch + streaming unified	Sub-second latency, complex state, real-time fraud

2Event time vs processing time

Discriminating question

What is the difference between event time and processing time in Flink?

from pyflink.datastream import StreamExecutionEnvironment
from pyflink.common.time import Time
from pyflink.datastream.window import TumblingEventTimeWindows

env = StreamExecutionEnvironment.get_execution_environment()

# Processing time : l heure système quand l event est traité
# Simple mais incohérent si les events arrivent en retard
stream.window(TumblingProcessingTimeWindows.of(Time.minutes(5)))

# Event time : l heure DANS l event (quand il s est passé)
# Cohérent, mais nécessite des watermarks pour les retardataires
stream \
    .assign_timestamps_and_watermarks(
        WatermarkStrategy
            .for_bounded_out_of_orderness(Duration.of_seconds(10))
            .with_timestamp_assigner(MyTimestampAssigner())
    ) \
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))

Event time recommended — for analytics. Results are deterministic even if events arrive late
Processing time — for cases where latency is critical and consistency is less important

3Stateful processing: Flink's advantage

Discriminating question

What is stateful processing in Flink? Give a concrete example.

from pyflink.datastream import KeyedProcessFunction
from pyflink.datastream.state import ValueStateDescriptor

# Détecter les sessions utilisateur avec état
class SessionDetector(KeyedProcessFunction):
    def open(self, runtime_context):
        # État persistant par clé (user_id)
        self.last_event_time = runtime_context.get_state(
            ValueStateDescriptor('last_event', Types.LONG())
        )
        self.session_start = runtime_context.get_state(
            ValueStateDescriptor('session_start', Types.LONG())
        )

    def process_element(self, event, ctx):
        current_time = event.timestamp
        last_time = self.last_event_time.value()
        
        if last_time is None or (current_time - last_time) > 30 * 60 * 1000:
            # Nouvelle session (> 30 minutes d inactivité)
            self.session_start.update(current_time)
        
        self.last_event_time.update(current_time)
        yield event

4Checkpointing and fault tolerance

Discriminating question

How does Flink guarantee exactly-once in case of failure?

Checkpointing — Flink periodically saves the state of all operators. In case of failure, resumes from the last checkpoint
Distributed snapshots — Chandy-Lamport algorithm: consistent snapshot without stopping processing
Exactly-once — combination of Flink checkpointing + transactions on sinks (Kafka transactions, JDBC transactions)
Configuration — env.enable_checkpointing(60000): checkpoint every 60 seconds
Backend — RocksDB for large states (spillover to disk), HashMapStateBackend for small states (all in RAM)

5Flink SQL: streaming with SQL

Discriminating question

How do you write a Flink job with SQL rather than the Java/Python API?

from pyflink.table import EnvironmentSettings, TableEnvironment

env_settings = EnvironmentSettings.in_streaming_mode()
tbl_env = TableEnvironment.create(env_settings)

# Définir la source Kafka
tbl_env.execute_sql("""
    CREATE TABLE orders (
        order_id STRING,
        user_id STRING,
        amount DOUBLE,
        event_time TIMESTAMP(3),
        WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
    ) WITH (
        'connector' = 'kafka',
        'topic' = 'orders',
        'properties.bootstrap.servers' = 'kafka:9092',
        'format' = 'json'
    )
""")

# Agrégation sur une fenêtre temporelle
tbl_env.execute_sql("""
    SELECT
        TUMBLE_START(event_time, INTERVAL '5' MINUTE) AS window_start,
        user_id,
        SUM(amount) AS total_amount
    FROM orders
    GROUP BY TUMBLE(event_time, INTERVAL '5' MINUTE), user_id
""")

6Flink production use cases

Discriminating question

In what enterprise contexts is Flink deployed?

Real-time fraud detection — analyze each transaction in a few milliseconds, cross-reference with account history
Immediate business alerts — alert when a product's stock falls below a threshold, in real time
Streaming ETL — transform Kafka events and load into BigQuery/Snowflake in near-real-time
Real-time feature engineering — compute features (moving average, recent count) for ML serving

from pyflink.table import StreamTableEnvironment, EnvironmentSettings

settings = EnvironmentSettings.new_instance().in_streaming_mode().build()
t_env = StreamTableEnvironment.create(environment_settings=settings)

t_env.execute_sql("""
    CREATE TABLE orders (
        order_id STRING,
        amount DOUBLE,
        customer_id STRING,
        event_time TIMESTAMP(3),
        WATERMARK FOR event_time AS event_time - INTERVAL '10' SECOND
    ) WITH (
        'connector' = 'kafka',
        'topic' = 'orders',
        'properties.bootstrap.servers' = 'kafka:9092',
        'format' = 'json'
    )
""")

result = t_env.sql_query("""
    SELECT customer_id,
        TUMBLE_START(event_time, INTERVAL '5' MINUTE) AS window_start,
        SUM(amount) AS total_5min
    FROM orders
    GROUP BY customer_id, TUMBLE(event_time, INTERVAL '5' MINUTE)
""")

Flink vs Spark Streaming - Flink: true streaming event by event, sub-second latency. Spark: micro-batch (second-level latency), better Python ecosystem
Exactly-once with checkpointing - Flink checkpoints state to S3/GCS. In case of failure, exact recovery without duplicates
Flink SQL - streaming pipelines in SQL. Confluent, Amazon MSF use Flink SQL as their main interface
CEP (Complex Event Processing) - detect patterns in sequences. Ex: 3 transactions > 1000 euros in 5 minutes on the same account
Managed Flink - Amazon MSF, Confluent Cloud, or self-hosted Kubernetes with the Flink operator

Flink vs Spark Streaming - Flink: true streaming event by event, sub-second latency. Spark Structured Streaming: micro-batch (second-level latency), better Python ecosystem
Exactly-once with checkpointing - Flink periodically checkpoints state to S3/GCS. In case of failure, exact recovery without duplicates
CEP (Complex Event Processing) - detect patterns in sequences of events: 3 transactions > 1000 euros in 5 minutes for the same account
Managed Flink - Amazon MSF, Confluent Cloud, or self-hosted Kubernetes with the Flink operator
Flink SQL - define streaming pipelines in SQL. Confluent, Amazon MSF use Flink SQL as their main interface

7Level grid

Level	Mastery	GO signal	NO-GO
Mid-level	Event time vs processing time, watermarks, Flink SQL	Explains the difference between event/processing time, has written Flink SQL	Does not know the difference between Flink and Spark Streaming
Senior	Stateful processing, checkpointing, exactly-once	Has implemented stateful processing, understands checkpointing	Does not know what stateful processing is

Vous recrutez un Data Engineer streaming avancé ?

Premier entretien gratuit. Rapport GO/NO-GO sous 48h.

Tester gratuitement Reserver un appel