Accueil›Blog›Test technique Apache Iceberg avancé : catalog, time travel, merge-on-read

Guide recrutement data

Test technique Apache Iceberg avancé : catalog, time travel, merge-on-read

Iceberg est devenu le standard ouvert pour les tables lakehouse. En entretien Expert, on évalue la maîtrise des fonctionnalités avancées : catalog REST, partition evolution, et les stratégies d écriture.

Data Builder·Juin 2025·7 min de lecture·Data Engineer

Sommaire

Iceberg Catalog
Partition Evolution
Time Travel avancé
Copy-on-Write vs Merge-on-Read
Maintenance de tables
Interopérabilité multi-engine
Grille

1Iceberg Catalog : l abstraction de gouvernance

Question discriminante

Qu est-ce qu un Iceberg Catalog ? Pourquoi est-il central dans une architecture Lakehouse ?

## Types de catalogs Iceberg

# 1. REST Catalog (standard émergent 2025)
# Serveur HTTP qui expose l API Iceberg REST
# Compatible avec tous les engines (Spark, Trino, Flink, DuckDB)
spark = SparkSession.builder \
    .config('spark.sql.catalog.prod', 'org.apache.iceberg.spark.SparkCatalog') \
    .config('spark.sql.catalog.prod.catalog-impl', 'org.apache.iceberg.rest.RESTCatalog') \
    .config('spark.sql.catalog.prod.uri', 'https://iceberg-catalog.monentreprise.com') \
    .getOrCreate()

# 2. AWS Glue Catalog (managé AWS)
.config('spark.sql.catalog.glue.catalog-impl', 'org.apache.iceberg.aws.glue.GlueCatalog')

# 3. Nessie (versioning Git-like pour le catalog)
.config('spark.sql.catalog.nessie.catalog-impl', 'org.projectnessie.iceberg.NessieCatalog')

Catalog = registre des tables — connaît l emplacement des metadata files pour chaque table
Multi-engine — Spark, Trino, Flink et DuckDB lisent tous depuis le même catalog
Nessie — catalog avec branches Git-like. Isoler des modifications de tables avant de les merger

2Partition Evolution : changer sans réécrire

Question discriminante

Qu est-ce que la partition evolution ? Pourquoi est-ce un avantage majeur d Iceberg sur Hive ?

from pyiceberg.catalog import load_catalog
from pyiceberg.transforms import MonthTransform, DayTransform

catalog = load_catalog('prod')
table = catalog.load_table('analytics.orders')

# Partition Evolution : changer le schéma de partition
# sans réécrire les données existantes
with table.update_spec() as update:
    # Avant : partitionné par mois
    # Après : partitionner par jour (pour les nouvelles partitions)
    update.remove_field('order_month')     # supprimer l ancienne
    update.add_identity('order_date')      # ajouter la nouvelle

# Les anciennes données restent dans leurs partitions par mois
# Les nouvelles données vont dans des partitions par jour
# Iceberg gère la coexistence automatiquement

# Avec Hive : impossible sans réécrire toutes les données !

3Time Travel avancé

Question discriminante

Comment requêtez-vous l état d une table Iceberg à un snapshot précis ?

-- Time Travel par timestamp
SELECT * FROM prod.analytics.orders
FOR SYSTEM_TIME AS OF '2025-01-15 08:00:00';

-- Time Travel par snapshot ID
SELECT * FROM prod.analytics.orders
FOR VERSION AS OF 1234567890;

-- Lister les snapshots disponibles
SELECT * FROM prod.analytics.orders.snapshots
ORDER BY committed_at DESC
LIMIT 10;

-- Comparer deux snapshots (identifier ce qui a changé)
SELECT * FROM prod.analytics.orders.changes
BETWEEN TIMESTAMP '2025-01-14 00:00:00'
AND TIMESTAMP '2025-01-15 00:00:00';

-- Rollback vers un état précédent
CALL prod.system.rollback_to_timestamp(
    'analytics.orders',
    TIMESTAMP '2025-01-14 12:00:00'
);

4Copy-on-Write vs Merge-on-Read

Question discriminante

Quelle est la différence entre CoW et MoR ? Quand choisissez-vous l un ou l autre ?

	Copy-on-Write (CoW)	Merge-on-Read (MoR)
Écriture	Réécriture des fichiers impactés	Écriture dans des fichiers delta (positional deletes)
Lecture	Lecture directe, fichiers propres	Merge à la lecture (fichiers data + deletes)
Coût écriture	Élevé (I/O)	Faible
Coût lecture	Faible	Potentiellement élevé (sans compaction)
Idéal pour	Lectures fréquentes, écritures rares	Mises à jour fréquentes (CDC), écritures intensives

5Maintenance des tables Iceberg

Question discriminante

Quelles opérations de maintenance planifiez-vous sur vos tables Iceberg ?

## Opérations de maintenance (scheduler via Airflow)

# 1. Compaction : fusionner les petits fichiers
SPARK.sql("""
    CALL prod.system.rewrite_data_files(
        table => 'analytics.orders',
        strategy => 'sort',
        sort_order => 'order_date, region',
        options => map('min-input-files', '5', 'target-file-size-bytes', '134217728')
    )
""")

# 2. Expirer les anciens snapshots
SPARK.sql("""
    CALL prod.system.expire_snapshots(
        table => 'analytics.orders',
        older_than => TIMESTAMP '2024-12-01 00:00:00',
        retain_last => 3
    )
""")

# 3. Supprimer les fichiers orphelins
SPARK.sql("""
    CALL prod.system.remove_orphan_files(
        table => 'analytics.orders',
        dry_run => false
    )
""")

6Interopérabilité : Spark + Trino + DuckDB

Question discriminante

Comment faites-vous cohabiter plusieurs engines sur les mêmes tables Iceberg ?

## Trino + Iceberg : SQL OLAP performant
CREATE TABLE iceberg.analytics.orders (
    order_id VARCHAR,
    amount DOUBLE,
    order_date DATE
) WITH (
    format = 'PARQUET',
    partitioning = ARRAY['month(order_date)'],
    sorted_by = ARRAY['order_date']
);

## DuckDB + Iceberg (v0.10+)
INSTALL iceberg; LOAD iceberg;
SELECT * FROM iceberg_scan(
    's3://mon-bucket/warehouse/analytics/orders',
    allow_moved_paths = true
);

## Règle de cohabitation :
# Lire depuis tous les engines
# Écrire depuis 1 seul engine à la fois (pas de concurrent writes)
# Sauf Iceberg Multi-Writer (optimistic locking)

-- Iceberg Time Travel et maintenance
SELECT * FROM prod.orders FOR SYSTEM_TIME AS OF '2025-01-01 00:00:00';
SELECT * FROM prod.orders FOR VERSION AS OF 42;

-- Schema Evolution sans réécriture
ALTER TABLE prod.orders ADD COLUMN discount DOUBLE;
ALTER TABLE prod.orders RENAME COLUMN client_id TO customer_id;

-- Partition Evolution
ALTER TABLE prod.orders REPLACE PARTITION FIELD month(order_date) WITH day(order_date);

-- Row-level deletes
DELETE FROM prod.orders WHERE status = 'test';

-- Compaction
CALL catalog.system.rewrite_data_files(table => 'prod.orders');

-- Expirer snapshots anciens
CALL catalog.system.expire_snapshots(
    table => 'prod.orders',
    older_than => TIMESTAMP '2025-01-01 00:00:00',
    retain_last => 10
);

Copy-on-write vs Merge-on-read - CoW (defaut) : reecrire les fichiers lors UPDATE/DELETE, lectures rapides. MoR : delete files separes, ecritures rapides, lectures plus lentes (fusion au read)
Catalogs Iceberg - REST Catalog (standard portable : Polaris, Unity Catalog), Hive Metastore, AWS Glue, Nessie (versioning Git-like). Choisir selon l ecosysteme
Multi-engine - meme table lue depuis Spark, Trino, Flink, DuckDB, BigQuery Omni sans conversion ni copie
vs Delta Lake - Iceberg : portabilite multi-engine maximale. Delta Lake : integration Databricks/Photon optimale. Les deux font Time Travel, ACID, schema evolution
Hidden partitioning - Iceberg cache la strategie de partition au SQL. Pas besoin de filtrer sur la colonne de partition exacte

7Grille par niveau

Niveau	Maitrise	Signal GO	NO-GO
Confirmé	Tables Iceberg basiques, time travel, catalogs	A créé des tables Iceberg, utilise le time travel, comprend les catalogs	Ne sait pas la différence entre Iceberg et un simple Parquet
Senior	Partition evolution, CoW vs MoR, maintenance planifiée	A fait une partition evolution, choisit CoW vs MoR selon le cas, planifie la compaction	Ne sait pas ce qu est la partition evolution

1Iceberg Catalog: the governance abstraction

Discriminating question

What is an Iceberg Catalog? Why is it central in a Lakehouse architecture?

## Iceberg catalog types

# 1. REST Catalog (emerging standard 2025)
# HTTP server that exposes the Iceberg REST API
# Compatible with all engines (Spark, Trino, Flink, DuckDB)
spark = SparkSession.builder \
    .config('spark.sql.catalog.prod', 'org.apache.iceberg.spark.SparkCatalog') \
    .config('spark.sql.catalog.prod.catalog-impl', 'org.apache.iceberg.rest.RESTCatalog') \
    .config('spark.sql.catalog.prod.uri', 'https://iceberg-catalog.monentreprise.com') \
    .getOrCreate()

# 2. AWS Glue Catalog (AWS managed)
.config('spark.sql.catalog.glue.catalog-impl', 'org.apache.iceberg.aws.glue.GlueCatalog')

# 3. Nessie (Git-like versioning for the catalog)
.config('spark.sql.catalog.nessie.catalog-impl', 'org.projectnessie.iceberg.NessieCatalog')

Catalog = table registry — knows the location of metadata files for each table
Multi-engine — Spark, Trino, Flink and DuckDB all read from the same catalog
Nessie — catalog with Git-like branches. Isolate table changes before merging them

2Partition Evolution: changing without rewriting

Discriminating question

What is partition evolution? Why is it a major advantage of Iceberg over Hive?

from pyiceberg.catalog import load_catalog
from pyiceberg.transforms import MonthTransform, DayTransform

catalog = load_catalog('prod')
table = catalog.load_table('analytics.orders')

# Partition Evolution: change the partition schema
# without rewriting existing data
with table.update_spec() as update:
    # Before: partitioned by month
    # After: partition by day (for new partitions)
    update.remove_field('order_month')     # remove the old one
    update.add_identity('order_date')      # add the new one

# Old data stays in its monthly partitions
# New data goes into daily partitions
# Iceberg handles coexistence automatically

# With Hive: impossible without rewriting all data!

3Advanced Time Travel

Discriminating question

How do you query the state of an Iceberg table at a specific snapshot?

-- Time Travel by timestamp
SELECT * FROM prod.analytics.orders
FOR SYSTEM_TIME AS OF '2025-01-15 08:00:00';

-- Time Travel by snapshot ID
SELECT * FROM prod.analytics.orders
FOR VERSION AS OF 1234567890;

-- List available snapshots
SELECT * FROM prod.analytics.orders.snapshots
ORDER BY committed_at DESC
LIMIT 10;

-- Compare two snapshots (identify what changed)
SELECT * FROM prod.analytics.orders.changes
BETWEEN TIMESTAMP '2025-01-14 00:00:00'
AND TIMESTAMP '2025-01-15 00:00:00';

-- Rollback to a previous state
CALL prod.system.rollback_to_timestamp(
    'analytics.orders',
    TIMESTAMP '2025-01-14 12:00:00'
);

4Copy-on-Write vs Merge-on-Read

Discriminating question

What is the difference between CoW and MoR? When do you choose one over the other?

	Copy-on-Write (CoW)	Merge-on-Read (MoR)
Write	Rewrite of impacted files	Write into delta files (positional deletes)
Read	Direct read, clean files	Merge at read time (data files + deletes)
Write cost	High (I/O)	Low
Read cost	Low	Potentially high (without compaction)
Ideal for	Frequent reads, rare writes	Frequent updates (CDC), write-intensive workloads

5Iceberg Table Maintenance

Discriminating question

What maintenance operations do you schedule on your Iceberg tables?

## Maintenance operations (scheduler via Airflow)

# 1. Compaction: merge small files
SPARK.sql("""
    CALL prod.system.rewrite_data_files(
        table => 'analytics.orders',
        strategy => 'sort',
        sort_order => 'order_date, region',
        options => map('min-input-files', '5', 'target-file-size-bytes', '134217728')
    )
""")

# 2. Expire old snapshots
SPARK.sql("""
    CALL prod.system.expire_snapshots(
        table => 'analytics.orders',
        older_than => TIMESTAMP '2024-12-01 00:00:00',
        retain_last => 3
    )
""")

# 3. Remove orphan files
SPARK.sql("""
    CALL prod.system.remove_orphan_files(
        table => 'analytics.orders',
        dry_run => false
    )
""")

6Interoperability: Spark + Trino + DuckDB

Discriminating question

How do you run multiple engines on the same Iceberg tables?

## Trino + Iceberg: performant OLAP SQL
CREATE TABLE iceberg.analytics.orders (
    order_id VARCHAR,
    amount DOUBLE,
    order_date DATE
) WITH (
    format = 'PARQUET',
    partitioning = ARRAY['month(order_date)'],
    sorted_by = ARRAY['order_date']
);

## DuckDB + Iceberg (v0.10+)
INSTALL iceberg; LOAD iceberg;
SELECT * FROM iceberg_scan(
    's3://mon-bucket/warehouse/analytics/orders',
    allow_moved_paths = true
);

## Coexistence rules:
# Read from all engines
# Write from 1 engine at a time (no concurrent writes)
# Except Iceberg Multi-Writer (optimistic locking)

-- Iceberg Time Travel and maintenance
SELECT * FROM prod.orders FOR SYSTEM_TIME AS OF '2025-01-01 00:00:00';
SELECT * FROM prod.orders FOR VERSION AS OF 42;

-- Schema Evolution without rewrite
ALTER TABLE prod.orders ADD COLUMN discount DOUBLE;
ALTER TABLE prod.orders RENAME COLUMN client_id TO customer_id;

-- Partition Evolution
ALTER TABLE prod.orders REPLACE PARTITION FIELD month(order_date) WITH day(order_date);

-- Row-level deletes
DELETE FROM prod.orders WHERE status = 'test';

-- Compaction
CALL catalog.system.rewrite_data_files(table => 'prod.orders');

-- Expire old snapshots
CALL catalog.system.expire_snapshots(
    table => 'prod.orders',
    older_than => TIMESTAMP '2025-01-01 00:00:00',
    retain_last => 10
);

Copy-on-write vs Merge-on-read - CoW (default): rewrite files on UPDATE/DELETE, fast reads. MoR: separate delete files, fast writes, slower reads (merge at read time)
Iceberg Catalogs - REST Catalog (portable standard: Polaris, Unity Catalog), Hive Metastore, AWS Glue, Nessie (Git-like versioning). Choose based on the ecosystem
Multi-engine - same table read from Spark, Trino, Flink, DuckDB, BigQuery Omni without conversion or copy
vs Delta Lake - Iceberg: maximum multi-engine portability. Delta Lake: optimal Databricks/Photon integration. Both support Time Travel, ACID, schema evolution
Hidden partitioning - Iceberg hides the partition strategy from SQL. No need to filter on the exact partition column

7Level grid

Level	Mastery	GO signal	NO-GO
Mid-level	Basic Iceberg tables, time travel, catalogs	Has created Iceberg tables, uses time travel, understands catalogs	Does not know the difference between Iceberg and plain Parquet
Senior	Partition evolution, CoW vs MoR, scheduled maintenance	Has performed partition evolution, chooses CoW vs MoR based on use case, schedules compaction	Does not know what partition evolution is

Vous recrutez un Data Engineer Iceberg Expert ?

Premier entretien gratuit. Rapport GO/NO-GO sous 48h.

Tester gratuitement Reserver un appel