Accueil›Blog›Scraping et automatisation Python avancee : Playwright, Scrapy, grande echelle

Guide recrutement data

Scraping et automatisation Python avancee : Playwright, Scrapy, grande echelle

Le scraping va au-dela de BeautifulSoup sur un site statique. En entretien Data Engineer Senior, on evalue la capacite a gerer les sites dynamiques, la grande echelle et les contraintes legales.

Data Builder·Juin 2025·6 min de lecture·Data Engineer

Sommaire

Crawling vs scraping vs automatisation
Ecosysteme des outils
Playwright vs Selenium
Scrapy a grande echelle
Anti-detection
Legislation
Grille

Le scraping est un outil puissant mais qui engage la responsabilite du developpeur. En entretien, on ne teste pas seulement la technique — on verifie aussi la connaissance des limites legales et ethiques.

1Crawling, scraping, automatisation : les 3 notions

Question discriminante

Quelle est la difference entre le crawling et le scraping ? Et l automatisation web ?

Crawling — parcourir automatiquement un site pour decouvrir des pages et des liens. C est ce que font les bots Google. Objectif : indexation
Scraping — extraire des donnees specifiques d une page web. Objectif : recuperer la donnee pour soi
Automatisation — reproduire automatiquement des actions humaines sur un site (remplir un formulaire, cliquer, telecharger). Objectif : remplacer une tache repetitive

2Ecosysteme Python du scraping 2025

Outil	Statique	Dynamique	Masse	Ideal pour
BeautifulSoup + requests	Oui	Non	Non	Pages statiques simples, prototypage
Playwright	Oui	Oui	Moyen	Sites dynamiques JS, automatisation UI
Selenium	Oui	Oui	Non	Sites dynamiques, tests UI (plus lent que Playwright)
Scrapy	Oui	Partiel	Oui	Scraping massif, pipelines de production
httpx + asyncio	Oui	Non	Oui	Requetes paralleles a haute performance

3Playwright : le successeur moderne de Selenium

Question discriminante

Pourquoi Playwright est-il prefere a Selenium en 2025 ? Quand utilisez-vous l un ou l autre ?

from playwright.async_api import async_playwright
import asyncio

async def scrape_spa():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        
        # Intercepter les requetes reseau
        page.on('response', lambda r: print(r.url) if '/api/' in r.url else None)
        
        await page.goto('https://exemple.com/products')
        
        # Attendre que les donnees soient chargees
        await page.wait_for_selector('.product-card', timeout=10000)
        
        products = await page.query_selector_all('.product-card')
        data = []
        for p_elem in products:
            name = await p_elem.query_selector('.title')
            price = await p_elem.query_selector('.price')
            data.append({
                'name': await name.inner_text(),
                'price': await price.inner_text()
            })
        
        await browser.close()
        return data

asyncio.run(scrape_spa())

Playwright vs Selenium — Playwright est plus rapide, supporte l async natif, meilleure gestion des timeouts, supporte les 3 moteurs (Chromium, Firefox, WebKit)
Interception reseau — si l app charge les donnees via une API JSON, la scraper directement est 10x plus simple que le scraping DOM

4Scrapy : scraping industriel

import scrapy

class ProductSpider(scrapy.Spider):
    name = 'products'
    start_urls = ['https://exemple.com/catalogue']
    
    custom_settings = {
        'DOWNLOAD_DELAY': 1.5,       # politesse
        'ROBOTSTXT_OBEY': True,      # respecte robots.txt
        'AUTOTHROTTLE_ENABLED': True, # adaptation automatique
        'ITEM_PIPELINES': {
            'myproject.pipelines.DatabasePipeline': 300,
        }
    }
    
    def parse(self, response):
        for product in response.css('.product-card'):
            yield {
                'name': product.css('.title::text').get(),
                'price': product.css('.price::text').get(),
                'url': product.css('a::attr(href)').get()
            }
        # Pagination automatique
        next_page = response.css('a.next::attr(href)').get()
        if next_page:
            yield response.follow(next_page, self.parse)

5Anti-detection : les techniques

Rotation User-Agent — varier les entetes HTTP pour ne pas etre identifie comme bot
Delais aleatoires — random.uniform(1, 3) entre les requetes. AutoThrottle dans Scrapy
Proxies rotatifs — distribuer les requetes sur des IPs differentes (Bright Data, Oxylabs)
429 Too Many Requests — respecter le header Retry-After, augmenter les delais
Premier reflexe — verifier si une API officielle existe. Le scraping est toujours un dernier recours

6Legislation : ce que tout scraper doit savoir

robots.txt — toujours consulter avant de scraper. Non juridiquement contraignant mais ethiquement obligatoire
CGU — certains sites interdisent explicitement le scraping. Violation possible = risque juridique
RGPD — scraper des donnees personnelles de citoyens europeens sans base legale = violation. Ex : LinkedIn (condamne a 500K$ dans l affaire hiQ)
Droit penal francais — extraction frauduleuse de donnees d un STAD est un delit (art. 323-3 Code penal)

import asyncio
from playwright.async_api import async_playwright
from bs4 import BeautifulSoup

async def scrape_dynamic_page(url: str) -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            user_agent="Mozilla/5.0 (compatible; DataCollector/1.0)",
            viewport={"width": 1280, "height": 720}
        )
        page = await context.new_page()
        
        # Intercepter les requêtes API (souvent plus propre que le scraping HTML)
        api_responses = []
        page.on("response", lambda r: api_responses.append(r) if "api/products" in r.url else None)
        
        await page.goto(url, wait_until="networkidle")
        await page.wait_for_selector(".product-list", timeout=10000)
        
        # Scroll pour charger le lazy loading
        await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        await asyncio.sleep(2)
        
        content = await page.content()
        await browser.close()
        
    soup = BeautifulSoup(content, "lxml")
    return {"products": [p.text for p in soup.select(".product-name")]}

Playwright vs Selenium — Playwright : async natif, plus rapide, meilleure gestion du JS moderne. Selenium : écosystème plus mature, plus de drivers disponibles
Intercepter les APIs — les sites chargent souvent des données via des appels API JSON. Intercepter ces requêtes avec Playwright est plus propre et plus stable que parser le HTML
Rate limiting — respecter un délai aléatoire entre requêtes (0.5-3s). Implémenter un backoff exponentiel sur les 429/503. Utiliser des proxies rotatifs si le site bloque
Stealth — playwright-stealth masque les indicateurs d'automatisation (navigator.webdriver, etc.). Indispensable pour les sites avec détection de bots
Stockage — sauvegarder le HTML brut en plus des données parsées. Permet de re-parser sans relancer le scraping si la logique d'extraction change

Playwright vs Selenium - Playwright : async natif, plus rapide, meilleure gestion du JS moderne. Selenium : ecosysteme plus mature, plus de drivers disponibles
Intercepter les APIs - les sites chargent souvent des donnees via des appels API JSON. Les intercepter avec Playwright est plus propre et stable que parser le HTML
Rate limiting respectueux - delai aleatoire entre requetes (0.5-3s). Backoff exponentiel sur les 429/503. Respecter robots.txt et les CGU du site
Stealth - playwright-stealth masque les indicateurs d automatisation (navigator.webdriver). Necessaire pour les sites avec detection de bots
Stockage du HTML brut - sauvegarder le HTML en plus des donnees parsees. Permet de re-parser sans relancer le scraping si la logique d extraction change

7Grille par niveau

Niveau	Maitrise	Signal GO	NO-GO
Junior	BeautifulSoup + requests, pagination, gestion erreurs	A scrape un site avec pagination, verifie robots.txt	Ne verifie pas robots.txt, ne gere pas les erreurs HTTP
Confirme	Playwright pour les SPA, Scrapy, anti-detection basique	Sait quand utiliser Playwright, a un projet Scrapy	Ne sait pas gerer les sites dynamiques JS
Senior	Proxies rotatifs, pipeline industrialise, aspects legaux maitrise	Cite spontanement les risques RGPD et la legislation, a deploye un scraper en production avec Airflow	Ne mentionne pas les aspects legaux

Scraping is a powerful tool but one that engages the developer's responsibility. In interviews, we don't just test technical skills — we also check knowledge of legal and ethical limits.

1Crawling, scraping, automation: the 3 concepts

Discriminating question

What is the difference between crawling and scraping? And web automation?

Crawling — automatically browsing a site to discover pages and links. This is what Google bots do. Goal: indexing
Scraping — extracting specific data from a web page. Goal: retrieving data for yourself
Automation — automatically reproducing human actions on a site (filling a form, clicking, downloading). Goal: replacing a repetitive task

2Python scraping ecosystem 2025

Tool	Static	Dynamic	Mass	Ideal for
BeautifulSoup + requests	Yes	No	No	Simple static pages, prototyping
Playwright	Yes	Yes	Medium	Dynamic JS sites, UI automation
Selenium	Yes	Yes	No	Dynamic sites, UI testing (slower than Playwright)
Scrapy	Yes	Partial	Yes	Mass scraping, production pipelines
httpx + asyncio	Yes	No	Yes	High-performance parallel requests

3Playwright: the modern successor to Selenium

Discriminating question

Why is Playwright preferred over Selenium in 2025? When do you use one or the other?

from playwright.async_api import async_playwright
import asyncio

async def scrape_spa():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        
        # Intercept network requests
        page.on('response', lambda r: print(r.url) if '/api/' in r.url else None)
        
        await page.goto('https://exemple.com/products')
        
        # Wait for data to be loaded
        await page.wait_for_selector('.product-card', timeout=10000)
        
        products = await page.query_selector_all('.product-card')
        data = []
        for p_elem in products:
            name = await p_elem.query_selector('.title')
            price = await p_elem.query_selector('.price')
            data.append({
                'name': await name.inner_text(),
                'price': await price.inner_text()
            })
        
        await browser.close()
        return data

asyncio.run(scrape_spa())

Playwright vs Selenium — Playwright is faster, supports native async, better timeout handling, supports all 3 engines (Chromium, Firefox, WebKit)
Network interception — if the app loads data via a JSON API, scraping it directly is 10x simpler than DOM scraping

4Scrapy: industrial scraping

import scrapy

class ProductSpider(scrapy.Spider):
    name = 'products'
    start_urls = ['https://exemple.com/catalogue']
    
    custom_settings = {
        'DOWNLOAD_DELAY': 1.5,       # politeness
        'ROBOTSTXT_OBEY': True,      # respect robots.txt
        'AUTOTHROTTLE_ENABLED': True, # automatic adaptation
        'ITEM_PIPELINES': {
            'myproject.pipelines.DatabasePipeline': 300,
        }
    }
    
    def parse(self, response):
        for product in response.css('.product-card'):
            yield {
                'name': product.css('.title::text').get(),
                'price': product.css('.price::text').get(),
                'url': product.css('a::attr(href)').get()
            }
        # Automatic pagination
        next_page = response.css('a.next::attr(href)').get()
        if next_page:
            yield response.follow(next_page, self.parse)

5Anti-detection: the techniques

User-Agent rotation — varying HTTP headers to avoid being identified as a bot
Random delays — random.uniform(1, 3) between requests. AutoThrottle in Scrapy
Rotating proxies — distributing requests across different IPs (Bright Data, Oxylabs)
429 Too Many Requests — respect the Retry-After header, increase delays
First reflex — check if an official API exists. Scraping is always a last resort

6Legislation: what every scraper must know

robots.txt — always check before scraping. Not legally binding but ethically mandatory
Terms of Service — some sites explicitly prohibit scraping. Possible violation = legal risk
GDPR — scraping personal data of European citizens without a legal basis = violation. Ex: LinkedIn (fined $500K in the hiQ case)
French criminal law — fraudulent extraction of data from an STAD is a criminal offense (art. 323-3 Penal Code)

import asyncio
from playwright.async_api import async_playwright
from bs4 import BeautifulSoup

async def scrape_dynamic_page(url: str) -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            user_agent="Mozilla/5.0 (compatible; DataCollector/1.0)


Vous recrutez un Data Engineer ?
Premier entretien gratuit. Rapport GO/NO-GO sous 48h.
Tester gratuitementReserver un appel