Apr 15, 2026 • Project

SpringBatch-ETL: migrating 73 million rows in 40 minutes

Java 21 / Spring Batch ETL for the PharmaGDD v1 to v2 migration: 73M MySQL rows at ~1.8M rows/min, PHP-serialized columns decoded by a Swoole daemon over TCP, replayed nightly by GitHub Actions.

Java 21Spring BatchSwooleMySQLDockerGitHub Actions

Context: The PharmaGDD rebuild (v1 Symfony to v2 Sylius) requires migrating the whole database, including columns stored in PHP's serialized format.
My role: Designed and built the ETL, then its pipeline: Docker images, nightly runs on a cloud VM, integrity checks.
Outcome: 73 million rows migrated in 40 minutes (~1.8M rows/min), replayed every night on the production dump.

The context

PharmaGDD is moving from its v1 (legacy Symfony) to a v2 built on Sylius. The schemas differ, and part of the data is stored in columns serialized with PHP’s serialize(), which Java cannot read as is.

We needed a tool able to migrate 73 million rows in a reasonable time, and to do it again as many times as needed before the cutover.

Decoding PHP from Java

This constraint drove the architecture.

One PHP script per row, run from the command line: ruled out immediately. Booting the interpreter on every call makes it unusable at this scale.
A PHP HTTP API: my first idea. But for millions of small calls, the cost of HTTP (headers, connections) outweighs the deserialization itself.
A Swoole daemon over TCP: the chosen solution. A PHP container runs permanently, keeps the connection open and answers in JSON. Java sends it the raw value and gets back a usable structure.

The pipeline

The ETL is written in Java 21 with Spring Batch, which splits the work into transactional chunks.

A processor chain: the first one sends the value to the Swoole daemon and gets the JSON back, the next ones handle column mapping and locale conversions.
Several tables in parallel through a thread pool (ThreadPoolTaskExecutor).
A generic job: tables and column renames (for example owner_id → product_id) are declared in application.yml, and the job is built at startup. Adding a table needs no code.
MySQL tuning: innodb_buffer_pool_size and max_allowed_packet adjusted to absorb bulk writes.

The result: about 1.8 million rows per minute, so 40 minutes for the whole run.

Replayed every night

A migration is not something you run once. To reach the cutover with a proven tool, the ETL runs every night on a cloud VM, orchestrated by GitHub Actions on a self-hosted runner:

Download the day’s production dump from an OVH S3 bucket, then import it into a source MySQL.
Run the ETL from Docker images (Java ETL and Swoole daemon), built by CI and pushed to the OVH registry.
Integrity checks on the target database: queries count orphaned rows (foreign keys without a parent). A non-zero result fails the run.
Clean up containers and data.

These checks catch, for example, a parent table missing from the configuration: the children are migrated, the parent is not, and the error shows up the next morning instead of days later.

What I take away

The choice of protocol matters as much as the code: moving from HTTP to TCP was the real unlock.
A data migration should be treated like software: versioned, packaged, replayed and checked automatically.
Making the job generic from the start saved writing one job per table.