Checkout.com has migrated its data orchestration platform to Google Cloud's Managed Service for Apache Airflow, a move that changed the reliability and cost profile of its data platform.
The payments company moved away from a self-managed Apache Airflow setup on another hyperscaler, where its data platform team handled server management, patching, upgrades, and incident response alongside day-to-day pipeline work.
That arrangement also created strain during periods of heavy use. Checkout.com said synchronising DAGs to the scheduler after deployment to Amazon S3 took about six minutes, while scaling the environment and onboarding new teams required manual steps.
Operational load
For data engineering teams, Airflow is often central to pipeline scheduling and workflow management. At Checkout.com, maintaining that core internally had become a drain on engineering time the team wanted to spend elsewhere.
In the old environment, package upgrades and dependency compatibility checks were recurring manual tasks. Checkout.com also cited stability problems, including the risk that one problematic DAG could affect the wider environment.
Google Cloud's managed Airflow service changes that model by shifting patching and upgrades to the provider and isolating DAG execution environments. According to Checkout.com, this reduced the chance that a single failed or resource-heavy workflow would disrupt the rest of the system.
Cost change
A central part of the migration was moving away from fixed provisioning. In the previous elastic container service setup, Checkout.com provisioned for peak worker demand and paid for that capacity continuously, regardless of actual workload.
With the managed service, worker numbers adjust automatically in line with demand. The company estimated that moving from fixed provisioning to dynamic scaling cut monthly costs by about 30%.
That cost reduction came alongside changes in day-to-day operations. Engineers no longer need to manage infrastructure resources directly, and scheduled maintenance windows have replaced manual upgrade management.
Developer workflow
Checkout.com said the migration also improved deployment and onboarding for data engineers. By storing DAGs in Cloud Storage, synchronisation became near-instant, reducing the wait time between deployment and execution.
Onboarding for teams using dbt also changed. Under the previous setup, engineers had to install and maintain separate virtual environments for different dbt versions, adding complexity whenever a new version or dependency set was needed.
The new setup uses containerised dbt runs, giving teams isolated dependencies for each model without the same level of manual setup. Platform staff also no longer need to redeploy the entire Airflow environment to add new roles or update Python packages.
Monitoring and troubleshooting were another focus of the migration. Integration with Google Cloud Monitoring and Cloud Logging gave engineers clearer visibility into task execution and let them investigate issues without always escalating them to the platform team.
Checkout.com also described using Gemini Cloud Assist within the Airflow DAG interface to investigate failed tasks. The tool generates a scorecard that weighs different explanations for errors using supporting and contradictory evidence, helping reduce the time needed to restore services.
The migration reflects a broader trend in cloud data engineering as companies reassess whether core orchestration tools should remain self-hosted or be handed to managed service providers. The trade-off often centres on control versus operating burden, especially for teams trying to reduce time spent on maintenance.
For Checkout.com, the case for change appears to have rested on a combination of reliability, speed, and cost. It framed the project as a way to free engineers from infrastructure work and return their focus to data product development.
"With Managed Service for Apache Airflow, we've achieved significant improvements in efficiency, scalability, and reliability. Managed infrastructure, automated scaling, faster deployments, and isolated execution environments have transformed how we operate," said Keisi Mancellari, Data Platform Engineer at Checkout.com.