Deprecation of Docker Compose

[nataliekwong]

Summary

Today, we are announcing that Docker Compose will be deprecated from future versions of Airbyte OSS after version 1.0 is released (expected in early August 2024).

Support for Docker Compose and running Airbyte via the run_ab_platform.sh script will end in August 2024.

What we saw

• Our users frequently experience issues when upgrading on Docker Compose, as it is very brittle to upgrade and can often result in Airbyte not restarting correctly.
• The path from a local deployment to a production is painful and requires manual mapping work. Our users have difficulty understanding the mapping of the Helm chart to Docker Compose environment variables, or config.

As Airbyte matures, we need to align on the best path forward for our users and set you up for success with Airbyte. Similar open-source companies provide Docker Compose as a deployment method for local testing. When it comes to production-readiness, Kubernetes has become the de-facto standard in the industry. Earlier this year, we also shared a survey to understand how we could improve the maintenance and upgrade experience for our own community. From your responses, we found:

• The majority of respondents deploy to production using Kubernetes
• 71% of the respondents were already familiar with Kubernetes, with a significant majority experienced in Helm.

With this data in hand, our team prioritized building a new local deployment tool, abctl, that was designed to deploy Airbyte locally using Kubernetes with minimal onboarding friction. abctl allows users to test what a production deployment would look like without needing to provide your own k8s cluster.

Next Steps

Our overall goal is to build a best-in-class tool to improve the initial user experience and ensure users are set up for a production-ready instance of Airbyte from the start, and require minimal migration.

Since our launch of abctl two months ago, we’ve observed the following signals from the adoption so far:

• 1.5x success rate of installation
• 2x more users can successfully start syncing data
• 1.5x sync speed improvement

Moving forward, as we move to deprecate Docker Compose, we want to make sure Docker Compose users can adequately migrate with enough notice and time to use abctl. To this end, we’ve created a migration path that we are beginning to test with external users. We are asking for volunteers from the Community to help ensure our migration paths can adequately cover the ways in which users currently use Docker Compose. This beta group of testers will work directly with our Engineering team to migrate existing Airbyte deployments to abctl.

To sign up for the beta group, please fill out the survey HERE.

We look forward to hearing from you!

[marcraminv]

This means all of these options will be deprecated, right?

[phammer]

Same question from my side.

[bgroff]

For ECS, we do not currently support it and do not have and plans on the roadmap to support ECS. For EC2, we have documentation here: https://docs.airbyte.com/deploying-airbyte/infrastructure/aws#using-an-ec2-instance-with-abctl that explains how to use abctl on an EC2 machine.

[jadetr]

We are hosting airbyte on a ubuntu server with portainer. Traefik is acting as a reverse proxy (handling ssh, authentication.) Traefik is configured right into the docker-compose file. Any hints on the recommended upgrade path for such a setup?

[marcosmarxm]

Hello @jadetr, thanks for sharing your setup. The team is on a migration tool. I'll share with them to see if it's possible to use the current migration tool.

[bgroff]

You can read the documentation on the migration path here: https://docs.airbyte.com/deploying-airbyte/infrastructure/aws#using-an-ec2-instance-with-abctl

Currently looking into how Traefik will work as an Ingress for abctl.

[ygrekconsutling]

Hi, just tried it on a fresh install on debian and got this error:

  INFO    Using Kubernetes provider:
            Provider: kind
            Kubeconfig: /home/foo/.airbyte/abctl/abctl.kubeconfig
            Context: kind-airbyte-abctl
 SUCCESS  Found Docker installation: version 27.0.3
 SUCCESS  Port 8000 appears to be available
  INFO    No existing cluster found, cluster 'airbyte-abctl' will be created
 SUCCESS  Cluster 'airbyte-abctl' created
  INFO    Namespace 'airbyte-abctl' created
  INFO    Persistent volume 'airbyte-minio-pv' created
  INFO    Persistent volume 'airbyte-volume-db' created
  INFO    Persistent volume claim 'airbyte-minio-pv-claim-airbyte-minio-0' created
  INFO    Persistent volume claim 'airbyte-volume-db-airbyte-db-0' created
  ERROR   Unable to fetch airbyte/airbyte Helm Chart
  ERROR   Unable to install Airbyte locally
  ERROR   could not install airbyte chart: could not fetch chart airbyte/airbyte: Chart.yaml file is missing

since i'm using the new install format, what's wrong and how to debug it if needed ?

[ygrekconsutling]

Finally installed the version 0.5and it worked

[whanata]

I'm currently hosting Airbyte using Docker Compose at this moment also and changes we need to make was around running Open Telemetry Collector in Docker Containers also etc.

Would be interested to make sure that using the ABCTL tool, we can still get Open Telemetry to function properly, or a pathway to use Open Telemetry to get some of the metrics from Airbyte.

We currently don't have much experience using Kubernetes, so we plan to run this using ABCTL at least for the time being in an EC2.

[bgroff]

There is documentation here for how to install Airbyte using abctl on an EC2 machine here: https://docs.airbyte.com/deploying-airbyte/infrastructure/aws#using-an-ec2-instance-with-abctl

[aviad-deri]

Hi,
we are also using OSS with docker compose.
we want to help with the migration to abctl testing.
please add us to this and tell us how to migrate.
:)

[whanata]

I've also just filled in the form for beta testing, but looking into abctl in the new few weeks to see what's possible around that.

[bgroff]

There is documentation on the migration here: https://docs.airbyte.com/using-airbyte/getting-started/oss-quickstart#migrating-from-docker-compose-optional

[honggyu-rr]

Can we get more documentation on the migration process and what it's actually doing? E.g. will it somehow integrate my .env variables as part of the process?

[anzelpwj-rev]

Signed up for the beta test. I've been migrating our Airbyte setup to Terraform - will there be any changes to how Terraform provisions Airbyte, or will that be unaffected?