Question about MySQL Integrated‑CDC pipeline on Classic compute workspace

[david888]

Hi everyone,

I am trying to set up an Integrated‑CDC pipeline for a MySQL RDS instance on our Databricks workspace. Our workspace only uses Classic compute; Serverless compute is not available.

According to the documentation, Integrated‑CDC for MySQL should support Classic compute. However, I cannot find the UI wizard for creating this MySQL CDC pipeline inside Catalog Explorer. There is no graphical creation page.

I understand that the pipeline can be created via REST API / Databricks SDK from a Notebook. Before I proceed to build the pipeline with SDK scripts, I would like to confirm:

1. Is there any workspace‑level feature flag or preview access that needs to be enabled on our account to use MySQL Integrated‑CDC, even when we only use Classic compute?
2. If the UI wizard is intentionally not available for Classic‑only workspaces, is the API‑only deployment officially supported for production usage for MySQL RDS sources?
3. Are there any known networking prerequisites I should double‑check between Databricks Classic cluster and the MySQL RDS database for the binlog‑based CDC ingestion?

Thank you for your help.

[ivanvyd]

@david888 the missing wizard is expected. MySQL Integrated CDC currently runs on classic compute only, and UI-based pipeline creation isn't available. Workspace-level feature enablement is required; the docs direct you to your Databricks account team.

REST, CLI, notebook SDK and Bundles are the documented creation paths. Create the Unity Catalog connection first and include "channel": "PREVIEW" in the pipeline specification.

For your production question, Integrated CDC is currently Beta, and Databricks states that Beta features are not intended for production use. API-based creation is documented, but that doesn't change the feature's support status.

For RDS, verify network routing from the pipeline’s classic compute to the database and security-group rules allowing its configured MySQL port, normally TCP 3306 (?). Restrictive network ACLs must also allow the return traffic.

On the source, check binlog_format=ROW, binlog_row_image=FULL and the required ingestion-user privileges. For RDS MySQL instances, automated backups enable binary logging; binlog retention is configured separately, and Databricks recommends seven days.

[Louis_Frolio]

Hello @david888 , I took a look at both internal and external documentation and here is what I found. Ivan (@ivanvyd) has this right, so I'll build on his answer rather than repeat it.

Taking your three questions in order.

1. Feature flag. Yes, enablement is required. The integrated CDC connector for MySQL is in Beta, and Beta features are controlled per workspace from the Previews page (Settings > Previews, admin only). On Premium workspaces, Beta features are generally on by default; on Enterprise, they're off by default. So have a workspace admin check that page first, and contact your Databricks account team only if the connector isn't listed there. A quick tell: if API creation fails with Integrated CDC connector is disabled, that's the workspace flag and nothing else.

2. UI and production. The missing wizard isn't about your workspace being Classic-only. UI creation isn't available for anyone yet; the documented paths are the REST API, Databricks CLI, a notebook with the SDK, or Declarative Automation Bundles. Create the Unity Catalog connection first, and every pipeline spec needs "channel": "PREVIEW" and "connector_type": "CDC". On production, the distinction to hold onto is "API-supported creation" versus "production-ready support." The API path is documented, but the connector is still Beta, and Databricks defines Beta as not for production, no SLA, and supported by the engineering team rather than the support team. One wrinkle that matters for you: the alternative, the gateway-based MySQL connector, is in Public Preview and production-ready, but it runs its ingestion pipeline on serverless. With no serverless in your workspace, that path isn't open either. So the integrated pipeline is realistically your only Lakeflow Connect route for MySQL CDC today, Beta label included. Validate the intended support level with your account team, and before you depend on it, test restart and recovery, source lag, schema changes, and binlog-expiry behavior yourself.

3. Networking and source prerequisites. With classic compute, the pipeline's cluster runs inside your workspace VPC, so the question is whether that VPC can reach RDS. Verify:

• DNS resolution and a route from the cluster subnets to the RDS endpoint.
• RDS security groups and network ACLs allow TCP 3306 (or your custom port), including return traffic.
• Automated backups are enabled with a nonzero retention; in RDS that's what turns on log_bin, since the parameter itself is read-only.
• binlog_format=ROW and binlog_row_image=FULL set in a custom parameter group.
• Binlog retention set separately: CALL mysql.rds_set_configuration('binlog retention hours', 168);. Minimum is 24 hours, Databricks recommends 168, and it should cover your longest expected pipeline outage. If RDS purges binlog before a pipeline update processes it, the only fix is a full refresh of the affected tables.
• The ingestion user has the privileges in the docs, and your tables have primary keys. Tables without one get treated as a composite key across all non-LOB columns, and duplicate rows can collapse into one.

Easiest sanity check before writing any SDK code: create the Unity Catalog connection in Catalog Explorer and click Test connection. If that passes, networking and credentials are sorted.

Two more first-build gotchas: the pipeline needs CREATE VOLUME on the destination schema (or the schema in data_staging_options) because it auto-creates a staging volume, and it runs in triggered mode, so schedule it with a Lakeflow Jobs task. The docs suggest 60 minutes as a starting interval.

Short version: you're reading the docs correctly. Confirm the preview is enabled, create and test the connection, deploy through the API with channel: PREVIEW, lock in binlog retention, and have the production conversation with your account team before a critical workload rides on it.

References:

• Create an integrated CDC pipeline for MySQL: https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/mysql-integrated-pipeline
• Configure MySQL for ingestion (source setup): https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/mysql-source-setup
• Configure Amazon RDS and Aurora MySQL (binlog and retention): https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/mysql-aws-rds-config
• Grant MySQL user privileges: https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/mysql-privileges
• Create a MySQL connection in Unity Catalog: https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/mysql-connection
• Manage Databricks previews: https://docs.databricks.com/aws/en/admin/workspace-settings/manage-previews
• Databricks preview release types: https://docs.databricks.com/aws/en/release-notes/release-types
• AWS: Configuring RDS for MySQL binary logging: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.MySQL.BinaryFormat.html

Regards, Louis.

[tracy74crawford]

Yes, workspace-level preview access or specific feature flags may be required for MySQL Integrated-CDC depending on your Databricks account tier, and while API-only deployments via the Databricks SDK are fully supported for production, you must ensure that your Classic cluster has outbound TCP access to your MySQL RDS port (usually 3306) and that binary logging (binlog) with row-based replication (binlog_format = ROW) is explicitly enabled on your RDS parameter group.