<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic Re: Issue with CDC ingestion pipeline in Data Engineering</title>
    <link>https://community.databricks.com/t5/data-engineering/issue-with-cdc-ingestion-pipeline/m-p/176504#M57160</link>
    <description>&lt;P&gt;Thank you all for your assistance.&amp;nbsp;&lt;/P&gt;&lt;P&gt;The staging catalog does have a&amp;nbsp; correct storage location configured (on S3). The pipeline created a cdc logs catalog (with attached volume) and has written some files. The staging catalog also has other schema's that are in use (test workspace).&amp;nbsp;&lt;/P&gt;&lt;P&gt;The metastore in use however does&amp;nbsp;&lt;STRONG&gt;not&lt;/STRONG&gt; have a storage location configured. Can this be the issue?&amp;nbsp;&lt;/P&gt;&lt;P&gt;Are there any risks involved with configuring an S3 storage location for an existing metastore?&amp;nbsp;&lt;/P&gt;</description>
    <pubDate>Mon, 16 Nov 2026 11:56:02 GMT</pubDate>
    <dc:creator>MichaelAnckaert</dc:creator>
    <dc:date>2026-11-16T11:56:02Z</dc:date>
    <item>
      <title>Issue with CDC ingestion pipeline</title>
      <link>https://community.databricks.com/t5/data-engineering/issue-with-cdc-ingestion-pipeline/m-p/176354#M57126</link>
      <description>&lt;P&gt;Hello everyone&lt;/P&gt;&lt;P&gt;I'm trying to setup an ingestion pipeline from a mysql server and keep running into an issue.&amp;nbsp;&lt;/P&gt;&lt;P&gt;The setup was validated and the pipeline created. When I run the pipeline the destination tables are created in the schema, but then the following error is displayed:&lt;/P&gt;&lt;P&gt;Error class: INVALID_PARAMETER_VALUE&lt;BR /&gt;Message: Input URI is not a valid UC Path: s3:/&lt;/P&gt;&lt;P&gt;The pipeline stops after this without further information.&amp;nbsp;&lt;/P&gt;&lt;P&gt;I would appreciate some information or pointers on how I can resolve this issue.&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Fri, 13 Nov 2026 15:18:05 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/issue-with-cdc-ingestion-pipeline/m-p/176354#M57126</guid>
      <dc:creator>MichaelAnckaert</dc:creator>
      <dc:date>2026-11-13T15:18:05Z</dc:date>
    </item>
    <item>
      <title>Re: Issue with CDC ingestion pipeline</title>
      <link>https://community.databricks.com/t5/data-engineering/issue-with-cdc-ingestion-pipeline/m-p/176369#M57133</link>
      <description>&lt;P&gt;Hi &lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/271661"&gt;@MichaelAnckaert&lt;/a&gt;,&lt;/P&gt;&lt;P&gt;The interesting part is the path in the error: s3:/ with nothing after it. That isn't coming from MySQL. It looks like the pipeline tried to resolve a storage location in Unity Catalog and got back an empty S3 path, and since the destination tables were already created by then, my first suspect is the staging location the connector writes to.&lt;/P&gt;&lt;P&gt;Three things I'd check:&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;1. Where the staging catalog and schema actually store data.&lt;/STRONG&gt;&lt;/P&gt;&lt;PRE&gt;DESCRIBE CATALOG EXTENDED &amp;lt;staging_catalog&amp;gt;;
DESCRIBE SCHEMA EXTENDED &amp;lt;staging_catalog&amp;gt;.&amp;lt;staging_schema&amp;gt;;&lt;/PRE&gt;&lt;P&gt;Look at the storage root or location in the output. If neither the catalog nor the schema has one, and the metastore has no root storage either, there's nowhere for the staging volume to live.&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;2. Default storage.&lt;/STRONG&gt; If the staging catalog was created on default storage (the option you get in a serverless workspace), the docs say it can only be accessed from serverless compute. The MySQL ingestion gateway runs on classic job compute, so it can't write there. Pointing staging at a catalog or schema with a managed location on your own S3 bucket (through an external location) gets around that.&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;3. Privileges on the staging schema.&lt;/STRONG&gt; You need USE SCHEMA, CREATE TABLE and CREATE VOLUME there, and the troubleshooting page also notes that the staging catalog can't be a foreign catalog.&lt;/P&gt;&lt;P&gt;If you'd rather stay fully serverless, the newer integrated CDC pipeline for MySQL can run on serverless compute, as long as serverless can reach your MySQL instance over the network.&lt;/P&gt;&lt;P&gt;If none of that matches, the pipeline's event log should have the full error with a stack trace. Post it here and I'm happy to take another look.&lt;/P&gt;&lt;P&gt;Cheers,&lt;BR /&gt;Suranga&lt;/P&gt;</description>
      <pubDate>Fri, 13 Nov 2026 21:13:21 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/issue-with-cdc-ingestion-pipeline/m-p/176369#M57133</guid>
      <dc:creator>suranga</dc:creator>
      <dc:date>2026-11-13T21:13:21Z</dc:date>
    </item>
    <item>
      <title>Re: Issue with CDC ingestion pipeline</title>
      <link>https://community.databricks.com/t5/data-engineering/issue-with-cdc-ingestion-pipeline/m-p/176442#M57144</link>
      <description>&lt;P&gt;Hi Michael,&lt;/P&gt;&lt;P&gt;I’ve seen this error with Lakeflow Connect database connectors. The s3:/ with nothing after it suggests the pipeline couldn’t work out a storage location for its staging data.&lt;/P&gt;&lt;P&gt;The ingestion gateway first writes the change data to a Unity Catalog volume in the staging catalog and schema you chose during setup. If that catalog or schema has no managed storage location, or the location is set up wrong, the path comes out empty and you get this error.&lt;/P&gt;&lt;P&gt;Things worth checking:&lt;/P&gt;&lt;UL&gt;&lt;LI&gt;&lt;STRONG&gt;Staging catalog/schema storage:&lt;/STRONG&gt; Make sure the staging catalog (or schema) has a valid managed location backed by an external location, or that your metastore has a root storage location.&lt;/LI&gt;&lt;LI&gt;&lt;STRONG&gt;Permissions:&lt;/STRONG&gt; The user or service principal running the pipeline needs rights on the external location and storage credential, plus CREATE VOLUME on the staging schema.&lt;/LI&gt;&lt;LI&gt;&lt;STRONG&gt;Default storage catalogs:&lt;/STRONG&gt; If the staging catalog uses default storage, try a catalog with your own S3 managed location instead.&lt;/LI&gt;&lt;LI&gt;&lt;STRONG&gt;Event log:&lt;/STRONG&gt; Look at the gateway pipeline’s event log for a more detailed error than the main pipeline shows.&lt;/LI&gt;&lt;/UL&gt;&lt;P&gt;If you point the gateway at a staging catalog/schema with a proper S3 managed location and run it again, it should get past this step.&lt;/P&gt;&lt;P&gt;Hope this helps!&lt;/P&gt;</description>
      <pubDate>Sun, 15 Nov 2026 14:14:19 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/issue-with-cdc-ingestion-pipeline/m-p/176442#M57144</guid>
      <dc:creator>rajkumar8k</dc:creator>
      <dc:date>2026-11-15T14:14:19Z</dc:date>
    </item>
    <item>
      <title>Re: Issue with CDC ingestion pipeline</title>
      <link>https://community.databricks.com/t5/data-engineering/issue-with-cdc-ingestion-pipeline/m-p/176451#M57146</link>
      <description>&lt;P&gt;Hi&amp;nbsp;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/271661"&gt;@MichaelAnckaert&lt;/a&gt;&amp;nbsp;,&lt;/P&gt;&lt;P&gt;&lt;SPAN class=""&gt;The error &lt;/SPAN&gt;Input URI is not a valid UC Path: s3:/&lt;SPAN&gt; indicates that Unity Catalog is trying to physically create the destination tables (or the pipeline staging area) in AWS S3,&lt;/SPAN&gt;&lt;SPAN&gt; but the storage path it has been handed is malformed.&lt;/SPAN&gt;&lt;SPAN&gt; A valid AWS S3 path must start with &lt;/SPAN&gt;s3://&amp;lt;bucket-name&amp;gt;/...&lt;SPAN&gt;,&lt;/SPAN&gt;&lt;SPAN&gt; but yours is being evaluated as just &lt;/SPAN&gt;s3:/&lt;SPAN&gt; (missing the second slash and the bucket name).&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Because the setup validation passed,&lt;/SPAN&gt;&lt;SPAN&gt; the Lakeflow connector itself is working fine.&lt;/SPAN&gt;&lt;SPAN&gt; The pipeline is simply inheriting a broken default storage path from your Unity Catalog setup when it attempts to write the actual data.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Here are the exact steps to find and fix the typo:&lt;/SPAN&gt;&lt;/P&gt;&lt;H3&gt;1. Inspect the Target Schema and Catalog Storage Locations&lt;/H3&gt;&lt;P&gt;&lt;SPAN&gt;When Lakeflow Connect creates managed tables,&lt;/SPAN&gt;&lt;SPAN&gt; it stores the data in the default storage location of the target schema.&lt;/SPAN&gt;&lt;SPAN&gt; If the schema doesn't have one,&lt;/SPAN&gt;&lt;SPAN&gt; it falls back to the Catalog's storage location.&lt;/SPAN&gt;&lt;/P&gt;&lt;OL&gt;&lt;LI&gt;&lt;P&gt;&lt;SPAN&gt;Open &lt;/SPAN&gt;&lt;STRONG&gt;Catalog Explorer&lt;/STRONG&gt;&lt;SPAN&gt; in your Databricks workspace.&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;LI&gt;&lt;P&gt;&lt;SPAN&gt;Navigate to the &lt;/SPAN&gt;&lt;STRONG&gt;Target Catalog&lt;/STRONG&gt;&lt;SPAN&gt; and the &lt;/SPAN&gt;&lt;STRONG&gt;Target Schema&lt;/STRONG&gt;&lt;SPAN&gt; you specified for the pipeline.&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;LI&gt;&lt;P&gt;&lt;SPAN&gt;On both the Schema and Catalog pages,&lt;/SPAN&gt;&lt;SPAN&gt; click the &lt;/SPAN&gt;&lt;STRONG&gt;Details&lt;/STRONG&gt;&lt;SPAN&gt; tab.&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;LI&gt;&lt;P&gt;&lt;SPAN&gt;Check the &lt;/SPAN&gt;&lt;STRONG&gt;Storage Location&lt;/STRONG&gt;&lt;SPAN&gt; property.&lt;/SPAN&gt;&lt;I&gt;Verification:&lt;/I&gt;&lt;SPAN&gt; If you see a path starting with &lt;/SPAN&gt;&lt;STRONG&gt;s3:/&lt;/STRONG&gt;&lt;SPAN&gt; instead of &lt;/SPAN&gt;&lt;STRONG&gt;s3://&lt;/STRONG&gt;&lt;SPAN&gt;,&lt;/SPAN&gt;&lt;SPAN&gt; this is the culprit.&lt;/SPAN&gt;&lt;SPAN&gt; You will need to recreate the schema (or catalog) with a properly formatted S3 URI,&lt;/SPAN&gt;&lt;SPAN&gt; or choose a schema that has a valid storage location.&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;/OL&gt;&lt;H3&gt;2. Verify the Metastore Root Storage&lt;/H3&gt;&lt;P&gt;&lt;SPAN&gt;If your target Catalog and Schema do not have explicit storage locations defined,&lt;/SPAN&gt;&lt;SPAN&gt; Unity Catalog will attempt to use the Metastore's root storage.&lt;/SPAN&gt;&lt;/P&gt;&lt;OL&gt;&lt;LI&gt;&lt;P&gt;&lt;SPAN&gt;If you have admin privileges,&lt;/SPAN&gt;&lt;SPAN&gt; click your username in the top right -&amp;gt; &lt;/SPAN&gt;&lt;STRONG&gt;Manage Account&lt;/STRONG&gt;&lt;SPAN&gt; -&amp;gt; &lt;/SPAN&gt;&lt;STRONG&gt;Data&lt;/STRONG&gt;&lt;SPAN&gt; (or check your workspace admin settings).&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;LI&gt;&lt;P&gt;&lt;SPAN&gt;Look at the Metastore's &lt;/SPAN&gt;&lt;STRONG&gt;Root Storage&lt;/STRONG&gt;&lt;SPAN&gt; path.&lt;/SPAN&gt;&lt;I&gt;Verification:&lt;/I&gt;&lt;SPAN&gt; Confirm it is formatted correctly with &lt;/SPAN&gt;&lt;STRONG&gt;s3://&lt;/STRONG&gt;&lt;SPAN&gt; and a valid bucket name.&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;/OL&gt;&lt;H3&gt;3. Check Staging Volume Configurations&lt;/H3&gt;&lt;P&gt;&lt;SPAN&gt;Lakeflow Connect relies on a staging volume for CDC ingestion.&lt;/SPAN&gt;&lt;/P&gt;&lt;OL&gt;&lt;LI&gt;&lt;P&gt;&lt;SPAN&gt;If you explicitly selected a volume for staging,&lt;/SPAN&gt;&lt;SPAN&gt; locate that Volume in Catalog Explorer.&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;LI&gt;&lt;P&gt;&lt;SPAN&gt;Check its &lt;/SPAN&gt;&lt;STRONG&gt;Storage Location&lt;/STRONG&gt;&lt;SPAN&gt; under the details tab.&lt;/SPAN&gt;&lt;I&gt;Verification:&lt;/I&gt;&lt;SPAN&gt; Ensure it is a valid,&lt;/SPAN&gt;&lt;SPAN&gt; fully qualified S3 URI.&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;/OL&gt;&lt;P&gt;&lt;SPAN&gt;Take a look at those Catalog Explorer properties and you will likely spot the malformed path immediately.&lt;/SPAN&gt;&lt;SPAN&gt; If you check those and they all look correct,&lt;/SPAN&gt;&lt;SPAN&gt; please reply back with:&lt;/SPAN&gt;&lt;/P&gt;&lt;UL&gt;&lt;LI&gt;&lt;P&gt;&lt;SPAN&gt;Are you writing to a Managed Schema or an External Schema?&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;LI&gt;&lt;P&gt;&lt;SPAN&gt;Are you letting Databricks create the staging volume automatically,&lt;/SPAN&gt;&lt;SPAN&gt; or did you specify an existing one?&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;/UL&gt;&lt;P&gt;&lt;SPAN&gt;We will get this pipeline running for you!&lt;/SPAN&gt;&lt;/P&gt;</description>
      <pubDate>Sun, 15 Nov 2026 20:56:25 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/issue-with-cdc-ingestion-pipeline/m-p/176451#M57146</guid>
      <dc:creator>ShamenParis</dc:creator>
      <dc:date>2026-11-15T20:56:25Z</dc:date>
    </item>
    <item>
      <title>Re: Issue with CDC ingestion pipeline</title>
      <link>https://community.databricks.com/t5/data-engineering/issue-with-cdc-ingestion-pipeline/m-p/176504#M57160</link>
      <description>&lt;P&gt;Thank you all for your assistance.&amp;nbsp;&lt;/P&gt;&lt;P&gt;The staging catalog does have a&amp;nbsp; correct storage location configured (on S3). The pipeline created a cdc logs catalog (with attached volume) and has written some files. The staging catalog also has other schema's that are in use (test workspace).&amp;nbsp;&lt;/P&gt;&lt;P&gt;The metastore in use however does&amp;nbsp;&lt;STRONG&gt;not&lt;/STRONG&gt; have a storage location configured. Can this be the issue?&amp;nbsp;&lt;/P&gt;&lt;P&gt;Are there any risks involved with configuring an S3 storage location for an existing metastore?&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Mon, 16 Nov 2026 11:56:02 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/issue-with-cdc-ingestion-pipeline/m-p/176504#M57160</guid>
      <dc:creator>MichaelAnckaert</dc:creator>
      <dc:date>2026-11-16T11:56:02Z</dc:date>
    </item>
    <item>
      <title>Re: Issue with CDC ingestion pipeline</title>
      <link>https://community.databricks.com/t5/data-engineering/issue-with-cdc-ingestion-pipeline/m-p/176573#M57173</link>
      <description>&lt;P&gt;Thanks for checking the staging side. Yes, it could be the issue. An empty bucket in "s3:/" looks like a storage root that resolved to nothing, and Unity Catalog only falls back to the metastore when neither the schema nor the catalog has a managed location. With no metastore storage, that fallback is empty.&lt;/P&gt;&lt;P&gt;Since the staging catalog is fine, I'd check the other places the pipeline writes: the destination catalog and schema, and the catalog and schema of the event log table if you publish one.&lt;/P&gt;&lt;PRE&gt;DESCRIBE CATALOG EXTENDED &amp;lt;destination_catalog&amp;gt;;
DESCRIBE SCHEMA EXTENDED &amp;lt;destination_catalog&amp;gt;.&amp;lt;destination_schema&amp;gt;;&lt;/PRE&gt;&lt;P&gt;If one of them has no storage location, I'd fix it there rather than at the metastore. Databricks recommends catalog-level storage for isolation anyway:&lt;/P&gt;&lt;PRE&gt;ALTER CATALOG &amp;lt;destination_catalog&amp;gt; SET MANAGED LOCATION 's3://&amp;lt;bucket&amp;gt;/&amp;lt;path&amp;gt;';&lt;/PRE&gt;&lt;P&gt;That needs Databricks SQL or Databricks Runtime 18.1 and above, plus an external location covering the path with CREATE MANAGED STORAGE. It only applies to new managed tables and volumes. Existing ones aren't moved.&lt;/P&gt;&lt;P&gt;Adding storage to the existing metastore is supported too. A few things to know first:&lt;/P&gt;&lt;UL&gt;&lt;LI&gt;It applies to every workspace attached to that metastore, not only your test one. Any catalog without its own location will start putting new managed tables and volumes there.&lt;/LI&gt;&lt;LI&gt;You need a dedicated S3 bucket in the metastore's region, an external location with CREATE MANAGED STORAGE, and an account admin to set it in the account console.&lt;/LI&gt;&lt;LI&gt;You can't change the path once it's set, but you can remove it and add a new one.&lt;/LI&gt;&lt;LI&gt;If you remove it later, catalogs that relied on it keep that path as their own storage root.&lt;/LI&gt;&lt;/UL&gt;&lt;P&gt;Docs: &lt;A href="https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/managed-storage" target="_blank" rel="noopener"&gt;Managed storage locations&lt;/A&gt; and &lt;A href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-metastore" target="_blank" rel="noopener"&gt;adding metastore-level storage&lt;/A&gt;&lt;/P&gt;</description>
      <pubDate>Mon, 16 Nov 2026 17:38:24 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/issue-with-cdc-ingestion-pipeline/m-p/176573#M57173</guid>
      <dc:creator>suranga</dc:creator>
      <dc:date>2026-11-16T17:38:24Z</dc:date>
    </item>
  </channel>
</rss>

