Friday
Hello everyone
I'm trying to setup an ingestion pipeline from a mysql server and keep running into an issue.
The setup was validated and the pipeline created. When I run the pipeline the destination tables are created in the schema, but then the following error is displayed:
Error class: INVALID_PARAMETER_VALUE
Message: Input URI is not a valid UC Path: s3:/
The pipeline stops after this without further information.
I would appreciate some information or pointers on how I can resolve this issue.
Friday
Hi @MichaelAnckaert,
The interesting part is the path in the error: s3:/ with nothing after it. That isn't coming from MySQL. It looks like the pipeline tried to resolve a storage location in Unity Catalog and got back an empty S3 path, and since the destination tables were already created by then, my first suspect is the staging location the connector writes to.
Three things I'd check:
1. Where the staging catalog and schema actually store data.
DESCRIBE CATALOG EXTENDED <staging_catalog>; DESCRIBE SCHEMA EXTENDED <staging_catalog>.<staging_schema>;
Look at the storage root or location in the output. If neither the catalog nor the schema has one, and the metastore has no root storage either, there's nowhere for the staging volume to live.
2. Default storage. If the staging catalog was created on default storage (the option you get in a serverless workspace), the docs say it can only be accessed from serverless compute. The MySQL ingestion gateway runs on classic job compute, so it can't write there. Pointing staging at a catalog or schema with a managed location on your own S3 bucket (through an external location) gets around that.
3. Privileges on the staging schema. You need USE SCHEMA, CREATE TABLE and CREATE VOLUME there, and the troubleshooting page also notes that the staging catalog can't be a foreign catalog.
If you'd rather stay fully serverless, the newer integrated CDC pipeline for MySQL can run on serverless compute, as long as serverless can reach your MySQL instance over the network.
If none of that matches, the pipeline's event log should have the full error with a stack trace. Post it here and I'm happy to take another look.
Cheers,
Suranga
yesterday
Hi Michael,
Iโve seen this error with Lakeflow Connect database connectors. The s3:/ with nothing after it suggests the pipeline couldnโt work out a storage location for its staging data.
The ingestion gateway first writes the change data to a Unity Catalog volume in the staging catalog and schema you chose during setup. If that catalog or schema has no managed storage location, or the location is set up wrong, the path comes out empty and you get this error.
Things worth checking:
If you point the gateway at a staging catalog/schema with a proper S3 managed location and run it again, it should get past this step.
Hope this helps!
yesterday
Hi @MichaelAnckaert ,
The error Input URI is not a valid UC Path: s3:/ indicates that Unity Catalog is trying to physically create the destination tables (or the pipeline staging area) in AWS S3, but the storage path it has been handed is malformed. A valid AWS S3 path must start with s3://<bucket-name>/..., but yours is being evaluated as just s3:/ (missing the second slash and the bucket name).
Because the setup validation passed, the Lakeflow connector itself is working fine. The pipeline is simply inheriting a broken default storage path from your Unity Catalog setup when it attempts to write the actual data.
Here are the exact steps to find and fix the typo:
When Lakeflow Connect creates managed tables, it stores the data in the default storage location of the target schema. If the schema doesn't have one, it falls back to the Catalog's storage location.
Open Catalog Explorer in your Databricks workspace.
Navigate to the Target Catalog and the Target Schema you specified for the pipeline.
On both the Schema and Catalog pages, click the Details tab.
Check the Storage Location property.Verification: If you see a path starting with s3:/ instead of s3://, this is the culprit. You will need to recreate the schema (or catalog) with a properly formatted S3 URI, or choose a schema that has a valid storage location.
If your target Catalog and Schema do not have explicit storage locations defined, Unity Catalog will attempt to use the Metastore's root storage.
If you have admin privileges, click your username in the top right -> Manage Account -> Data (or check your workspace admin settings).
Look at the Metastore's Root Storage path.Verification: Confirm it is formatted correctly with s3:// and a valid bucket name.
Lakeflow Connect relies on a staging volume for CDC ingestion.
If you explicitly selected a volume for staging, locate that Volume in Catalog Explorer.
Check its Storage Location under the details tab.Verification: Ensure it is a valid, fully qualified S3 URI.
Take a look at those Catalog Explorer properties and you will likely spot the malformed path immediately. If you check those and they all look correct, please reply back with:
Are you writing to a Managed Schema or an External Schema?
Are you letting Databricks create the staging volume automatically, or did you specify an existing one?
We will get this pipeline running for you!
14 hours ago
Thank you all for your assistance.
The staging catalog does have a correct storage location configured (on S3). The pipeline created a cdc logs catalog (with attached volume) and has written some files. The staging catalog also has other schema's that are in use (test workspace).
The metastore in use however does not have a storage location configured. Can this be the issue?
Are there any risks involved with configuring an S3 storage location for an existing metastore?
8 hours ago
Thanks for checking the staging side. Yes, it could be the issue. An empty bucket in "s3:/" looks like a storage root that resolved to nothing, and Unity Catalog only falls back to the metastore when neither the schema nor the catalog has a managed location. With no metastore storage, that fallback is empty.
Since the staging catalog is fine, I'd check the other places the pipeline writes: the destination catalog and schema, and the catalog and schema of the event log table if you publish one.
DESCRIBE CATALOG EXTENDED <destination_catalog>; DESCRIBE SCHEMA EXTENDED <destination_catalog>.<destination_schema>;
If one of them has no storage location, I'd fix it there rather than at the metastore. Databricks recommends catalog-level storage for isolation anyway:
ALTER CATALOG <destination_catalog> SET MANAGED LOCATION 's3://<bucket>/<path>';
That needs Databricks SQL or Databricks Runtime 18.1 and above, plus an external location covering the path with CREATE MANAGED STORAGE. It only applies to new managed tables and volumes. Existing ones aren't moved.
Adding storage to the existing metastore is supported too. A few things to know first:
Docs: Managed storage locations and adding metastore-level storage