topic Unknown Location of files for tables created using DLT in Data Engineering

Unknown Location of files for tables created using DLT

ankitmit — Wed, 29 Jan 2025 08:46:47 GMT

Hi all,

I created catalog and schema using managed location

But I don’t see any catalogs directory within the s3 bucket path mentioned in the image above.

Also, I created a schema with managed location, and I expected all the tables created within this schema using DLT to be stored within the storage location path but this is not the case.

We're creating our tables using DLT

I am not sure, what’s going wrong here, and we still don’t see any files related to our tables in s3 buckets. We don’t know where exactly the tables files are being stored on our s3 buckets, even though we mentioned the path in catalog and schema levels.

Looking forward to hearing some feedbacks from you.

With Kindest Regards,
Ankit Mittal.

Re: Unknown Location of files for tables created using DLT

VZLA — Wed, 29 Jan 2025 15:03:57 GMT

Hello, thank you for your question.

Could you confirm that your issue is regarding the location of files for tables created using Delta Live Tables (DLT) when utilizing managed storage locations at the catalog and schema levels? Specifically, it seems that:

You have created a catalog and schema with managed locations in an S3 bucket.
Despite the managed locations being specified, you do not see the expected directory or files in the S3 bucket for tables created using DLT.

Am I correct in understanding your issue? If so, I can help clarify why this might be happening and suggest some troubleshooting steps. Let me know if additional context is needed!

Re: Unknown Location of files for tables created using DLT

ankitmit — Wed, 29 Jan 2025 15:45:29 GMT

Hi @VZLA ,
Yes, that's exactly what is happening!!
I am trying to understand where the files are being created in s3 buckets for tables using DLT and figure out if we can can somehow specify where the files should be created for these tables so that we can have more control over it, that's why I was tried to create a catalog and schema with managed location.

Please suggest some troubleshooting steps to help resolve this issue.

With Kindest Regards,
Ankit Mittal.