topic Is there a way to specify a header, set the delimiter, etc...in DLT? in Data Engineering

Is there a way to specify a header, set the delimiter, etc...in DLT?

Chris_Shehu — Tue, 20 Dec 2022 18:14:54 GMT

I was looking forward to using the Data Quality features that are provided with DLT but as far as I can the ingestion process is more restrictive than other methods. It doesn't seem like you can do much as far as setting delimiter type, headers or any other options really. Am I missing something? I've read through the documentation already.

Thanks,

Re: Is there a way to specify a header, set the delimiter, etc...in DLT?

Anonymous — Tue, 20 Dec 2022 18:26:10 GMT

DLT uses Autoloader to ingest data. With autoloader, you can provide read options for the table. https://docs.databricks.com/ingestion/auto-loader/options.html#csv-options has the docs on CSV. I attached a picture of an example.

Re: Is there a way to specify a header, set the delimiter, etc...in DLT?

Chris_Shehu — Tue, 20 Dec 2022 18:28:02 GMT

That is exactly what I needed! Thanks Josephk. I guess when searching I needed to look for the auto loader options not DLT SQL options.

Re: Is there a way to specify a header, set the delimiter, etc...in DLT?

Anonymous — Tue, 20 Dec 2022 18:47:33 GMT

Glad to help!