- 2895 Views
- 3 replies
- 3 kudos
Resolved! Is there a way to specify a header, set the delimiter, etc...in DLT?
I was looking forward to using the Data Quality features that are provided with DLT but as far as I can the ingestion process is more restrictive than other methods. It doesn't seem like you can do much as far as setting delimiter type, headers or an...
- 2895 Views
- 3 replies
- 3 kudos
- 3 kudos
DLT uses Autoloader to ingest data. With autoloader, you can provide read options for the table. https://docs.databricks.com/ingestion/auto-loader/options.html#csv-options has the docs on CSV. I attached a picture of an example.
- 3 kudos
- 1389 Views
- 3 replies
- 1 kudos
Incorrect results with df.query()
I have tried pulling a single row from a .csv using df.query()However, the data being returned doesn't coincide with the data I'm expecting - it is pulling a different row. Here is my code:df = spark.read.option("header",True).csv(data_fldr + "config...
- 1389 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Peter Ott​ Does @Hubert Dudek​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 1 kudos