Resolved! Best practices for using autoloader
I’m looking to follow best practices with Databricks Auto Loader. When ingesting different file formats, is it considered good practice to always read data as strings first, or is it better to use format-specific readers (e.g., JSON, CSV, binary) fro...
- 44 Views
- 1 replies
- 0 kudos
Latest Reply
Hey @DynDe , Use the format-specific readers from the start and let Auto Loader handle schema, rather than reading everything as generic strings/text yourself. Key points: Always set cloudFiles.format to the real file format (json, csv, xml, parquet,...
- 0 kudos