Autoloader with SQLite db files

eva_mcmf — Fri, 30 Jun 2023 06:45:21 GMT

Hi Everyone,

Is it possible to ingest SQLite db files with Databricks Autoloader? Is it currently supported? If so, could you please share an example?

Re: Autoloader with SQLite db files

lucasrocha — Wed, 05 Jun 2024 23:23:19 GMT

Hello @eva_mcmf , I hope this message finds you well.

As per the documentation, Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage. Auto Loader can load data files from AWS S3, Azure Data Lake Storage Gen2, Google Cloud Storage, Azure Blob Storage, ADLS Gen1, and Databricks File System. Auto Loader can ingest JSON, CSV, XML, PARQUET, AVRO, ORC, TEXT, and BINARYFILE file formats.

Currently I couldn't find this possibility of directly ingesting from SQLite with Autoloader through the documentation.
How does Auto Loader work: https://docs.databricks.com/en/ingestion/auto-loader/index.html#how-does-auto-loader-work

If you believe that ingesting SQLite db files with Auto Loader would be beneficial, you can submit an idea in Databricks' Ideas Portal.

Best regards,
Lucas Rocha

topic Re: Autoloader with SQLite db files in Data Engineering

Autoloader with SQLite db files

Re: Autoloader with SQLite db files