ManojkMohan
Honored Contributor II

@RIDBX  The Free Edition only allows access to serverless compute resources, and many advanced streaming features are not supported. For example, custom storage locations and online/streaming tables are explicitly noted as unsupported features in this tier.

https://docs.databricks.com/aws/en/getting-started/free-edition-limitations

Databricks in general requires files to be present in cloud storage mounted to your workspace https://docs.databricks.com/aws/en/getting-started/free-edition-limitations

Alternate approaches

  • simulate streaming by incrementally adding new files or data batches to a source folder or table and then re-running batch queries that consume only the new data
  • Upload files in small increments to a cloud-mounted directory (like DBFS). Then use Auto Loader with batch mode to process newly added files during each job run. This mimics streaming ingestion
  • In notebooks, use loops or scheduled notebook jobs that periodically append new data files or rows into a Delta table, reading from that table each time as new input arrives