- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-10-2024 09:58 PM
I'm encountering some issues while trying to read a public dataset from a URL using Databricks. Here's the code snippet(along with errors) I'm working with:
I'm confused about Delta format error here.
- When I read data from a URL, how would it have a Delta log associated with it? Delta logs seem relevant for data stored in Databricks, not external URLs.
- Why is Databricks suggesting Delta format for this scenario?
I'm a bit stuck here. Any pointers or advice, please?
- Labels:
-
Delta Lake
-
Spark
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-11-2024 03:42 AM
Please disable the formatCheck in notebook and check if you could read the data
The configuration command %sql SET spark.databricks.delta.formatCheck.enabled=false will disable the format check for Delta tables in Databricks.
Databricks delta format check is a feature that validates whether the format of the source files is one of CSV, JSON, AVRO, ORC, PARQUET, TEXT, or BINARYFILE and if that meets that condition it will raise exception that you are facing.
This format check is because to ensure only delta format must be loaded for any operation.
If the format check is disabled, Databricks will not perform this validation and bypass this format check.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-11-2024 03:42 AM
Please disable the formatCheck in notebook and check if you could read the data
The configuration command %sql SET spark.databricks.delta.formatCheck.enabled=false will disable the format check for Delta tables in Databricks.
Databricks delta format check is a feature that validates whether the format of the source files is one of CSV, JSON, AVRO, ORC, PARQUET, TEXT, or BINARYFILE and if that meets that condition it will raise exception that you are facing.
This format check is because to ensure only delta format must be loaded for any operation.
If the format check is disabled, Databricks will not perform this validation and bypass this format check.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-03-2024 07:25 AM
Thank you @MuthuLakshmi . It helped.

