topic Re: Getting errors while reading data from URL in Data Engineering

Getting errors while reading data from URL

sharma_kamal — Mon, 11 Mar 2024 04:58:38 GMT

I'm encountering some issues while trying to read a public dataset from a URL using Databricks. Here's the code snippet(along with errors) I'm working with:

I'm confused about Delta format error here.

When I read data from a URL, how would it have a Delta log associated with it? Delta logs seem relevant for data stored in Databricks, not external URLs.
Why is Databricks suggesting Delta format for this scenario?

I'm a bit stuck here. Any pointers or advice, please?

Re: Getting errors while reading data from URL

MuthuLakshmi — Mon, 11 Mar 2024 10:42:14 GMT

@sharma_kamal

Please disable the formatCheck in notebook and check if you could read the data

The configuration command %sql SET spark.databricks.delta.formatCheck.enabled=false will disable the format check for Delta tables in Databricks.

Databricks delta format check is a feature that validates whether the format of the source files is one of CSV, JSON, AVRO, ORC, PARQUET, TEXT, or BINARYFILE and if that meets that condition it will raise exception that you are facing.

This format check is because to ensure only delta format must be loaded for any operation.

If the format check is disabled, Databricks will not perform this validation and bypass this format check.

Re: Getting errors while reading data from URL

sharma_kamal — Wed, 03 Apr 2024 14:25:28 GMT

Thank you @MuthuLakshmi . It helped.