COPY INTO from Volume failure (rabbit hole)

alwaysmoredata — Fri, 17 Jan 2025 23:28:37 GMT

hey guys, I am stuck on a loading task, and I simply can't spot what is wrong.

The following query fails:

COPY INTO `test`.`test_databricks_tokenb3337f88ee667396b15f4e5b2dd5dbb0`.`pipeline_state`

FROM '/Volumes/test/test_databricks_tokenb3337f88ee667396b15f4e5b2dd5dbb0/_temp_load_volume/file_1737154045813408000'

FILEFORMAT = PARQUET;

with the following error:

python:

DatabaseTerminalException(ServerOperationError("The source directory did not contain any parsable files of type PARQUET. Please check the contents of '/Volumes/test/test_databricks_tokenb3337f88ee667396b15f4e5b2dd5dbb0/_temp_load_volume/file_1737154045813408000'."))

databricks query:

[COPY_INTO_SOURCE_SCHEMA_INFERENCE_FAILED] The source directory did not contain any parsable files of type PARQUET. Please check the contents of '/Volumes/test/test_databricks_tokenb3337f88ee667396b15f4e5b2dd5dbb0/_temp_load_volume/file_1737154045813408000'. The error can be silenced by setting 'spark.databricks.delta.copyInto.emptySourceCheck.enabled' to 'false'.

and I have downloaded and read the parquet file with pandas - file is perfectly fine...

What is wrong? I am stuck...

Re: COPY INTO from Volume failure (rabbit hole)

NandiniN — Sat, 01 Feb 2025 05:16:14 GMT

I see you are reading just 1 file, ensure that there are no zero-byte files in the directory. Zero-byte files can cause schema inference to fail.

Double-check that the directory contains valid Parquet files using parquet tools. Sometimes, even if the file appears fine when read with pandas, there might be issues with how the file is stored or named in the directory.

topic Re: COPY INTO from Volume failure (rabbit hole) in Get Started Discussions

COPY INTO from Volume failure (rabbit hole)

Re: COPY INTO from Volume failure (rabbit hole)