Delta Table created on s3 has all null values

Constantine — Mon, 11 Apr 2022 19:54:25 GMT

I have data in a Spark Dataframe and I write it to an s3 location. It has some complex datatypes like structs etc. When I create the table on top on the s3 location by using

CREATE TABLE IF NOT EXISTS table_name
USING DELTA
LOCATION 's3://.../...';

The table has all null values in it and I am not sure what is going wrong

Re: Delta Table created on s3 has all null values

Hubert-Dudek — Mon, 11 Apr 2022 20:05:13 GMT

@John Constantine ,

Try to load it as DataFrame (spark.read.delta(path)) and validate what is loading,
It could be easier to mount the S3 location as a folder to ensure that all data is there (dbutils or %fs to check) and that the connection is working correctly.
Try also REFRESH [TABLE] table_name,
Share more code, not sure what is loaded precisely. For example, the delta folder should be loaded, not a particular file,
There are parts/versions of delta in the delta folder written as a parquet. You can load them separately to DEBUG is all ok.

topic Re: Delta Table created on s3 has all null values in Data Engineering

Delta Table created on s3 has all null values

Re: Delta Table created on s3 has all null values