Delta Table created on s3 has all null values

[Constantine]

I have data in a Spark Dataframe and I write it to an s3 location. It has some complex datatypes like structs etc. When I create the table on top on the s3 location by using

CREATE TABLE IF NOT EXISTS table_name
USING DELTA
LOCATION 's3://.../...';

The table has all null values in it and I am not sure what is going wrong

[Hubert-Dudek · Accepted solution]

@John Constantine​ ,

• Try to load it as DataFrame (spark.read.delta(path)) and validate what is loading,
• It could be easier to mount the S3 location as a folder to ensure that all data is there (dbutils or %fs to check) and that the connection is working correctly.
• Try also REFRESH [TABLE] table_name,
• Share more code, not sure what is loaded precisely. For example, the delta folder should be loaded, not a particular file,
• There are parts/versions of delta in the delta folder written as a parquet. You can load them separately to DEBUG is all ok.