User16857282152
Databricks Employee
Databricks Employee

I would start by debugging a small collection of records,

First count nulls, perhaps your import failed to read that column for some formatting reason.

Select count(*) from table where target_column is NOT NULL;

compare with

Select count(*) from table where target_column is NULL;

If the table is massive and costs are significant, combine the above into a single query.

Then grab a few rows

Create testTable as select * from sourceTable where target_column is not null LIMIT 10;

Note the limit clause, just grab a few rows.

That should give you an idea of what is wrong, if anything with the actual data as imported and what you are expecting the data to be.