Databricks Community

Red_blue_green · 02-08-2024

Hello,I have currently a delta folder as a table with several columns that are nullable. I want to migrate data to the table and overwrite the content using Pyspark, add several new columns and make them not nullable. I have found a way to make the c...

Red_blue_green · 02-12-2024

Hey @Dhruv-22 , yes, this is how I understand your problem. You have a folder perhaps with delta files and try to overwrite it with parquet files? I'm not familiar with parquet but I believe you need to delete parquet files each time before you crea...

Red_blue_green · 02-08-2024

Thank you for the suggestion but my current constraint is unfortunately to work with delta files. So saving as a table would not be enough.

Red_blue_green · 02-08-2024

Hi,this is how the delta format work. With overwrite you are not deleting the files in the folder or replacing them. Delta is creating a new file with the overwritten schema and data. This way you are also able to return to former versions of the del...

Databricks Community

User Stats

User Activity

Databricks: Change the existing schema of columns to non-nullable for a delta table using Pyspark?

Re: Managed table overwrites existing location for delta but not for oth

Re: Databricks: Change the existing schema of columns to non-nullable for a delta table using Pyspar

Re: Managed table overwrites existing location for delta but not for oth