Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-09-2026 05:12 AM
I've created a Lakeflow job to run 5 notebook tasks, one for each silver table- Customers, Accounts, Transactions, Loans and Branches.
In Customers notebook, after writing the data to delta table using auto loader, I'm applying the non null and primary key constraint on customer_id:
spark.sql("ALTER TABLE capstone_project.silver.customers_silver ALTER COLUMN customer_id SET NOT NULL")
spark.sql("ALTER TABLE capstone_project.silver.customers_silver ADD CONSTRAINT pk_customer_id PRIMARY KEY (customer_id)")
Rest other tables have a foreign key constraint in a similar fashion on customer_id. And their relevant notebooks are dependent on customers notebook.
After the successful execution of customers notebook, the rest others throw an error that there is no primary key constraint in customers table. When I check the customers table, yes! there exists no primary key.
But, I have a code that has got executed and returned a dataframe. Then where did the primary key go? I'm unable to identify the root cause. Please assist
