- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-28-2023 01:53 AM
Hello ,
If i refer to above code you created then error is like below pyspark.errors.exceptions.AnalysisException: Failed to read dataset 'Temp_Table'. Dataset is not defined in the pipeline. for each of 5 Temp tables
Below is flow at high level for my DLT Pipeline.
Step1 - 5 Bronze level tables are created and loaded from JSON files
Step2 - 5 Temp tables are created from 5 bronze tables ( created in step 1) above with Boolean bad flag ( derived)
Step3 - 5 Clean and 5 Quarantine tables are created by seperating Good & Bad data based on Bad Flag.
Step4 - 5 Gold layer tables are created from 5 clean tables created in Step 3.
Earlier i had separate notebook for each step which worked great. But when i combined all these into one notebook i am running into issues which i am NOT able to understand. Each table is in separate cell in all steps as such.