Yesterday I had a basic DLT pipeline up and running, and was able to query the hive_metastore tables successfully. The pipeline uses autloader to ingest a few csv files from cloud storage to streaming live bronze and silver tables. Today after starting the pipeline, when I click on any of the tables or try to execute a query against them, I get this error "Failure to Initialize Configuration". The pipeline is running successfully with no errors, but I am not able to query any of the streaming live tables.
Troubleshooting attempts:
- Deleted checkpoint files
- Deleted tables
- Deleted schema
- Changed the schema
- Ran the pipeline with "Full Refresh All"
- Created a new pipeline to run the same notebook.
- Ran a different DLT pipeline for a different notebook and dataset. The pipeline and sql tables all work fine.
Is there a way to access a more specific error? Is there something I am missing on my troubleshooting steps that I should do to reset to clear out these errors?
________________
UPDATE
I did some additional testing by creating a new pipeline to process the same dataset with the same notebook, but did not specify a storage location. This worked fine. So it appears this might be an issue with the Storage Location designation on the pipeline. Any suggestions on how to further troubleshoot this would be much appreciated!
