If I understood you description well, it sounds as if you are actually writing the data to dbfs and not to the external location when creating the table and writing to silver. Are you sure you are defining the external path on table creation? For in...
Hi Sandy!My 2 cents on your issue. This looks more like a design issue rather than a technical issue. From the sound of it, it looks like your notebook is having too many operations and if a failure occurs, everything repeats again which is not ideal...
If you are interested in distributed machine learning I would suggest Scaling Machine Learning with Spark by Adi Pola k . It is just out so it has all the new goodies including the latest in ML Flow. I am half way though it already but it looks like ...
Hi!I tried debugging your code and I think that the error you get is simply because the column exists in two instances of your dataframe within your loop.I tried adding some extra debug lines in your merge_dataframes function:and after executing that...
