Thanks for the write up. Worked for me. Question. In my case, I let DLT write its tables to an abfss://data@... storage path using your config pattern above. It successfully writes the tables, but when I view the tables in hive_metastore in data expl...
Same issue using abfss//... path and using it for the dlt pipeline storage location. @Robert Thornton​ , i mounted the abfss path as '/mnt/data' using the same service principal and secret as I used when i configured the oauth. I then changed my dlt ...
@Michal Rawluk - can you be more specific as to the setting in Azure we need to change to allow DataBricks to create the cluster? I can't find the limit-setting you're referring to in my Azure portal/res-group.
