Databricks Community

g96g · ‎04-18-2023

I have this strange case where data is not written back to data lake.

I have 3 container- . Bronze, Silver and Gold. I have done the mounting and have not problem to read the source data and write it Bronze layer ( using hive meta store catalog).

Then I pick the data from bronze for cleanisng and creating Silver tables and writing it to Silver. In data brick I can query this silver tables but when I check the silver container in azure portal it empty.

Below is the location i use and I have checked multiple times if I use correct storage name and its right one.

LOCATION 'dbfs:/mnt/silver@{{ table.STORAGE_ACCOUNT_NAME }}.dfs.core.windows.net/{{ table.TABLE_SOURCE }}/{{ table.TABLE_NAME }}';

Serlal · ‎04-18-2023

If I understood you description well, it sounds as if you are actually writing the data to dbfs and not to the external location when creating the table and writing to silver. Are you sure you are defining the external path on table creation? For instance have a look here:

https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-external-tables

Anonymous · ‎04-23-2023

Hi @Givi Salu

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you.

Cheers!