cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

data is not written back to data lake

g96g
New Contributor III

I have this strange case where data is not written back to data lake.

I have 3 container- . Bronze, Silver and Gold. I have done the mounting and have not problem to read the source data and write it Bronze layer ( using hive meta store catalog).

Then I pick the data from bronze for cleanisng and creating Silver tables and writing it to Silver. In data brick I can query this silver tables but when I check the silver container in azure portal it empty.

Below is the location i use and I have checked multiple times if I use correct storage name and its right one.

LOCATION 'dbfs:/mnt/silver@{{ table.STORAGE_ACCOUNT_NAME }}.dfs.core.windows.net/{{ table.TABLE_SOURCE }}/{{ table.TABLE_NAME }}';

3 REPLIES 3

Serlal
New Contributor III

If I understood you description well, it sounds as if you are actually writing the data to dbfs and not to the external location when creating the table and writing to silver. Are you sure you are defining the external path on table creation? For instance have a look here:

https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-external-tables

Anonymous
Not applicable

Hi @Givi Salu​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

g96g
New Contributor III

no I have not

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.