topic Re: data is not written back to data lake in Data Engineering

data is not written back to data lake

g96g — Tue, 18 Apr 2023 07:14:57 GMT

I have this strange case where data is not written back to data lake.

I have 3 container- . Bronze, Silver and Gold. I have done the mounting and have not problem to read the source data and write it Bronze layer ( using hive meta store catalog).

Then I pick the data from bronze for cleanisng and creating Silver tables and writing it to Silver. In data brick I can query this silver tables but when I check the silver container in azure portal it empty.

Below is the location i use and I have checked multiple times if I use correct storage name and its right one.

LOCATION 'dbfs:/mnt/silver@{{ table.STORAGE_ACCOUNT_NAME }}.dfs.core.windows.net/{{ table.TABLE_SOURCE }}/{{ table.TABLE_NAME }}';

Re: data is not written back to data lake

Serlal — Tue, 18 Apr 2023 15:12:24 GMT

If I understood you description well, it sounds as if you are actually writing the data to dbfs and not to the external location when creating the table and writing to silver. Are you sure you are defining the external path on table creation? For instance have a look here:

https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-external-tables

Re: data is not written back to data lake

Anonymous — Mon, 24 Apr 2023 04:34:51 GMT

Hi @Givi Salu

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you.

Cheers!

Re: data is not written back to data lake

g96g — Tue, 02 May 2023 10:42:43 GMT

no I have not