topic Re: writing to blob storage from databricks (parquet format) in Data Engineering

writing to blob storage from databricks (parquet format)

yagmur — Thu, 15 Aug 2024 07:23:58 GMT

Hi,
I am supposed to create transformation notebook. But i am having trouble when i am trying to save the transformed file into blob storage.
I didn't use any layer, just the layer which i am performing transformation in. if i use wasbs i receive different error, or if i use abfss i receive different. I tried to mount and save them into my dbfs, it worked but i couldnt carry them into blob storage.
When i use connection (azure services princble) i am getting the first error i mentioned. (the error pictures are attached)
Thanks in advance

PS: If i try to save to raw container which i mounted it and where it read the files, i will be saved. I dont understand even though i have connected without mount at the same time why i am having that error message?

Re: writing to blob storage from databricks (parquet format)

szymon_dybczak — Thu, 15 Aug 2024 11:20:29 GMT

Hi @yagmur ,

Did you assigned required permission to service principal on storage account?
And make sure you're configuring connection to storage account in proper way. You should have something similiar to the code below:

configs = { "fs.azure.account.auth.type" : "OAuth", "fs.azure.account.oauth.provider.type" : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider", "fs.azure.account.oauth2.client.id" :"your_client_id", "fs.azure.account.oauth2.client.secret" : "your_secret", "fs.azure.account.oauth2.client.endpoint" : f"https://login.microsoftonline.com/your_tenant_id/oauth2/token" }; dbutils.fs.mount( source = f"abfss://your_container@your_storage_account.dfs.core.windows.net/", mount_point = f"/mnt/your_mount_point_name", extra_configs = configs );

Anyway, nowadays you should use Unity Catalog and configure storage using Storage Credential

Re: writing to blob storage from databricks (parquet format)

yagmur — Tue, 27 Aug 2024 10:00:14 GMT

Thanks for the effort but that is the same what i did. Still not working