cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

farooqurrehman
by New Contributor
  • 1120 Views
  • 3 replies
  • 2 kudos

Unable to connect/read files from ADLS Gen2 using account key

It gives error[RequestId=5e57b66f-b69f-4e8b-8706-3fe5baeb77a0 ErrorClass=METASTORE_DOES_NOT_EXIST] No metastore assigned for the current workspace.using the following codespark.conf.set(  "fs.azure.account.key.mystorageaccount.dfs.core.windows.net", ...

  • 1120 Views
  • 3 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Moderator
  • 2 kudos

Hi @Farooq ur rehman​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 2 kudos
2 More Replies
sunil_smile
by Contributor
  • 3375 Views
  • 9 replies
  • 11 kudos

Resolved! How i can add ADLS Gen2 - OAuth 2.0 as Cluster scope for my High concurrency Shared Cluster (without unity catalog)?

Hi All,Kindly help me , how i can add the ADLS gen2 OAuth 2.0 authentication to my high concurrency shared cluster. I want to scope this authentication to entire cluster not for particular notebook.Currently i have added them as spark configuration o...

image.png image
  • 3375 Views
  • 9 replies
  • 11 kudos
Latest Reply
Kaniz
Community Manager
  • 11 kudos

Hi @Sunilprasath Elangovan​ , We haven’t heard from you since the last response from @Hubert Dudek​​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the community as it can b...

  • 11 kudos
8 More Replies
Netty
by New Contributor III
  • 2299 Views
  • 5 replies
  • 7 kudos

Resolved! What's the easiest way to upsert data into a table? (Azure ADLS Gen2)

I had been trying to upsert rows into a table in Azure Blob Storage (ADLS Gen 2) based on two partitions (sample code below). insert overwrite table new_clicks_table partition(client_id, mm_date) select click_id ,user_id ,click_timestamp_gmt ,ca...

  • 2299 Views
  • 5 replies
  • 7 kudos
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 7 kudos

Below code might help youPython- (df.write .mode("overwrite") .option("partitionOverwriteMode", "dynamic") .saveAsTable("default.people10m") )   SQL- SET spark.sql.sources.partitionOverwriteMode=dynamic; INSERT OVERWRITE TABLE default.people10m...

  • 7 kudos
4 More Replies
LukaszJ
by Contributor III
  • 1991 Views
  • 5 replies
  • 4 kudos

Resolved! Mount Azure Blob Storage with Cluster access control

Hello.I want to mount and share for the one group the container from Azure Blob Storage (It could be simple blob storage or Azure Data Lake Storage gen 2). But I am not able to do it because I am using Cluster with Table Access Control.This is my cod...

  • 1991 Views
  • 5 replies
  • 4 kudos
Latest Reply
LukaszJ
Contributor III
  • 4 kudos

I have a good solution to the problem:I am using Python library.There are some documentation.Topic to be closed.Best regards,Łukasz

  • 4 kudos
4 More Replies
PraveenKumar188
by New Contributor
  • 1705 Views
  • 4 replies
  • 2 kudos

Resolved! Is is possible to Mount multiple ADLS Gen2 Storage paths in single workspace

Hello Experts,We are looking on feasibility of mounting more that one ADLS Gen2 storages on a single workspace of databricks.Best RegardsPraveen

  • 1705 Views
  • 4 replies
  • 2 kudos
Latest Reply
Erik
Valued Contributor II
  • 2 kudos

Yes, its possible, we are doing it. Just mount them to different folders like @Werner Stinckens​ is saying.

  • 2 kudos
3 More Replies
SailajaB
by Valued Contributor III
  • 12782 Views
  • 6 replies
  • 6 kudos

Resolved! Unable to mount the blob storage account as soft delete got enabled

Hi Team,when we try to mount or access the blob storage where soft delete enabled. But it is getting failed with below errororg.apache.hadoop.fs.FileAlreadyExistsException: Operation failed: "This endpoint does not support BlobStorageEvents or So...

  • 12782 Views
  • 6 replies
  • 6 kudos
Latest Reply
Kaniz
Community Manager
  • 6 kudos

Hi @Sailaja B​ , As per the Managed identities for Azure resource authentication document: NoteIf your blob account enables soft delete, system-assigned/user-assigned managed identity authentication is not supported in Data Flow.If you access the blo...

  • 6 kudos
5 More Replies
Databricks_Venk
by New Contributor
  • 5728 Views
  • 2 replies
  • 2 kudos
  • 5728 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @Venkata Vadapalli​ , You may follow the below steps to create a mount point using Azure Key-vault.You should have the following information:• Client ID (a.k.a. Application ID) => Key Name as ClientID = 06exxxxxxxxxxd60ef• Client Secret (a.k.a. Ap...

  • 2 kudos
1 More Replies
steven_vcnt
by New Contributor III
  • 1354 Views
  • 3 replies
  • 4 kudos

Resolved! ADLSGen2: Cannot display Delta table summary from Data>Table tab on Databricks

Hello, I have set up my account storage on Azure with an ADLSGen2 and I have succeeded to save the delta table on my ADLSGen2, from there I have created my delta table on Databricks.From there I am unable to display the summary of my delta table unde...

Errorimage
  • 1354 Views
  • 3 replies
  • 4 kudos
Latest Reply
steven_vcnt
New Contributor III
  • 4 kudos

Hello,Following Hubert comment, in order to create a delta table on Databricks from Azure, I had to use CLONE argument in order to copy the data plus the metadata of my delta table on Azure. In order to set up the connection between Databricks and A...

  • 4 kudos
2 More Replies
tap
by New Contributor III
  • 5593 Views
  • 9 replies
  • 11 kudos

Could Not Connect to ADLS Gen2 Using ABFSS

I'm new to Databricks, not sure what can I do about this issue. I run a simple comment to list all file paths but get SSLHandshakeException.Is there any way to resolve this?  The full error messageExecutionError              Traceback (most recent ca...

2021-12-06_23h04_24
  • 5593 Views
  • 9 replies
  • 11 kudos
Latest Reply
Anonymous
Not applicable
  • 11 kudos

@suet pooi tan​ - Thank you for letting us know.

  • 11 kudos
8 More Replies
halfwind22
by New Contributor III
  • 6470 Views
  • 11 replies
  • 12 kudos

Resolved! Unable to write csv files to Azure BLOB using pandas to_csv ()

I am using a Py function to read some data from a GET endpoint and write them as a CSV file to a Azure BLOB location.My GET endpoint takes 2 query parameters,param1 and param2. So initially, I have a dataframe paramDf that has two columns param1 and ...

  • 6470 Views
  • 11 replies
  • 12 kudos
Latest Reply
halfwind22
New Contributor III
  • 12 kudos

@Hubert Dudek​ I cant issue a spark command to executor node, throws up an error ,because foreach distributes the processing.

  • 12 kudos
10 More Replies
Labels