cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
sensanjoy
by • Contributor
  • 2260 Views
  • 3 replies
  • 1 kudos

Resolved! Loading data from dataframe to Azure Storage Queue/Message Queue.

Hi Experts,We do have one use case where we have batch load that create a dataframe at end and now we want to load this data at Azure Storage Queue/Message Queue so that some Rest API can read the data/messages from the queue later and process it acc...

Data Engineering
Reply
  • 2260 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
sensanjoy
Contributor
  • 1 kudos

@Suteja Kanuri​  looking for your input here. Thanks.

  • 1 kudos
2 More Replies
sintsan
by • New Contributor II
  • 649 Views
  • 1 replies
  • 1 kudos

Resolved! spark.sparkContext.setCheckpointDir - External Azure Storage

Is it possible to direct spark.sparkContext.setCheckpointDir to an external Azure Storage Container location (instead of DBFS) & if so how, there's very little documentation on that.

Data Engineering
Reply
  • 649 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

yes,the directory must be an HDFS path if running on a cluster.All you need to do is provide the correct path.

  • 1 kudos
pkgltn
by • New Contributor III
  • 569 Views
  • 0 replies
  • 0 kudos

Mounting a Azure Storage Account path on Databricks

Hi,I have a Databricks instance and I mounted the Azure Storage Account. When I run the following command, the output is ExecutionError: An error occurred while calling o1168.ls.: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: java.util...

Data Engineering
Reply
  • 569 Views
  • 0 replies
  • 0 kudos
Bhanu1
by • New Contributor III
  • 2653 Views
  • 4 replies
  • 6 kudos

Resolved! Is it possible to mount different Azure Storage Accounts for different clusters in the same workspace?

We have a development and a production data lake. Is it possible to have a production or development cluster access only respective mounts using init scripts?

Data Engineering
Reply
  • 2653 Views
  • 4 replies
  • 6 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 6 kudos

Hi @Bhanu Patlolla​ ​, We haven’t heard from you on the last response from @Werner Stinckens​ and @Hubert Dudek​  and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be h...

  • 6 kudos
3 More Replies
bchaubey
by • Contributor II
  • 1033 Views
  • 1 replies
  • 0 kudos

How to read Azure storage data through databricks in python . Could you please help me.

Data Engineering
Reply
  • 1033 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16764241763
Honored Contributor
  • 0 kudos

@Bhagwan Chaubey​ May be you can give this a try, if this is a Blob Storage account.https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python?tabs=environment-variable-windowsFor Datalake storage, please try belowhttps://do...

  • 0 kudos
Labels