cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
sensanjoy
by • Contributor
  • 4426 Views
  • 3 replies
  • 1 kudos

Resolved! Loading data from dataframe to Azure Storage Queue/Message Queue.

Hi Experts,We do have one use case where we have batch load that create a dataframe at end and now we want to load this data at Azure Storage Queue/Message Queue so that some Rest API can read the data/messages from the queue later and process it acc...

Data Engineering
Reply
  • 4426 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
sensanjoy
Contributor
  • 1 kudos

@Suteja Kanuri​  looking for your input here. Thanks.

  • 1 kudos
2 More Replies
sintsan
by • New Contributor II
  • 1499 Views
  • 1 replies
  • 1 kudos

Resolved! spark.sparkContext.setCheckpointDir - External Azure Storage

Is it possible to direct spark.sparkContext.setCheckpointDir to an external Azure Storage Container location (instead of DBFS) & if so how, there's very little documentation on that.

Data Engineering
Reply
  • 1499 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

yes,the directory must be an HDFS path if running on a cluster.All you need to do is provide the correct path.

  • 1 kudos
pkgltn
by • New Contributor III
  • 1002 Views
  • 0 replies
  • 0 kudos

Mounting a Azure Storage Account path on Databricks

Hi,I have a Databricks instance and I mounted the Azure Storage Account. When I run the following command, the output is ExecutionError: An error occurred while calling o1168.ls.: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: java.util...

Data Engineering
Reply
  • 1002 Views
  • 0 replies
  • 0 kudos
Bhanu1
by • New Contributor III
  • 4418 Views
  • 3 replies
  • 6 kudos

Resolved! Is it possible to mount different Azure Storage Accounts for different clusters in the same workspace?

We have a development and a production data lake. Is it possible to have a production or development cluster access only respective mounts using init scripts?

Data Engineering
Reply
  • 4418 Views
  • 3 replies
  • 6 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 6 kudos

Yes it is possible. Additionally mount is permanent and done in dbfs so it is enough to run it one time. you can have for example following configuration:In Azure you can have 2 databricks workspace,cluster in every workspace can have env variable is...

  • 6 kudos
2 More Replies
bchaubey
by • Contributor II
  • 1645 Views
  • 1 replies
  • 0 kudos

How to read Azure storage data through databricks in python . Could you please help me.

Data Engineering
Reply
  • 1645 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16764241763
Honored Contributor
  • 0 kudos

@Bhagwan Chaubey​ May be you can give this a try, if this is a Blob Storage account.https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python?tabs=environment-variable-windowsFor Datalake storage, please try belowhttps://do...

  • 0 kudos
Labels