cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

jwilliam
by Contributor
  • 3366 Views
  • 4 replies
  • 2 kudos

Resolved! How to mount Azure Blob Storage with OAuth2?

We already know that we can mount Azure Data Lake Gen2 with OAuth2 using this:configs = {"fs.azure.account.auth.type": "OAuth", "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider", ...

  • 3366 Views
  • 4 replies
  • 2 kudos
Latest Reply
dssatpute
New Contributor II
  • 2 kudos

Try replacing wasbs with abfss and dfs with blob in the URI, should work! 

  • 2 kudos
3 More Replies
RayelightOP
by New Contributor II
  • 1306 Views
  • 1 replies
  • 2 kudos

Azure Blob Storage sas-keys expired for Apache Spark Tutorial

"Apache Spark programming with databricks" tutorial uses Blob storage parquet files on Azure. To access those files a sas key is used in the configuration files. Those keys were generated 5 years ago, however they expired in the begining of this mont...

  • 1306 Views
  • 1 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Moderator
  • 2 kudos

Adding @Vidula Khanna​ and @Kaniz Fatma​ for visibility to help with your request

  • 2 kudos
KVNARK
by Honored Contributor II
  • 2742 Views
  • 1 replies
  • 6 kudos

Resolved! grant the access permissions for specific container and also for specific folder within container in Azure Blob storage

Hi,regarding permissions for Azure Storage.we have created the Storage account (blob storage) and within the account we are going to create many containers and in which container we are going to have multiple folders and files.we want to grant permis...

  • 2742 Views
  • 1 replies
  • 6 kudos
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 6 kudos

Hi @KVNARK .​ ,You can use the service principle in the azure active directory to grant specific access to that app and use that app credentials to create a new mount point.That will help you to give specific storage permission to users.

  • 6 kudos
bonyfus
by New Contributor II
  • 2410 Views
  • 3 replies
  • 0 kudos

Error when accessing the file from azure blob storage

I am getting the following error when accessing the file in Azure blob storagejava.io.FileNotFoundException: File /10433893690638/mnt/22200/22200Ver1.sps does not exist.Code:ves_blob = dbutils.widgets.get("ves_blob") try: dbutils.fs.ls(ves_blob ) e...

  • 2410 Views
  • 3 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

that is certainly an invalid path, as the error shows.with %fs ls /mnt you can show the directory structure of the /mnt directory, assuming the blob storage is mounted.if not, you need to define the access ( URL etc.)

  • 0 kudos
2 More Replies
tariq
by New Contributor III
  • 8168 Views
  • 5 replies
  • 7 kudos

Databricks Azure Blob Storage access

I am trying to access files stored in Azure blob storage and have followed the documentation linked below:https://docs.databricks.com/external-data/azure-storage.htmlI was successful in mounting the Azure blob storage on dbfs but it seems that the me...

  • 8168 Views
  • 5 replies
  • 7 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 7 kudos

Hi, @Ravindra Ch​ , could you please check the firewall settings in Azure networking?

  • 7 kudos
4 More Replies
LukaszJ
by Contributor III
  • 3406 Views
  • 5 replies
  • 4 kudos

Resolved! Mount Azure Blob Storage with Cluster access control

Hello.I want to mount and share for the one group the container from Azure Blob Storage (It could be simple blob storage or Azure Data Lake Storage gen 2). But I am not able to do it because I am using Cluster with Table Access Control.This is my cod...

  • 3406 Views
  • 5 replies
  • 4 kudos
Latest Reply
LukaszJ
Contributor III
  • 4 kudos

I have a good solution to the problem:I am using Python library.There are some documentation.Topic to be closed.Best regards,Łukasz

  • 4 kudos
4 More Replies
RohitKulkarni
by Contributor II
  • 2289 Views
  • 2 replies
  • 2 kudos

Get file from SharePoint to copy into Azure blob storage

Hello Team,I am trying to copy the xlx files from sharepoint and move to the Azure blob storageUSERNAME = app_config_client.get_configuration_setting(key='BIAppConfig:SharepointUsername',label='BIApp').valuePASSWORD = app_config_client.get_configura...

  • 2289 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Rohit Kulkarni​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 2 kudos
1 More Replies
Surendra
by New Contributor III
  • 7728 Views
  • 5 replies
  • 8 kudos

Resolved! Databricks notebook is taking 2 hours to write to /dbfs/mnt (blob storage). Same job is taking 8 minutes to write to /dbfs/FileStore. I would like to understand why write performance is different in both cases.

Problem statement:Source file format : .tar.gzAvg size: 10 mbnumber of tar.gz files: 1000Each tar.gz file contails around 20000 csv files.Requirement : Untar the tar.gz file and write CSV files to blob storage / intermediate storage layer for further...

databricks_write_to_dbfsMount databricks_write_to_dbfsMount
  • 7728 Views
  • 5 replies
  • 8 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 8 kudos

Hi @Hubert Dudek​ , I Just wanted to thank you. We’re so lucky to have customers like you!The way you are helping our community is incredible.

  • 8 kudos
4 More Replies
AJ270990
by Contributor II
  • 5737 Views
  • 8 replies
  • 3 kudos

Resolved! Powerpoint file operations in Databricks

Hi Team, I am writing a python code in Azure Databricks where I have mounted a Azure storage and accessing the input dataset from Azure storage resource. I am accessing the input data from Azure storage and generating charts from that data in databri...

  • 5737 Views
  • 8 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Abhishek Jain​ Thanks for sending in your query. We are glad that you found a solution. Would you like to mark the answer as best so the other members can benefit from it too?Cheers!

  • 3 kudos
7 More Replies
Mradula
by New Contributor
  • 733 Views
  • 0 replies
  • 0 kudos

Displaying the queried data from mounted data from Azure Blob storage to databricks is slow

I have mounted by Azure blob storage json file to databricks which has around 18GB and trying to perform a simple count operation on it and I am noticing that it takes 14 mins for the same in the Community edition . seeking answers on whether this is...

14 min count
  • 733 Views
  • 0 replies
  • 0 kudos
AzureDatabricks
by New Contributor III
  • 6158 Views
  • 5 replies
  • 1 kudos

Parallel processing of json files in databricks pyspark

How we can read files from azure blob storage and process parallel in databricks using pyspark.As of now we are reading all 10 files at a time into dataframe and flattening it.Thanks & Regards,Sujata

  • 6158 Views
  • 5 replies
  • 1 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

spark.read.json("/mnt/dbfs/<ENTER PATH OF JSON DIR HERE>/*.jsonyou first have to mount your blob storage to databricks, I assume that is already done.https://spark.apache.org/docs/latest/sql-data-sources-json.html

  • 1 kudos
4 More Replies
MallikSunkara
by New Contributor II
  • 8181 Views
  • 4 replies
  • 0 kudos

how to pass arguments and variables to databricks python activity from azure data factory

how to pass arguments and variables to databricks python activity from azure data factory

  • 8181 Views
  • 4 replies
  • 0 kudos
Latest Reply
CristianIspan
New Contributor II
  • 0 kudos

try importing argv from sys. Then if you have the parameter added correctly in DataFactory you could get it in your python script typing argv[1] (index 0 is the file path).

  • 0 kudos
3 More Replies
AlaQabaja
by New Contributor II
  • 4639 Views
  • 3 replies
  • 0 kudos

Get last modified date or create date for azure blob container

Hi Everyone, I am trying to implement a way in Python to only read files that weren't loaded since the last run of my notebook. The way I am thinking of implementing this is to keep of the last time my notebook has finished in a database table. Nex...

  • 4639 Views
  • 3 replies
  • 0 kudos
Latest Reply
Forum_Admin
Contributor
  • 0 kudos

Hello! I just wanted to share my point of view on the topic of dating sites. I have been looking for a decent Asian catch-up site for a very long time, in addition to them I found https://hookupsearch.org/asian-hookup-sites/. We definitely recommend...

  • 0 kudos
2 More Replies
Labels