cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Spencer_Kent
by New Contributor III
  • 7283 Views
  • 11 replies
  • 3 kudos

Shared cluster configuration that permits `dbutils.fs` commands

My workspace has a couple different types of clusters, and I'm having issues using the `dbutils` filesystem utilities when connected to a shared cluster. I'm hoping you can help me fix the configuration of the shared cluster so that I can actually us...

insufficient_permissions_on_shared_cluster shared_cluster_config individual_use_cluster
  • 7283 Views
  • 11 replies
  • 3 kudos
Latest Reply
jacovangelder
Contributor III
  • 3 kudos

Can you not use a No Isolation Shared cluster with Table access controls enabled on workspace level? 

  • 3 kudos
10 More Replies
marvin1
by New Contributor III
  • 4737 Views
  • 5 replies
  • 0 kudos

"Unable to upload to DBFS Query" Error running SQL warehouse query?

I have sql warehouse endpoints that work fine when querying from applications such as Tableau, but just running the included sample query against a running endpoint from the Query Editor from the workspace is returning "Unable to upload to DBFS Query...

  • 4737 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Marvin Ginns​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 0 kudos
4 More Replies
Bas1
by New Contributor III
  • 8388 Views
  • 17 replies
  • 20 kudos

Resolved! network security for DBFS storage account

In Azure Databricks the DBFS storage account is open to all networks. Changing that to use a private endpoint or minimizing access to selected networks is not allowed.Is there any way to add network security to this storage account? Alternatively, is...

  • 8388 Views
  • 17 replies
  • 20 kudos
Latest Reply
Odee79
New Contributor II
  • 20 kudos

How can we secure the storage account in the managed resource group which holds the DBFS with restricted network access, since access from all networks is blocked by our Azure storage account policy?

  • 20 kudos
16 More Replies
MattPython
by New Contributor
  • 14356 Views
  • 4 replies
  • 0 kudos

How do you read files from the DBFS with OS and Pandas Python libraries?

I created translations for decoded values and want to save the dictionary object the DBFS for mapping. However, I am unable to access the DBFS without using dbutils or PySpark library. Is there a way to access the DBFS with OS and Pandas Python libra...

image.png image image image
  • 14356 Views
  • 4 replies
  • 0 kudos
Latest Reply
User16789202230
New Contributor II
  • 0 kudos

db_path = 'file:///Workspace/Users/l<xxxxx>@databricks.com/TITANIC_DEMO/tested.csv' df = spark.read.csv(db_path, header = "True", inferSchema="True")

  • 0 kudos
3 More Replies
learnerbricks
by New Contributor II
  • 4169 Views
  • 4 replies
  • 0 kudos

Unable to save file in DBFS

I have took the azure datasets that are available for practice. I got the 10 days data from that dataset and now I want to save this data into DBFS in csv format. I have facing an error :" No such file or directory: 'No such file or directory: '/dbfs...

  • 4169 Views
  • 4 replies
  • 0 kudos
Latest Reply
pardosa
New Contributor II
  • 0 kudos

Hi,after some exercise you need to aware folder create in dbutils.fs.mkdirs("/dbfs/tmp/myfolder") it's created in /dbfs/dbfs/tmp/myfolderif you want to access path to_csv("/dbfs/tmp/myfolder/mytest.csv") you should created with this script dbutils.fs...

  • 0 kudos
3 More Replies
YosepWijaya
by New Contributor
  • 24619 Views
  • 4 replies
  • 1 kudos

How can I embed image to the cell using markdown or code?

I have been trying to embed the image from the dbfs location, when I run the code, the image is unknown or question mark. I have tried following code: The path of the file is dbfs:/FileStore/tables/svm.jpgdisplayHTML("<img src ='dbfs:/FileStore/tabl...

  • 24619 Views
  • 4 replies
  • 1 kudos
Latest Reply
AlexBrulliea
New Contributor II
  • 1 kudos

Is there a way to embed an image from mounted storage into my markdown cell? Or can this only be done using the dbfs files?

  • 1 kudos
3 More Replies
kinsun
by New Contributor II
  • 10989 Views
  • 5 replies
  • 0 kudos

Resolved! DBFS and Local File System Doubts

Dear Databricks Expert,I got some doubts when dealing with DBFS and Local File System.Case01: Copy a file from ADLS to DBFS. I am able to do so through the below python codes:#spark.conf.set("fs.azure.account.auth.type", "OAuth") spark.conf.set("fs.a...

  • 10989 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @KS LAU​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your q...

  • 0 kudos
4 More Replies
bearys
by New Contributor II
  • 2023 Views
  • 2 replies
  • 2 kudos

Illegal character in partition path when attempting REORG ... (PURGE)

I have a large delta table partitioned by an identifier column that I now have discovered has blank spaces in some of the identifiers, e.g. one partition can be defined by "Identifier=first identifier". Most partitions does not have these blank space...

  • 2023 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @bearys, The error message suggests an illegal character in the path at a specific index. The error is pointing to a blank space in the path "dbfs:/mnt/container/table_name/Identifier=first identifier/part-01347-8a9a157b-6d0d-75dd-b1b7-2aed12e057...

  • 2 kudos
1 More Replies
PrithwisMukerje
by New Contributor II
  • 77373 Views
  • 5 replies
  • 4 kudos

Resolved! How to download a file from dbfs to my local computer filesystem?

I have run the WordCount program and have saved the output into a directory as follows counts.saveAsTextFile("/users/data/hobbit-out1") subsequently I check that the output directory contains the expected number of files %fs ls /users/data/hobbit-ou...

  • 77373 Views
  • 5 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

@PrithwisMukerje ,    To download a file from DBFS to your local computer filesystem, you can use the Databricks CLI command databricks fs cp.   Here are the steps:   1. Open a terminal or command prompt on your local computer.2. Run the follow...

  • 4 kudos
4 More Replies
Data_Engineer_3
by New Contributor III
  • 13776 Views
  • 17 replies
  • 7 kudos

Resolved! FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above

I am new to learning Spark and working on some practice; I have uploaded a zip file in DBFS /FileStore/tables directory and trying to run a python code to unzip the file; The python code is as: from zipfile import *with ZipFile("/FileStore/tables/fli...

  • 13776 Views
  • 17 replies
  • 7 kudos
Latest Reply
883022
New Contributor II
  • 7 kudos

What if changing the runtime is not an option? I'm experiencing a similar issue using the following:%pip install -r /dbfs/path/to/file.txtThis worked for a while, but now I'm getting the Errno 2 mentioned above. I am still able to print the same file...

  • 7 kudos
16 More Replies
gtyhchang
by New Contributor II
  • 860 Views
  • 2 replies
  • 1 kudos

pandas issue

We identify a potential bug in either DBFS or Pandas that when writting a dataframe using Pandas `to_csv`, `to_parquet`, `to_pickle` etc to a mounted ADLS location with read-only service principle didn't throw permission deny exceptions. However, met...

  • 860 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Yung-Hang Chang​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 1 kudos
1 More Replies
fijoy
by Contributor
  • 1594 Views
  • 3 replies
  • 0 kudos

Is there a utility to convert between "/dbfs" and "dbfs:" path strings?

Is there a built-in utility function, e.g., dbutils, that can convert between path strings that start with "dbfs:" and "/dbfs"?Some operations, e.g, copying from one location in DBFS to another using dbutils.fs.cp() expect the path starting with "/db...

  • 1594 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Fijoy Vadakkumpadan​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best a...

  • 0 kudos
2 More Replies
Dilorom
by New Contributor
  • 4818 Views
  • 5 replies
  • 3 kudos

What is a recommended directory for creating a database with a specified path?

I was going through Data Engineering with Databricks training, and in DE 3.3L - Databases, Tables & Views Lab section, it says "Defining database directories for groups of users can greatly reduce the chances of accidental data exfiltration." I agree...

  • 4818 Views
  • 5 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Dilorom A​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...

  • 3 kudos
4 More Replies
StephanieRivera
by Valued Contributor II
  • 5027 Views
  • 4 replies
  • 2 kudos
  • 5027 Views
  • 4 replies
  • 2 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 2 kudos

Hi, You can refer to https://docs.databricks.com/files/unzip-files.html. You can curl the file you want and then it can be unzipped as mentioned in the doc. Please let us know if this helps.Also, please tag @Debayan with your next update which will n...

  • 2 kudos
3 More Replies
billaspiel
by New Contributor II
  • 10880 Views
  • 3 replies
  • 0 kudos

Resolved! Python open function is unable to detect the file in dbfs

hi , Im a newbie learning spark using databricks , I did some investigation and searched if this questions was been asked earlier in community forum but unable to find anything so . 1. DBFS is unable to detect the file even though its present in it...

  • 10880 Views
  • 3 replies
  • 0 kudos
Latest Reply
Dflo
New Contributor II
  • 0 kudos

I am having similar issues currently. I can read or access my storage account but when I attempted to read or access the container it told me path not found. I create the container and have full access as an owner.

  • 0 kudos
2 More Replies
Labels