Databricks Community

dollyb

In a notebook attached to.a normal personal cluster I can successfully do this:%fs ls file:/Workspace/Repos/$userName/$repoName/$folderNameWhen I run an init-script on a UC volume that the does the same thing, I'm getting this error:ls: cannot access...

dollyb

The way I understand it, mount points are deprecated in UC. db.fs.mount() doesn't even seem to work in newer DB runtimes.But what is the solution when Databricks features don't allow using UC volumes? E.g. specifying a compute's logging path won't wo...

dollyb · 04-15-2024

Hello,I'm using a local Docker Spark 3.5 runtime to test my Databricks Connect code. However I've come across a couple of cases where my code would work in one environment, but not the other.Concrete example, I'm reading data from BigQuery via spark....

dollyb · 03-03-2024

Hi there,what's the best way to differentiate in what environment my Spark session is running? Locally I develop with databricks-connect's DatabricksSession, but that doesn't work when running a workflow job which requires SparkSession.getOrCreate()....

dollyb · 02-21-2024

Hi,I'm using Databricks Connect to run Scala code from IntelliJ on a Databricks single node cluster.Even with the simplest code, I'm experiencing this error:org.apache.spark.SparkException: grpc_shaded.io.grpc.StatusRuntimeException: INTERNAL: org.ap...

dollyb

I'm not using dbutils in my init-script, just the unix ls command.

dollyb

This should be updated for Unity Catalog workspaces.

dollyb

It's simply the statement I listed in my post. There are no variables, I just wrote it like that to not post the exact values. So it's something like this in the script and that's it:ls file:/Workspace/Repos/my_user_name@company.com/my_repo_name/my_f...

dollyb

Found the error message at last: DatabricksServiceException: IO_ERROR: java.io.IOException: This request is not authorized to perform this operation using this permission. Please see the cause for further information.

dollyb

I can actually see the mount point does exist when I do DBUtils.getDBUtils().fs.mounts()-> MountInfo(/mnt/cluster-logs,wasbs://xxx@xxx.blob.core.windows.net/cluster-logs/,) But still no logs arrive.

Databricks Community

User Stats

User Activity

Accessing Workspace / Repo file works in notebook, but not from job

Logging to an external location via UC volume

Differences between Spark SQL and Databricks

How to detect if running in a workflow job?

Databricks Connect Scala -

Re: Accessing Workspace / Repo file works in notebook, but not from job

Re: How to connect your Azure Data Lake Storage to Azure DatabricksStandard Workspace &#xd83d;&#xdc4

Re: Accessing Workspace / Repo file works in notebook, but not from job

Re: Logging to an external location via UC volume

Re: Logging to an external location via UC volume