How do i Create a workspace object with SP ownership

[Naveenkumar1811]

Hi Team,

I have a scenario that i have a jar file(24MB) to be put on workspace directory. But the ownership should be associated to the SP with any Individual ID ownership. Tried the Databricks CLI export option but it has limitation of 10 MB max.

Please suggest.

[Raman_Unifeye]

Are you trying from CLI ?

I would try uploading it to Unity Catalog Volume using UC Files API running with SP auth. It allows to upload upto 5GB size of file.

[Coffee77]

Trick I use to do something similar to that: Run a job with that service principal. In theory, all new objects created will be configured with that service principal ownership. Job can be as simple as a task with a notebook importing your file.

In my case, "Creator" is the same as "Run as" but those can be different. In my scenario are the same because I use DAB with same SP to deploy jobs.

[szymon_dybczak]

Hi @Naveenkumar1811 ,

Setup a Unity Catalog volume and then you can use following rest api call. It supports file up to 5 GB:

Upload a file | Files API | REST API reference | Azure Databricks

I guess dabricks cp command should also work with volume - that's another option. So once you have databricks volumen you can try following:

databricks cp path_to_your_file dbfs:/Volumes/your_catalog_name/your_schema_name/Volume_name/path/to/data

[Raman_Unifeye]

Reference Link - https://docs.databricks.com/aws/en/volumes/volume-files#upload-files-to-a-volume

[Naveenkumar1811]

Hi Team,

My Workspace is not Unity Catalogue Enabled... Do we have any Solution for the workspace without UC?

Thanks,

Naveen

[Naveenkumar1811]

Any Suggestion here?

[stbjelcevic]

Use Workspace files (500 MB file limit) and create the file under the SP’s home while running as the SP identity, ensuring association with the SP. 

Docs on workspace files: https://docs.databricks.com/aws/en/files/workspace 

[Coffee77]

How to create that file while running under SP identity? Service principals are designed solely for API access and cannot log into the Databricks UI. So, you mean Databricks API should be used to import those files under service principal security context? And, wouldn't it be the same as using Databricks CLI as in the end I guess CLI is making use of that API? Thanks!

[Coffee77]

Inspecting underlying HTTP traffic while using Databricks UI to import files in Workspace, it turns out (as expected) Databricks API is used, with requests similar to:

So, @Naveenkumar1811 use Databricks API with SP identity in a similar way as expected above. Docs on API are here: https://docs.databricks.com/api/azure/workspace/workspace/import . I guess, limitation on size is not in the API side, but inside Databricks CLI .