cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

jasont41
by New Contributor II
  • 1324 Views
  • 1 replies
  • 2 kudos

Resolved! Trouble with host url parameterization

I am attempting to parameterize a databricks yaml so I can deploy it to multiple databricks accounts via Gitlab CICD, and have ran into a snag when parameterizing the workpace host value. My variable block looks like this: variables:    databricks_ho...

  • 1324 Views
  • 1 replies
  • 2 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 2 kudos

Hi @jasont41 ,Your assumption is correct. You can't use variable for host mapping. You can find information about it in the following documentation entry: https://docs.databricks.com/en/dev-tools/bundles/settings.html#other-workspace-mappings

  • 2 kudos
PabloCSD
by Valued Contributor II
  • 5458 Views
  • 3 replies
  • 3 kudos

Resolved! Use a Service Principal Token instead of Personal Access Token for Databricks Asset Bundle

How can I connect using a Service Principal Token, I did this, but it is not a PAT: databricks configure Databricks host: https:// ... Personal access token: ****  I also tried this, but didn't work either: [profile] host = <workspace-url> client_id ...

  • 5458 Views
  • 3 replies
  • 3 kudos
Latest Reply
PabloCSD
Valued Contributor II
  • 3 kudos

Thanks Pedro, we did it, for anyone in the future (I added fake host and service principal id's):1. Modify your databricks.yml so it have the service principal id and the databricks host: bundle: name: my_workflow # Declare to Databricks Assets Bu...

  • 3 kudos
2 More Replies
dspatil
by New Contributor II
  • 927 Views
  • 2 replies
  • 0 kudos

Automatic schema rendering of files in unity catalog

Hi team,Can anyone please confirm if Unity catalog supports automatic schema rendering from csv, json, pdfs, and structured/unstructured files?Meaning, if i create a volume with path/location to folder (or S3 bucket) having such files, can unity cata...

  • 927 Views
  • 2 replies
  • 0 kudos
Latest Reply
dspatil
New Contributor II
  • 0 kudos

ok..thanks a lot @gchandra .So, I am new to Unity Catalog and particularly interested (and evaluating) the open sourced version of unity catalog (https://www.unitycatalog.io/)I know that, we can create volumes and those in turn can point to csv, json...

  • 0 kudos
1 More Replies
AlbertWang
by Valued Contributor
  • 1560 Views
  • 2 replies
  • 1 kudos

Resolved! Error getting locations Unsupported response format: STREAM [Azure Databricks - Catalog Explorer]

Hi all,I saw this error when I checked my External Locations and Storage Credentials and used Catalog Explorer. The error message gives zero information for diagnosis. Do you have any idea what the reason is?Thank you.

AlbertWang_1-1727297979431.png AlbertWang_2-1727297996274.png AlbertWang_3-1727298157976.png
  • 1560 Views
  • 2 replies
  • 1 kudos
Latest Reply
AlbertWang
Valued Contributor
  • 1 kudos

And it works now ... No idea what happened ~

  • 1 kudos
1 More Replies
chinmay0924
by New Contributor III
  • 2982 Views
  • 4 replies
  • 1 kudos

How to disable spark connect in the databricks compute?

I want to be able to access the RDD methods of a Dataframe, but it seems that this is not supported in spark connect. I have been trying to disable spark connect in the spark config using,spark.databricks.service.server.enabled false but when I check...

  • 2982 Views
  • 4 replies
  • 1 kudos
Latest Reply
Bareaj
New Contributor II
  • 1 kudos

I have found that when the cluster is shared, it automatically uses that type of session, and in that case, I have not been able to disable it. I don't know if this is your situation. I have avoided some problems that I had with the previous clause.

  • 1 kudos
3 More Replies
ViliamG
by New Contributor
  • 3431 Views
  • 0 replies
  • 0 kudos

MLFlow Tracking versions

Hi team,we are migrating from self-self hosted MLFlow Tracking server to the Databricks-hosted one. However, there are concerns about the unclear process of version changes and releases at the Tracking server side. Is there any public  information av...

  • 3431 Views
  • 0 replies
  • 0 kudos
AlbertWang
by Valued Contributor
  • 3804 Views
  • 5 replies
  • 2 kudos

Resolved! How to create Storage Credential using Service Principal [Azure]

As the document indicates, An Azure Databricks access connector is a first-party Azure resource that lets you connect managed identities to an Azure Databricks account. You must have the Contributor role or higher on the access connector resource in ...

  • 3804 Views
  • 5 replies
  • 2 kudos
Latest Reply
AlbertWang
Valued Contributor
  • 2 kudos

Thank you, @szymon_dybczak. This is what I thought. After deploying the Databricks workspace, it automatically creates the Databricks managed `Access Connector for Azure Databricks` in the Databricks managed resource group.As I understand, I should c...

  • 2 kudos
4 More Replies
sdick_vg
by New Contributor
  • 4895 Views
  • 1 replies
  • 1 kudos

Cluster Upsize Issue: Storage Download Failure Slow

Hi,We're currently experiencing the following issue across our entire Databricks Workspace when either starting a cluster, running a workflow, or upscaling a running cluster. The following errors we receive on our AP clusters and job clusters are bel...

  • 4895 Views
  • 1 replies
  • 1 kudos
Latest Reply
filipniziol
Esteemed Contributor
  • 1 kudos

Hi @sdick_vg ,The error is about connectivity issues when trying to reach Azure Storage.Have you maybe enabled any kind of firewall in your organization recently?Could you run for example code to test DNS resolution to your storage account:Have you m...

  • 1 kudos
jkdatabricks
by New Contributor
  • 1561 Views
  • 1 replies
  • 0 kudos

Error: PERMISSION_DENIED: AWS IAM role does

Hello, We are trying to setup a new workspace. However we are getting following error.  Workspace failed to launch.Error: PERMISSION_DENIED: AWS IAM role does not have READ permissions on url s3://jk-databricks-prods3/unity-catalog/742920957025975.Pl...

  • 1561 Views
  • 1 replies
  • 0 kudos
Latest Reply
caldempseyai
New Contributor II
  • 0 kudos

Hey! I'm experiencing this with the latest Terraform release. Try 1.51.0 if you are deploying via TF, downgrading fixed this for me.

  • 0 kudos
dpc
by Contributor
  • 1557 Views
  • 2 replies
  • 0 kudos

table deployment (DDL) from one catalog to another

HelloWe have a development, a test and a production environmentHow do you generally deploy DDL changes?So, alter a table in development and apply to test then productione.g.table1 has column1, column2, column3I add column4I now want to deploy this ch...

  • 1557 Views
  • 2 replies
  • 0 kudos
Latest Reply
dpc
Contributor
  • 0 kudos

Thanks.I'll step through this solution and see if I can get it working

  • 0 kudos
1 More Replies
weilin0323
by New Contributor III
  • 1391 Views
  • 1 replies
  • 0 kudos

Testing and Issues Related to Admin Role Changes

Hello,I would like to ask a question regarding user permissions.Currently, all team members are admins. Recently, we plan to change the admin roles so that only I and another user, A, will be admins. The other members will retain general usage permis...

  • 1391 Views
  • 1 replies
  • 0 kudos
Latest Reply
weilin0323
New Contributor III
  • 0 kudos

Hi @Kaniz_Fatma , Can you please help me to delete my post as I accidentally resubmitted the post.Thank you.

  • 0 kudos
cyborg-de
by New Contributor
  • 3808 Views
  • 0 replies
  • 0 kudos

DNS resolution across vnet

Hi, I have created a new databricks workspace in Azure with backend private link. Settings are Required NSG rules - No Azure Databricks RuleNSG rules for AAD and azfrontdoor were added as per documentation. Private endpoint with subresource  databric...

  • 3808 Views
  • 0 replies
  • 0 kudos
MosefJJunior3rd
by New Contributor
  • 937 Views
  • 1 replies
  • 0 kudos

How to fully enable row-level concurrency on Databricks 14.1

Hey guys,I hope whoever's reading this is doing well.We're trying to enable row-level concurrency on Databricks 14.1. However, the documentation seems a bit contradictory as to whether this is possible, and whether the whole capability of row-level c...

  • 937 Views
  • 1 replies
  • 0 kudos
Latest Reply
gchandra
Databricks Employee
  • 0 kudos

Yes, your understanding is correct. As best practice, use 14.3 LTS DBR, as LTS has longer support compared to 6 months.

  • 0 kudos
JohnHarmer
by New Contributor
  • 657 Views
  • 1 replies
  • 0 kudos

Unable to publish notebook

Hello, I receive an error each time I attempt to publish my notebook. the error message does not provide any additional information on why it is occurring. I need to publish this for a bootcamp assignment, so please let me know what I can do to resol...

  • 657 Views
  • 1 replies
  • 0 kudos
Latest Reply
gchandra
Databricks Employee
  • 0 kudos

Try it from a different browser, or try it in incognito mode.

  • 0 kudos
saravjeet
by New Contributor III
  • 1415 Views
  • 2 replies
  • 0 kudos

Allow non-admin users to view the driver logs from a Unity Catalog-enabled pipeline

We are trying to enable the option to allow log reading to the non-admin users in  the databricks wokspace but we are not able to see the obvious way to check them. The documentation is not showing after enabling the below property where to check the...

  • 1415 Views
  • 2 replies
  • 0 kudos
Latest Reply
gchandra
Databricks Employee
  • 0 kudos

Do they see these links in the pipeline?

  • 0 kudos
1 More Replies