- 48 Views
- 1 replies
- 0 kudos
Delta Jira data import to Databricks
We need to import large amount of Jira data into Databricks, and should import only the delta changes. What's the best approach to do so? Using the Fivetran Jira connector or develop our own Python scripts/pipeline code? Thanks.
- 48 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @greengil, Have you considered Lakeflow Connect? Databricks now has a native Jira connector in Lakeflow Connect that can achieve what you are looking for. It's in beta, but something you may want to consider. It ingests Jira into Delta with incr...
- 0 kudos
- 200 Views
- 7 replies
- 0 kudos
Unable to connect to any cluster from a notebook
I'm experiencing an unusual issue following my return from annual leave. I'm unable to connect to any compute from a notebook (both Classic Compute and Serverless) this is despite having Can Manage permissions on the clusters.The error shown is: "Unk...
- 200 Views
- 7 replies
- 0 kudos
- 296 Views
- 4 replies
- 2 kudos
Resolved! What is the best way to use Unity catalog with medallion architecture using ADLS2
Hi,I am using a medallion architecture on Azure Data Lake Storage Gen2 with Azure Databricks. Currently, I am storing data in Parquet format (not Delta tables), and I am planning to implement Unity Catalog (UC).As part of this setup, I understand tha...
- 296 Views
- 4 replies
- 2 kudos
- 2 kudos
I was going to follow 3rd but then it violets our medallion. And we don't have that much data to separate it physically. So going with 1st approach. But Thank you very much @karthickrs, I'll keep this in mind
- 2 kudos
- 149 Views
- 1 replies
- 0 kudos
Lakebase in an enterprise setup
Hi,Has anyone managed to get the new Lakebase autoscaling fully working in an enterprise Azure setup?We are currently facing issues when setting up Lakebase autoscaling in a Databricks environment without a public IP, where all traffic is routed priv...
- 149 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, so I think what is happening as different types of tools get routed to lakebase via different paths and I believe you need to open up an additional private link. The docs here explain it pretty well. https://learn.microsoft.com/en-us/azure/datab...
- 0 kudos
- 93 Views
- 1 replies
- 0 kudos
[Unity Catalog] Lack of Credential Type When GCS Interworking in Database ricks in AWS Environment
Hi, I'm using Databricks in AWS environment andI'm trying to link the data from GCP GCS to Unity Catalog.[Official document]I tried to set it up by referring to the official guide of Databricks below.▶ Create service credentials guides[Problem situat...
- 93 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi — this is expected behavior, not a bug. Unity Catalog storage credentials in the UI are cloud-specific to your workspace deployment. Since your workspace runs on AWS, you only see AWS IAM Role and Cloudflare API Token. The GCP Service Account opti...
- 0 kudos
- 168 Views
- 2 replies
- 0 kudos
Can't able to enable Saml SSO
We are plannig to implement the architecturePingFederate is Front IDP -> Microsoft Entra is Actual IDP where we have our Users -> SSO Saml --> Databricks(SP)I have done all the config setups everything is working as expected, the request is Finally ...
- 168 Views
- 2 replies
- 0 kudos
- 0 kudos
This may not be solvable here. Do you have Support via Databricks? You'll need to create a Support Ticket (direct if AWS else MSFT ticket if Azure). If not, ping your Account Team for help. Just saying cos it might require the type of debugging that'...
- 0 kudos
- 313 Views
- 3 replies
- 0 kudos
Resolved! Get resource permissions using terraform
Is there a way to retrieve resources (cluster, job, volume, catalog and all other objects) permissions?On terraform docs there's a resource databricks_permissions but I didn't find a data source databricks_permissions, grants ou similar.How can I get...
- 313 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @fkseki, There isn’t a data "databricks_permissions" (or similar) in the Databricks Terraform, only the databricks_permissions resource, and that resource is authoritative for the full ACL of the object. That means Terraform can’t read the current...
- 0 kudos
- 297 Views
- 2 replies
- 1 kudos
Resolved! Manage MFA
I can't seem to manage my MFA. It just sends me into a loop. Can someone help me look into this please?
- 297 Views
- 2 replies
- 1 kudos
- 161 Views
- 2 replies
- 0 kudos
AI Runtime Public Preview?
How can I get access to AI runtime? I see that it is in 'Public Preview' but I dont see a feature toggle for it on the account Previews page.https://docs.databricks.com/aws/en/machine-learning/ai-runtime/
- 161 Views
- 2 replies
- 0 kudos
- 0 kudos
The confusion comes from AI Runtime having two components with different preview states.Single-node tasks are already open and do not require any toggle. You can just open a notebook, click the Connect dropdown, select Serverless GPU, choose your acc...
- 0 kudos
- 156 Views
- 2 replies
- 1 kudos
Which role is recommended to create and manage Unity Catalog objects—Workspace Admin or Metastore Ad
Which role is recommended to create and manage Unity Catalog objects (catalog, schema, Storage credentials, External Location)—Workspace Admin or Metastore Admin—and why?
- 156 Views
- 2 replies
- 1 kudos
- 1 kudos
I am designing the security model for our Databricks platform and need guidance on role selection for managing Unity Catalog. Which role should be used for creating and managing Unity Catalog objects such as Storage Credentials, External Locations, C...
- 1 kudos
- 334 Views
- 5 replies
- 3 kudos
Spark config ignored in job run settings
i am talking about this setting: i tried so far:spark.executor.cores 8andspark.log.level INFOBoth documented here: https://spark.apache.org/docs/latest/configuration.htmlbut neither i see effect nor i see them set if i check sparkui -> environment ta...
- 334 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi, You're not putting in the wrong place, it's just that Databricks doesn't allow certain configs, because they are managed by Databricks for you. For example your core spark config you've shown above won't be recognised as this is set by selected c...
- 3 kudos
- 441 Views
- 2 replies
- 1 kudos
Resolved! Running Browser Based Agentic Applications on Databricks
Hi,We are evaluating whether it is possible to host a browser‑based agentic application on Databricks.Our application performs frontend UI automation using the browser-use Python library and also exposes FastAPI endpoints to drive a UI..Application O...
- 441 Views
- 2 replies
- 1 kudos
- 1 kudos
TLDR: Databricks Apps/serverless won’t support this pattern; classic compute with Databricks Container Services is your only real option on Databricks, and even that has trade‑offs. For serious browser automation, run it off‑platform and integrate wi...
- 1 kudos
- 263 Views
- 1 replies
- 0 kudos
Power BI Service refresh failures caused by intermittent Databricks credential invalidation
Hi everyone,We are experiencing an intermittent issue in Power BI Service where datasets and dataflows connected directly to Databricks start failing refreshes because the credentials appear to be lost or invalidated, even though no manual changes we...
- 263 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, This is a known issue with how the Power BI Service handles OAuth refresh tokens with Databricks. Databricks enables Single-Use Refresh Tokens (SURTs) by default for Power BI integrations. If a token refresh fails mid-flight (network hiccup, timi...
- 0 kudos
- 984 Views
- 1 replies
- 1 kudos
Unable to connect Claude connector to Databricks MCP server (Genie)
Hello everyone,I am trying to connect Claude’s Custom Connector (Remote MCP) to a Databricks Managed MCP Server using a Genie space, but I keep encountering an authentication connection error. I would appreciate guidance on whether this is a configur...
- 984 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, updating from my previous post to give the latest update. As of two weeks ago, there is a public preview for this, it needs to be enabled in your workspace. You also need to have the Claude IP added to Databricks if you have any IP workspace rest...
- 1 kudos
- 300 Views
- 2 replies
- 2 kudos
Resolved! Automatic Identity Management with Nested Groups and API Access
Hi all,I’m exploring Automatic Identity Management for synchronizing nested groups in Azure Databricks. According to the documentation, this feature supports nested groups: Automatic Identity ManagementHowever, the same article notes that groups sync...
- 300 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi, I've just had a look internally and there is some discussion about making this functionality available but I can't give you a definitive idea of when this might be. In terms of workarounds the best one I can find is to use Tarracurl to make raw ...
- 2 kudos
