Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best prac...
1340 Posts
Activity in Databricks Platform Discussions
- 33 Views
- 0 replies
- 0 kudos
Foreign Catalog Table - Missing Most Rows
I have a table pulled from Snowflake via a foreign catalog that's only returning 100 rows in Databricks even though there's billions of records in the source Snowflake table. What steps should I take to diagnose the limitation of rows?
- 33 Views
- 0 replies
- 0 kudos
- 176 Views
- 1 replies
- 0 kudos
credential delegation not found (404) / permission delegation name (500)
Hi, We run a monitoring integration that reads a Genie space's conversations via the Conversation API, using a service principal with CAN MANAGE on the space - capturing each conversation's prompt, generated SQL, and query result.Listing conversation...
- 176 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Summary: Yes — result retrieval is tied to the query’s credential/delegation model. Your 404/500 messages point to missing or unresolved permission delegation for those stored results, not missing CAN MANAGE on the space. If you need a monitoring se...
- 0 kudos
- 88 Views
- 1 replies
- 0 kudos
Zerobus ingestion fails in Databricks
I have premium databricks and i am trying to use ZeroBus with java SDK. my SP has admin rights and the righst below have been granted to the zerobus table GRANT USE CATALOG ON CATALOG <catalog> TO `<service-principal-id>`; GRANT USE SCHEMA ON SCHEMA ...
- 88 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Use a Databricks OAuth token for Zerobus, not a PAT or a federated/token-exchange token. The Java SDK docs now use client_id + client_secret, and Zerobus requires a token scoped to api://databricks/workspaces/<workspace_id>/zerobusDirectWriteApi with...
- 0 kudos
- 151 Views
- 3 replies
- 0 kudos
Dashboard Creation Requiring Workspace Access Entitlement?
Hi Team, I have analysts set up in Databricks with the "Databricks SQL access" entitlement who are no longer able to create a new Dashboard on their own and just getting a blank page when they try. The API response to their create dashboard request i...
- 151 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello, Thank you both for your responses. @Ashwin_DSA, I had an analyst test again and they were able to create a dashboard; looks like the hotfix worked. Thank you for your team's prompt attention to this matter. Sincerely, Matthew
- 0 kudos
- 199 Views
- 4 replies
- 0 kudos
Bundle Validation Error: Volume lifecycle Field Rejected Despite Being in Schema
Environment:Bundle engine: directResource type: Volume (Unity Catalog)Problem:I'm trying to use the lifecycle field with prevent_destroy: true on a Unity Catalog volume resource in my Declarative Automation Bundle (DAB). the validation fails with:Err...
- 199 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi, The configuration is valid and the CLI version supports the lifecycle field. However, the error is most likely coming from outdated Databricks Python packages in your virtual environment at /home/animesh/datazone/.venv/. CLI uses the packages fro...
- 0 kudos
- 2189 Views
- 3 replies
- 2 kudos
SCD-2 backfilling with streaming tabels
Hi there,Im new to Databricks and trying to build a SCD2 type table using AUTO CDC approach. while it quite simple to create a scd2 table Im unable to do a backfill.Full context.I have raw data(order, customer info) from 2019 and creating a dimension...
- 2189 Views
- 3 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi @JD18, Welcome to Databricks and Thanks for raising this. SCD Type 2 backfilling with streaming tables is a common need, and the good news is that the AUTO CDC framework (formerly APPLY CHANGES INTO) has built-in capabilities to handle this -- you...
- 2 kudos
- 251 Views
- 5 replies
- 1 kudos
Resolved! Permissions on job with SQL task
Hello,I am authenticated as GCP service account. I have to create query, run job as service principle and get result.I'm both cases, I set permissions CAN_MANAGE for group "users" for both query and job. When I create the query with RunAsMode.OWNER, ...
- 251 Views
- 5 replies
- 1 kudos
Latest Reply
- 1 kudos
Hello @balajij8,I tried to use SQL File Task and was able to achieve the result. Only issue was, that I was not able to get output using WorkspaceClient.jobs.get_run_output(). It worked will direct call of /api/2.2/jobs/runs/get-output endpoint.Thank...
- 1 kudos
- 516 Views
- 4 replies
- 3 kudos
Resolved! Managing IPYNB cell timestamps in source control
We're in the process of converting over our Databricks notebooks from .py file to .ipynb. We have disabled storing notebook output in source control at the workspace level.However, what we're discovering is that every cell in our notebooks has 3 time...
- 516 Views
- 4 replies
- 3 kudos
Latest Reply
- 3 kudos
Hi @emorgoch, Thanks for raising this. This appears to be a regression rather than expected behaviour. Internally, the issue has been identified around .ipynb handling in Git folders, and the intended fix is to stop serialising these execution timest...
- 3 kudos
- 123 Views
- 1 replies
- 0 kudos
Dashboard Variable Display name
In databricks documentation it says that there's a display name when creating dashboard variables but I'm not seeing it in the dashboard settings. Does anyone know where to find them?
- 123 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @amekojc Could you please share the exact steps you are following? I just tried this in my workspace and saw the display name option. I recorded a short video for you to review. Can you confirm if you are following the same steps?
- 0 kudos
- 64 Views
- 0 replies
- 0 kudos
Best AI Note-Taking Apps for iPad in 2026: Features, Use Cases, and Recommendations
The iPad has become much more than a tablet. For many professionals, students, researchers, and creators, it serves as a digital workspace for meetings, brainstorming sessions, lectures, project planning, and everyday productivity. Combined with the ...
- 64 Views
- 0 replies
- 0 kudos
- 282 Views
- 3 replies
- 3 kudos
Resolved! Did something change with the setup of Unity Catalogs?
Just some background. I deliver quite some trainings on Azure Databricks and in these trainings students have to provisions their own Azure Databricks workspace and UC. In the past this always went smoothly.Go to the Azure Portal Create a storage acc...
- 282 Views
- 3 replies
- 3 kudos
Latest Reply
- 3 kudos
Hi Ogbenisho,I have one follow up question on the 6 steps you describe:In step 4 and 5 you assume there is already a UC metastore to create the credentials and external location in.And then in step 6 you create it the UC metastoreBu what if you have ...
- 3 kudos
- 117 Views
- 1 replies
- 0 kudos
Serving endpoint - automatic system updates
Dear Community I need some support with investigation related to Serving Endpoints. Recently some of endpoints with deployed ML models display message:This endpoint is out of compliance because it is too old and automatic system updates have failed.P...
- 117 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi, Could you provide a workspaceID and endpointID and any other details such as screenshots? Will check to see what I can find based on those details, ~Mo.
- 0 kudos
- 86 Views
- 0 replies
- 0 kudos
enable databricks in azure for the whole tenant
Hi,We have to enable the databricks in azure cloud for the whole organisation in azure.I want to know the very 1st level i.e how to enable or integrate the databricks account with azure tenant and after that how I can enbale it for selected subscript...
- 86 Views
- 0 replies
- 0 kudos
- 217 Views
- 2 replies
- 0 kudos
select table_changes() return unexpected results
Hi Folks,My understanding isSELECT *FROM table_changes('mytable', x)should return all changes from version x through the latest available version, regardless of when I run it. However what happen to me is:When immediately after a streaming table upda...
- 217 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Thank you, Ashwin, for looking into my question.My table is appended whenever a new file arrives. Each file contains the same number of records (8,676), but with a different extraction date.Regarding the results of your query above, two additional fi...
- 0 kudos
- 296 Views
- 4 replies
- 0 kudos
Integration of mongodb and Databricks Serverless
Hi, I'm looking to integrate the databricks serverless workspace with Mongodb atlas through the private endpoint route. Created a NCC rule from databricks. From Mongodb side I cannot see the request to approve the connection, Thanks, Tarun
- 296 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
I have been exploring similar integration setups, and the connection between MongoDB and Databricks Serverless looks interesting, especially for teams managing large data workflows. Proper configuration of connectors, authentication, and data pipelin...
- 0 kudos
| User | Count |
|---|---|
| 1837 | |
| 885 | |
| 793 | |
| 472 | |
| 312 |
