Hey Databricks Community! Join us for a free, community-sponsored BrickTalk on Thursday, November 5! BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with data and ...
- 527 Views
- 0 replies
- 2 kudos
2 weeks ago
NOVEMBER · THE TOUR IS ON THE MOVE Nine stops. Three regions.One epic November. THIS MONTH'S ROUTE Munich → Atlanta → Amsterdam → London → Toronto → Los Angeles (Anaheim) → New York City → Milan → Tokyo A free, one-day event on how organizations...
- 1140 Views
- 1 replies
- 0 kudos
a week ago
DATA + AI LIVE Time: AMER Nov 11 / 9 AM PT / 12 PM ET APAC Nov 12 / 9:30 AM IST / 12 PM SGT / 1 PM JST-KST / 3 PM AEDT EMEA Nov 17 / 9 AM GMT / 10 AM CET Register for the Workshop Note: Marking RSVP on this Community event does not register you fo...
- 980 Views
- 0 replies
- 2 kudos
2 weeks ago
REGISTRATION IS NOW OPEN The Data + AI World Tour is heading your way A free, one-day event on how organizations are building AI they can trust to deliver real outcomes - by grounding models in their own enterprise data and business context. THE...
- 3577 Views
- 0 replies
- 4 kudos
09-01-2026
Free · Hands-on · Instructor-led AI/BI + Genie Launchpad From dashboards your team can trust to answers in plain language with Genie. A three-day arc of live, hands-on sessions, each built for a different role. Join only the day or days that fit you...
- 663 Views
- 0 replies
- 1 kudos
a week ago
Responsible gaming initiatives are becoming increasingly important as regulation, gameplay experiences and the betting and gaming industry continue to expand. The Responsible Gaming Solution Accelerator, developed in partnership with AWS, shows how t...
- 504 Views
- 1 replies
- 4 kudos
2 weeks ago
Recap • September 14 – October 11 Community Pulse Four weeks of articles, learning content, and conversations – the highlights, all in one read. Community Stars The members who showed up and made a real difference : @ivanvyd @data_pulse @Tho...
- 708 Views
- 0 replies
- 4 kudos
3 weeks ago
Community Activity
- 29 Views
- 1 replies
- 0 kudos
Unable to create metastore in account console with my Azure Account
Azure Managed Identity Credential with Access Connector Id /subscriptions/c03723bd-79a8-401a-8392-a6aef2b0d7b9/resourceGroups/RG_databricks_masterclass/providers/Microsoft.Databricks/accessConnectors/ac_meta_new and Managed Identity ID: for Account I...
- 29 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @mdpashainfo ,Thanks for sharing the full error. This message means the access connector exists in Azure, but Databricks can't match it to your Databricks account. Usually one of two things causes that:The managed identity isn't enabled on the con...
- 0 kudos
- 42 Views
- 2 replies
- 0 kudos
Zerobus support for private-endpoint storage (MLflow traces to Unity Catalog)
We'd like to store MLflow 3 traces in Unity Catalog, using an experiment linked to a UC table-prefix trace location. In our Azure UK South workspace, linking the location fails with:PERMISSION_DENIED: Zerobus failed to open the table. Make sure the t...
- 42 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Private endpoint storage for traces remains a listed limitation. Store Open Telemetry traces in Unity Catalog lists this as a limitation among other constraints, Traces cannot yet be written to storage protected by Private Link. It is stated as a kno...
- 0 kudos
- 180200 Views
- 245 replies
- 46 kudos
Resolved! Databricks Certification Exam Suspended? Our Support Team Can Help
Taking exams is so stressful already, and if your exam gets suspended it is extra stressful. Don't worry! Our support team is awesome and will help you get back on track. The most important step is to file a ticket in our Help Center. Be sure to inc...
- 180200 Views
- 245 replies
- 46 kudos
Latest Reply
- 46 kudos
Hello @Cert-Team @cert-ops,My exam was suspended during the initial checks, before the exam started.I submitted an appeal (#01053519) on 23 October after my original request (#01052483) was closed. It has been a week since my scheduled exam, and I ha...
- 46 kudos
- 39 Views
- 2 replies
- 0 kudos
Best approach and tools for migrating from Oracle to Databricks?
Hey everyone,We're planning to migrate our data warehouse from Oracle to Databricks. We have a mix of standard tables and a good amount of legacy PL/SQL code to deal with.What is the best way to approach this migration? specifically:1) How do you usu...
- 39 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
For converting legacy PL/SQL, begin with Genie Migrate (agentic SQL converter). It supports the Oracle dialect and outputs either Databricks SQL or notebooks using the DataFrame API, while iteratively validating and fixing the converted code before w...
- 0 kudos
- 141 Views
- 2 replies
- 0 kudos
ai_decide() failing with Endpoint is not enabled for batch inference in Free Edition
I'm running ai_decide() in a notebook (SQL cell, serverless compute) and it's now consistently failing with:[INTERNAL_ERROR] Remote session creation failed: INVALID_ARGUMENT: BAD_REQUEST: Endpoint ai_decide is not enabled for batch inference in this ...
- 141 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
The issue is tied to how Databricks Free Edition handles model serving resources. Specifically, the Free Edition imposes hard restrictions on these capabilities - there is no support for batch inference endpoints, no custom models and certain foundat...
- 0 kudos
- 337 Views
- 4 replies
- 0 kudos
Cluster environment variables not visible to Java Dataset.foreachPartition / mapPartitions
We run a Spark JAR task (Java) on a standard access mode cluster. Credentials come from cluster environmentvariables set as secret references in the cluster's Environment variables field, e.g.MY_PASSWORD={{secrets/<scope>/<key>}}.The variables are vi...
- 337 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @capoorhimanshu,Your worry about the closure is fair. A secret that rides inside the serialized task is harder to control than one that never leaves a secret store.As far as I can tell, there's no supported setting on standard access mode that pus...
- 0 kudos
- 82 Views
- 3 replies
- 0 kudos
Free Edition — S3 storage credential validation fails at "Assume Role" though IAM config is verified
Workspace ID 7474651448614968 (Free Edition, AWS). Credential databricks-dataengineer-v2 (IAM role arn:aws:iam::278137443935:role/DATABRICKS-DATAENGINEER) fails validation at the Assume Role step; CREATE EXTERNAL LOCATION returns PERMISSION_DENIED .....
- 82 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Arisk, I agree with @DoTA that your AWS setup looks right. The master role ARN you're using matches the one in the AWS setup docs (https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual), and I coul...
- 0 kudos
- 26 Views
- 0 replies
- 0 kudos
Moving a partitioned Delta table to liquid clustering without rebuilding it
If you have a big Delta table that was partitioned years ago, you probably know the feeling. Half the partitions are tiny, a few are huge, and queries that filter on anything other than the partition column scan far more than they should. Liquid clus...
- 26 Views
- 0 replies
- 0 kudos
- 115 Views
- 1 replies
- 0 kudos
Why I started evaluating the pipeline instead of choosing the best model at each stage
I recently worked on a multi-label classification pipeline on Databricks, using embeddings, scikit-learn classifiers, Delta Lake, and MLflow to track experiments.One result caught my attention: the embedding model that ranked fourth in my initial scr...
- 115 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Your fourth-place embedding winning the full combo is a great example of why picking the best model at each stage can mislead you. Stage metrics measure each piece alone, but the classifier and threshold only see what the embedding hands them.What te...
- 0 kudos
- 589 Views
- 2 replies
- 2 kudos
Resolved! Streaming read fails with "Detected a data update" after a restatement job touches the source
Hi everyone,Looking for the right pattern here rather than a workaround.Setup. DBR 15.4 LTS, Unity Catalog. A silver streaming table reads from a bronze Delta table with a normal streaming read. Bronze is append only in the ordinary course of busines...
- 589 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi @Islam_hoti, good instinct not to just slap skipChangeCommits on and walk away. Both of your ideas work. Which one I'd pick depends on how often corrections happen.Option A: rare, batched corrections (what you have). The Delta streaming docs descr...
- 2 kudos
- 422 Views
- 2 replies
- 0 kudos
Resolved! "Organization permissions" in DAB's
Hi all,I've created a Databricks app in git, and it is deployed through DAB's. I want my team to have access to it and manage it and view insights and logs thus i have added in the dab CAN_MANAGE permission to my App DAB to the appropriate group.When...
- 422 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Nice find, and you're right. That group isn't really hidden. The Apps permissions doc says "Anyone in my organization can use" simply gives CAN USE to the All account users group, which shows up as account users when you pull the app with the CLI.The...
- 0 kudos
- 54 Views
- 1 replies
- 0 kudos
Managed table conversion and vacuum
Hello everyone,I am looking for clarification about the 14-day rollback period after converting an external table to a managed table using: ALTER TABLE <table_name> SET MANAGEDDatabricks document...
- 54 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Davide, this thread never got a reply, but you asked the same thing in SET MANAGED and rollback in relation to VACUUM and posted your test result there, so I'm linking it for anyone who lands here first.Short version: the 14-day rollback window i...
- 0 kudos
- 80 Views
- 2 replies
- 0 kudos
Real time Streaming to AI Engine at Databricks
Hi there,Usercase: Fraud Detection Technology : Databricks, Kafka, Microsoft Foundry, Dashboard showing real-time reportsAsk: Please let me know if any reference or technology details to achieve 1-2 milli second (Max) latency using above technology s...
- 80 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @abhietc31 I assume this is what you are looking for :https://docs.databricks.com/aws/en/structured-streaming/real-time/conceptsPossible in both pyspark structured streaming pipelines or lakeflow pipelines. Depends how you are curating your pipeli...
- 0 kudos
- 43 Views
- 1 replies
- 0 kudos
Add resource to app space
Hi,I recently started trying out the Genie App Builder first presented at DAIS. For this I created an app space and I wanted to connect my app to a lakebase project. But I can't seem to find any options to do this like it works with apps which are no...
- 43 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @JanW Based on the current documentation, this looks like a limitation of the new Genie App Builder + App Spaces model.Genie App Builder apps are created inside an App Space, and App Spaces currently define governance mainly through user authoriz...
- 0 kudos
- 36 Views
- 0 replies
- 0 kudos
Databricks Free Edition – AWS S3 Storage Credential Validation Fails at Assume Role
Hi Databricks Team,I'm trying to connect an AWS S3 bucket to Databricks Unity Catalog using an IAM role in Databricks Free Edition.EnvironmentDatabricks Edition: Free EditionCloud: AWSWorkspace region: us-east-2Metastore region: us-east-2S3 bucket re...
- 36 Views
- 0 replies
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2293 | |
| 1505 | |
| 1255 | |
| 1189 | |
| 961 |
Latest from our Blog
Serving the Lakebase Data API Through Azure API Management
Why use APIM with Data API? The Data API is one of the underrated features of Lakebase. Traditionally, teams that wanted to share a database over HTTP built a layer of REST services in front of it. Th...
[PARTNER BLOG] Scaling Auto Loader on AWS: Moving to File Notification Mode with Managed File Events
Contents IntroductionUnderstanding the Key ConceptsWhy File Notification with Managed File Events?Key Benefits of Managed File EventsArchitecture OverviewPrerequisitesPhase 1: AWS IAM SetupPhase 2: Da...
End-to-End Cyber Lakehouse Streaming via Zerobus Ingest, Spark Declarative Pipelines, and OCSF
Streaming raw events into a cyber lakehouse and normalizing them to a standard schema is a common pattern for security analytics. This blog walks through an end-to-end architecture that uses Databrick...
AI Functions for Synthetic Test Data: Generate Realistic Datasets in Pure SQL
The views expressed in this post are my own and do not represent those of my employer. Not long ago, generating realistic synthetic data meant building custom pipelines with GANs, Transformers, or RN...
