Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best prac...
1391 Posts
Activity in Databricks Platform Discussions
- 19 Views
- 0 replies
- 0 kudos
Databricks Free Edition GitHub push fails with "Resolver error"
Hi,I'm using Databricks Free Edition and connected a GitHub repository through the Databricks GitHub integration. I can clone/connect to the repository and create/edit files, but when I try to Commit & Push, Databricks returns:Error pushing changesRe...
- 19 Views
- 0 replies
- 0 kudos
- 158 Views
- 4 replies
- 4 kudos
Feature enablement for Foundation Model Unity Catalog permissions
I am trying to restrict workspace users' access to Databricks Foundation Models using the guidance in the Foundation Model Unity Catalog Permissions documentation.I have revoked EXECUTE permission for all users from the system.ai schema. However, wor...
- 158 Views
- 4 replies
- 4 kudos
Latest Reply
- 4 kudos
It persists after enablement. The legacy databricks-* endpoints are a separate serving path from both pay-per-token and provisioned throughput, so they aren't gated by the system.ai EXECUTE revoke. It isn't pre-enablement behaviour that clears once t...
- 4 kudos
- 542 Views
- 11 replies
- 1 kudos
Can I run jobs continuously without interruption on a permanent free (Community Edition) account?
Chào mọi người,Tôi hiện đang sử dụng Databricks Community Edition (tài khoản miễn phí) và muốn hỏi về việc chạy các tác vụ theo lịch trình/liên tục trên đó.Cụ thể: 1. Liệu có thể chạy một tác vụ liên tục (ví dụ: tác vụ xử lý dữ liệu trực tuyến hoặc t...
- 542 Views
- 11 replies
- 1 kudos
Latest Reply
- 1 kudos
@FastFoodBro You can use Databricks Free account for personal use only (personal use, learning, experimentation - not for production or commercial use). You can upgrade to a paid plan to access full platform features as you will face interruptions in...
- 1 kudos
- 20031 Views
- 2 replies
- 2 kudos
ADD COLUMN IF NOT EXISTS does not recognize "IF NOT EXIST". How do I add a column to an existing delta table with SQL if the column does not already exist?
How do I add a column to an existing delta table with SQL if the column does not already exist?I am using the following code: <%sqlALTER TABLE table_name ADD COLUMN IF NOT EXISTS column_name type; >but it prints the error: <[PARSE_SYNTAX_ERROR] Synta...
- 20031 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Here some SQL devilry that does not require SQL Scripting nor pythons. It adds a my_extra_column FLOAT to the target table with null values. I'm as shocked as anyone that this works, (and if it's undocumented behavior, someone in Databricks Command ...
- 2 kudos
- 76 Views
- 1 replies
- 0 kudos
Streaming read fails with "Detected a data update" after a restatement job touches the source
Hi everyone,Looking for the right pattern here rather than a workaround.Setup. DBR 15.4 LTS, Unity Catalog. A silver streaming table reads from a bronze Delta table with a normal streaming read. Bronze is append only in the ordinary course of busines...
- 76 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
On DBR 15.4 LTS, use legacy CDF when corrections must reach silver, provided CDF was enabled before this UPDATE and the required version remains retained (source changes, CDF availability and retention). Before MERGE, reduce each microbatch to one f...
- 0 kudos
- 49 Views
- 0 replies
- 0 kudos
Before dropping a column, check whether Genie has been using it
I was reviewing a schema change recently and wanted a quick answer to a simple question:Could this column be used by a Genie Agent somewhere without me realizing it?There's now a pretty useful way to check I want to share with you.Databricks' March 2...
- 49 Views
- 0 replies
- 0 kudos
- 92 Views
- 0 replies
- 0 kudos
Login Error Assistance - Reference ID: a728332e-daf7-436d-a676-99ac4b3b0033
Unable to log in to the Databricks account to opt for the subscription to use the compute options, but in Azure, able to create workflows.
- 92 Views
- 0 replies
- 0 kudos
- 78 Views
- 1 replies
- 0 kudos
Error with Databricks ODBC Driver OAuth 2.0
Hi,I am facing some issues while configuring Databricks ODBC driver for Tableau connection purposes. After setting up all the parameters and settings I get the following error when I use OAuth 2.0 mechanism with Browser Based Authorization Code. Has ...
- 78 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@Hatim_Vohra The error is happening before Databricks authentication completes because ODBC driver is using browser-based OAuth, but it isn’t able to launch the browser.If you want to continue with the ODBC, check the OAuth options and enable Ignore ...
- 0 kudos
- 119 Views
- 1 replies
- 2 kudos
Secrets Redaction seems to work only when the retrieved value is assigned to a varible
I randomly encountered the following behavior when setting up Big Query federation in Databricks Free Edition and wondered if this is expected behavior or just something that happens in the Executing cell 1 as captured in the screenshot prints out th...
- 119 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
hi, this is expected behaviour and is not specific to Free Edition. Databricks documents that a secret assigned through dbutils.secrets.get() and subsequently printed should be replaced with [REDACTED]. Redaction is best-effort and applies to literal...
- 2 kudos
- 104 Views
- 2 replies
- 4 kudos
Photon enabled but a large share of the plan is falling back, cost up and runtime flat
Hi everyone,Trying to work out whether this is expected or whether I have misconfigured something.We enabled Photon on a job cluster running a nightly aggregation over roughly 2TB. The expectation was the usual improvement. What we got instead was ru...
- 104 Views
- 2 replies
- 4 kudos
Latest Reply
- 4 kudos
Really useful real-world scenario. I’d be interested to see how others approach pinpointing Photon fallbacks, especially with Python UDFs and nested structs. A before/after comparison of a UDF rewrite would also be a great way to quantify whether the...
- 4 kudos
- 170 Views
- 2 replies
- 2 kudos
Image Annotation
What is Image Annotation?What are the Steps of Image Annotation?What are the Different Techniques of Image Annotation?Types Used in Image AnnotationHow are Companies Handling Image Annotation?Features to Look for in Image Annotation Service Providers...
- 170 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Great overview of image annotation! The sections on different techniques, real-world use cases, and pricing factors are especially helpful for anyone getting started with AI/ML. Looking forward to exploring the future trends in image annotation!
- 2 kudos
- 510 Views
- 5 replies
- 5 kudos
Resolved! Difference between Workspace and Unity Catalog experiments when using MLflow autologging?
Hi everyone,I am trying to understand the exact differences between using Workspace experiments versus Unity Catalog experiments, specifically in the context of MLflow autologging (mlflow.autolog()).Does autologging behave differently depending on wh...
- 510 Views
- 5 replies
- 5 kudos
Latest Reply
- 5 kudos
Great question! I’d also be interested to know if autologging has any behavioral or permission differences with Unity Catalog experiments, especially around governance and model lineage.
- 5 kudos
- 76 Views
- 1 replies
- 0 kudos
At what data size do you stop reaching for Spark?
Hi everyone,A question I keep having with my team and I would like to hear how others think about it.A lot of the jobs we run are not big. Plenty of our pipelines process a few gigabytes, some considerably less. We run them on Spark because that is w...
- 76 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Eu evitaria definir um limite rígido de GB.No Databricks, eu manteria o Spark como mecanismo de execução padrão para padronização, governança, observabilidade e evolução mais fácil das cargas de trabalho. Para cargas de trabalho pequenas, a otimizaçã...
- 0 kudos
- 157 Views
- 2 replies
- 2 kudos
Best practices for SLA monitoring and automated retries across hundreds of Lakeflow Jobs
Hi everyone,We are in the process of migrating our legacy Airflow DAGs (largely time-based cron schedules) to Lakeflow Jobs, and I want to fully leverage the platform's event-driven capabilities rather than just replicating the old "timer-based" patt...
- 157 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi,Migrating the same way. The messy parts.On file arrival triggers, first check whether your external locations have file events enabled, because the behaviour differs sharply. Without them you get a cap of 50 triggered jobs per workspace and at mos...
- 2 kudos
- 185 Views
- 4 replies
- 4 kudos
Non-deterministic ROW_NUMBER() results across Unity Catalog environments
We have a notebook that calculates the "mode" (most frequent value) of a customer_parent_name field using ROW_NUMBER() OVER (PARTITION BY ... ORDER BY count DESC). The exact same notebook code is deployed to two Unity Catalog-governed environments (D...
- 185 Views
- 4 replies
- 4 kudos
Latest Reply
- 4 kudos
Hi,Your diagnosis is correct, and the fix is simpler than you are expecting.On question one, yes, this is expected and it is not a Databricks or Unity Catalog behaviour. ROW_NUMBER with an ORDER BY that does not define a total order is non determinis...
- 4 kudos
| User | Count |
|---|---|
| 1838 | |
| 885 | |
| 809 | |
| 472 | |
| 313 |
