- 903 Views
- 2 replies
- 1 kudos
Resolved! Auto Loader on UC Volumes stopped resolving wildcards
The following spark.readStream / cloudFiles configuration was confirmed working on2026-04-30, but stopped working on 2026-05-26. No code or config changes were madebetween these dates, so I assume something was changed implicitly on the Databricks si...
- 903 Views
- 2 replies
- 1 kudos
- 1 kudos
We are facing a similar issue, not limited to Autoloader but also affecting DLT pipelines and classic ETL job. The behavior is intermittent, jobs run fine and then fail unexpectedly, though they typically succeed on retry if retries are enabled. We t...
- 1 kudos
- 2766 Views
- 1 replies
- 0 kudos
Resolved! Serverless compute outbound IP whitelisting for external API calls
I’m trying to understand the networking implications of moving some logic to Databricks Serverless / SDP.My current setup is a notebook running as a job on classic compute, and this works because outbound traffic goes through a NAT Gateway, so we can...
- 2766 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @mnissen1337, Your understanding is basically right. With classic compute, the workload runs in your own VNet/VPC, so using your own NAT Gateway to present a stable public egress IP is a standard pattern. With serverless, the compute runs in the D...
- 0 kudos
- 1447 Views
- 5 replies
- 1 kudos
Resolved! Rendering HTML in ipywidgets output
Hi, I am experimenting with IPywidgets on Databricks (company Azure account and Free Edition).I have an HTML that I would like to render so typically what happens is that simple things like:import ipywidgets as widgets output = widgets.Output() wit...
- 1447 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @vvanag, What you’re seeing is expected to some extent in Databricks Notebooks. Databricks supports ipywidgets, but it doesn’t guarantee full Jupyter or Colab parity, and there are a few documented limitations on how widgets render and behave in n...
- 1 kudos
- 708 Views
- 1 replies
- 1 kudos
Resolved! Oracle HIVE Metadata to Databricks UC migration
Dear Databricks Member, Could you please recommend the best solution for the following use case? Use Case: The Hive Metastore is currently used to manage metadata and is backed by an Oracle database. This metastore stores critical information such as...
- 708 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @HarshVardhan1, The best approach for your scenario is usually not to migrate the Oracle database itself into Unity Catalog. Instead, the supported pattern is to migrate the Hive Metastore objects into Unity Catalog using Databricks migration tool...
- 1 kudos
- 1049 Views
- 3 replies
- 1 kudos
SCD2 table migration using LakeFlow
A source SQL DB of an operational systems delivers daily snapshots to a legacy DWH with SCD2 logic enabled. Now for a migration to Databricks. Lets look at the table "customer" (SCD2; with customer_id, valid_from and valid_to columns). On migration d...
- 1049 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @peter_hoeltschi !I think your workspace or user permissions only allow SQL/serverless compute not classi clusters because even if you have enterprise pay-as-you-go at the account level does not automatically mean every user can create all-purp...
- 1 kudos
- 1031 Views
- 4 replies
- 1 kudos
Inconsistent Cluster Log Persistence to Volume/S3 (stderr, stdout, log4j-active.log)
Saving logs from an all-purpose cluster to Volume or S3 is not consistent, because stderr, stdout, and log4j-active.log get overwritten when the cluster is restarted between minutes 01 and 59.Tested case:A job is configured to start every 20 minutes,...
- 1031 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @ccsalt , This is a known limitation. Log rotation (renaming to log4j-YYYY-MM-DD-HH.log.gz) only happens on the hour boundary. The active log file log4j-active.log has always the same name and is overwritten if a cluster restart happens within one...
- 1 kudos
- 21836 Views
- 7 replies
- 0 kudos
Incorrect reading csv format with inferSchema
Hi All,There is a CSV with a column ID (format: 8-digits & "D" at the end).When trying to read a csv with .option("inferSchema", "true"), it returns the ID as double and trim the "D". Is there any idea (apart from inferSchema=False) to get correct ...
- 21836 Views
- 7 replies
- 0 kudos
- 0 kudos
i am getting the following error when working in Databricks free edition newly logged inthe code i wrote :from pyspark.sql.functions import *data = "workspace.default.asl.csv"df = spark.read.format("csv").option("header","true").option("inferSchemacc...
- 0 kudos
- 9294 Views
- 5 replies
- 2 kudos
Resolved! Turn off AI assistance in notebooks
Hi, has anyone found a way that the AI assistant can be turned off in notebooks? I would be happy to keep code introspection but I find I'm more often hitting escape than accepting the AI's suggestions (or removing the code it has suggested when I ac...
- 9294 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Ashley1, There are a few different levels where you can control the AI assistance behavior in notebooks. Here is a breakdown: USER-LEVEL: DISABLE AI AUTOCOMPLETE (INLINE SUGGESTIONS) This is the setting that controls the "ghost text" inline code ...
- 2 kudos
- 1946 Views
- 2 replies
- 2 kudos
Resolved! Lakehouse Federation Join Pushdown
Hi,I experimented a little bit with lakehouse federation. I created a connection and foreign catalog that references a SQL Server and activated the public preview feature for Join Pushdowns. After finishing my experiments, it seems to me that the fea...
- 1946 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi,I have experimented a bit with join pushdowns and experienced that intra-schema-Joins are not pushed down if there is a table joined between that is part of another schema.Example:In the following example the Join between x1 and x2 is pushed down ...
- 2 kudos
- 1423 Views
- 5 replies
- 2 kudos
Resolved! spark.databricks.sql.excel.enabled false at cluster level
Native databricks excel data source is GAhttps://www.reddit.com/r/databricks/comments/1t4un82/native_excel_support_is_now_ga/https://docs.databricks.com/aws/en/query/formats/excelHowever, as long as it is not possible to read from another adress than...
- 1423 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @der ,Most likely because spark.databricks.sql.excel.enabled is a Databricks SQL/session-level internal config, not a SparkConf setting.This specific key appears to be read from the Spark SQL session config, so setting it after the notebook sessi...
- 2 kudos
- 667 Views
- 1 replies
- 2 kudos
Ingestion Gateway DDL Objects Missing - Lakeflow Connect
Facing below issue and need a solution to proceed furtherCategory: ErrorMessage: DDL objects missing on table 'DB.dbo.client'. Execute the DDL objects script and full refresh the table on the Ingestion Pipeline. Error message: 'Reason:- Catalog is no...
- 667 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @shan-databricks ,The INGESTION_GATEWAY_DDL_OBJECTS_MISSING error means that while CDC is enabled on DB.dbo.client, the LakeFlow-specific DDL support objects (triggers, stored procedures) that allow Databricks to track schema changes (DDL events l...
- 2 kudos
- 2643 Views
- 5 replies
- 1 kudos
Resolved! Lakeflow partial data ingestion for first load
Hi Team,I am doing ingestion of 10 tables from Azure SQL through Lakeflow connect. I have created gateway and ingestion pipelines using databricks SDK. I am starting ingestion pipeline only when gateway is in Running status with resources. I observed...
- 2643 Views
- 5 replies
- 1 kudos
- 1 kudos
HiI finally found a solution that works smoothly to capture the full snapshot on the initial run. Here is the step-by-step approach I implemented:Create a Status Check Function: I wrote a custom function that queries the event_log for a given Pipelin...
- 1 kudos
- 869 Views
- 2 replies
- 0 kudos
Cross-region S3 reads suddenly fail with 400 Bad Request — eu-west-1 metastore to af-south-1 bucket
What changedA production daily job that has worked unchanged for ~8 months started failing on 2026-05-18 ~23:46 UTC. The notebook does a plain spark.read.json("s3://BUCKET/...") against a bucket in af-south-1. The metastore is in eu-west-1. Same code...
- 869 Views
- 2 replies
- 0 kudos
- 0 kudos
Your debugging is really thorough and you've already done the hard work of isolating this. The 400 with an empty body (no proper S3 error code like InvalidArgument) on an opt-in region is almost always one thing: SigV4 signing region mismatch. af-sou...
- 0 kudos
- 2921 Views
- 11 replies
- 6 kudos
Resolved! R plots not rendering
Has anyone been experiencing the issue of R plots not rendering in notebooks, starting a few days ago?t's not related to splarkly or plotly, or specifc data types, or anything. For example in base R: plot(1:3, 5:7) calculates without error, but does ...
- 2921 Views
- 11 replies
- 6 kudos
- 6 kudos
Looks like the issue has been resolved. Thanks everyone for chiming in and thanks 'bricks for whatever you did to resolve this.Plankton out!
- 6 kudos
- 1146 Views
- 1 replies
- 1 kudos
DQX - datacontract cli
Hello Guyz, Someone can i combine dqx databricks rules check with datacontract cli ? If yes can we share your idea? https://gpt.datacontract.com/sources/cli.datacontract.com/Cordially,
- 1146 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @seefoods, Just came across this post. In case you are still looking for an answer, I see these as complementary rather than overlapping tools. A practical approach would be to keep the data contract as the source of truth in datacontract.yaml, us...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 585 | |
| 349 | |
| 287 |
