Hi,So i've been playing around with transformWithState over the last few weeks and have got the code to work exactly as desired.Wrote some integration tests that execute fine within the workspace in serverless and everything, the issue is however is ...
Hi Databricks Community,I am trying to enable Unity Catalog in my Azure Databricks account. As part of the setup, I created an Azure Databricks Access Connector and assigned its managed identity the required permissions on my ADLS Gen2 storage accoun...
Hey, Check the IAMs for both connector and ADLS, your problem will be solved...I have come across the same issue and have cleared it by assigning a role through IAM. LMK, if you have solved it Thanks
When I first started working with streaming in Databricks, I had a very simple picture in my head:Streaming = keep the cluster running and process data continuously.New data arrives → Spark sees it → process it → repeat forever.Technically, that can ...
Hello.Ive created a custom cluster - m4.large. When i try to execute some code in this cluster the behaviour is as follows:- Cluster starts, have running status- I run code, for example print("Hello")- Code runs indefinitely- I click interrupt, it st...
Hey , @darek554 I'd bet this is just the instance being too small. m4.large is 2 cores / 8 GB, and on a single node the driver and executor have to share that usually the executor never actually comes up, so your command just sits there waiting for ...
HelloI have a question about how VACUUM works.We executed the following VACUUM command from a PySpark job:spark.sql("""VACUUM catalog_name.schema_name.table_name RETAIN 168 HOURS""")The Delta table has a table property retention period of 365 days, b...
Hello Louis,Thank you so much for the incredibly detailed and helpful answer – it was a great help. I have the following question:We have a daily job that loads data from one Databricks schema into one or more Databricks schemas using CDF.In addition...
Hi All,We are trying to create a Databricks agent and were looking for a step by step approach if there is any. We did go through the Author an agent and deploy it on Databricks Apps | Databricks on AWS but weren't very clear on creation of a new age...
If you're approaching this specifically as a migration validation use case, I'd separate the agent into two layers: the validation tools and the agent reasoning layer.For example, expose controlled tools for querying Hive and Databricks, then have th...
Hi,When reading Delta Lake file (created by Auto Loader) with this code: df = ( spark.readStream .format('cloudFiles') .option("cloudFiles.format", "delta") .option("cloudFiles.schemaLocation", f"{silver_path}/_checkpoint") .load(bronz...
The key point is using the correct format for Delta tables rather than treating them like raw files. The same idea of using the right tool for the job applies in other technical workflows too, such as a delta executor apk when handling Android relate...
Calling get_token() on a Unity Catalog service credential fails for any scope using the api:// App ID URI format. Only https://-scheme resource scopes succeed. The same api:// scopes work correctly with a service principal.Reproductioncredential = db...
Hello, a quick update on thisI reached out to the databricks support team and they confirm that Unity Catalog service credentials do not support "api://" URIs. this is a known product limitation. They asked me to file for a feature request with Datab...
Hi experts ,now databricks for support for Iceberg open table format conversion yet not fully though.Databricks calls this Delta Lake UniForm (Universal Format). When tentatively Databricks can support fully open table format Iceberg, Hudi etc.Is the...
I have bronze layer data which is directly referred in silver. there are one or two columns from bronze layer which are used to build silver column.is there any way I can have column-level lineage shown in databricks like table lineage@DATA @lineage
Hi @prathamesh1982 , Good news: there's nothing to enable. @balajij8 has it right, Unity Catalog captures column-level lineage automatically. I'll add some prerequisites and gotchas to check if the lineage doesn't appear. If your bronze and silver t...
Hi All,If you run streaming or continuous pipelines that commit to Delta tables on S3 every few minutes, your _delta_log folders are probably much bigger than your tables. A CDC or MERGE flow commits every cycle whether or not data arrived, and with ...
@brianjorge178 @cartergray70543 Thanks for your thoughts. I had a long chat with Databricks support team, and here is where have landed so far if it interests you:What we believed. The pipelines committed every few minutes, so each table's transactio...
Are there any settings that can be used to influence the frequency at which the auto SCD API runs the tombstone garbage collection process in Spark Declarative Pipelines? I've seen a couple community posts that referenced the following: pipelines.a...
Hi @de01 ,Short answer: Those two settings (pipelines.applyChanges.tombstoneGCFrequencyInSeconds and pipelines.cdc.tombstoneGCFrequencyInSeconds) are internal/undocumented configurations. They are not part of the public API surface and may have been ...
I’ve been thinking about how enterprise AI is changing, especially after attending the Data + AI Summit 2026 and hearing Ali Ghodsi emphasize the importance of enterprise context. It made the direction much clearer to me. We already have very capable...
There is one thing I'd add which is that many large enterprises still struggle with governance, ownership and sometimes even with common definitions of what their data means. The interesting question is not only how much context we can give to tools ...
Hi,Is there a plan to have if statement support in DAB YAML file?For example, I would like to have different schedule for jobs based on the environment (higher frequency in PROD, lower frequency in DEV, QA). How can I do it in Databricks via workflow...
Hi @QuanDo1 You don't need to use if/else for this use case. Databricks Asset Bundles (DAB) already supports variables that can be overridden per target environment.You can define your variables once and then assign different values for each target (...
Hi,I'm seeing inconsistent behavior when reading Excel files using the built-in connector Lakeflow Connector with spark.read.format("excel") (doc: https://docs.databricks.com/aws/en/query/formats/excel). I read an .xlsx file from S3 using this functi...
Yes, this can happen with Excel readers when formula evaluation/cached values are handled differently. In an .xlsx file, the formula itself and its last calculated value are stored separately, so the reader may return the formula string if it doesn't...
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
