- 698 Views
- 4 replies
- 0 kudos
Databricks Access Issue with UC
Hi All ,We are facing issues while reading Storage account where stream data from data verse in Unity catalog through External table but not every time . It was running fine with hiveAn error occurred while calling o393.sql.: org.apache.spark.SparkEx...
- 698 Views
- 4 replies
- 0 kudos
- 0 kudos
one of these:External location path doesn't exactly match your ABFS path in Unity CatalogStorage credential missing Storage Blob Data Contributor on that containerService principal token expiring mid-jobTable LOCATION path changed slightly during mig...
- 0 kudos
- 249 Views
- 2 replies
- 0 kudos
foreachPartition
Is there any difference between pyspark.RDD.foreachPartition vs pyspark.sql.DataFrame.foreachPartition under the hood? The PySpark documentation describes pyspark.sql.DataFrame.foreachPartition as "a shorthand for df.rdd.foreachPartition()"If DataFra...
- 249 Views
- 2 replies
- 0 kudos
- 0 kudos
keep what's working, migrate only if your use case grows beyond side effects.
- 0 kudos
- 21 Views
- 1 replies
- 0 kudos
Sizing Tables and delt logs/CDF
Hi,I need to compare the sizes of my delta tables , what's the correct approach ?Table size reported by analyze command ? , but how do I check the delta log size , if I enable CDF .. how do I know the CDF log size(the overhead it adds) ? , kind of l...
- 21 Views
- 1 replies
- 0 kudos
- 0 kudos
DESCRIBE DETAIL is the go-to for data size but it only tells you half the story. The delta log and CDF overhead are invisible to it they sit in separate folders on storage and you have to check them directly.Think of it as three buckets for any Delta...
- 0 kudos
- 79 Views
- 1 replies
- 2 kudos
Why We Moved Our Operational Database Into Databricks — And Stopped Managing Two Stacks
Lakebase just went GA. Here's what a production migration actually looks like.For most of the last decade, our data infrastructure lived in two separate worlds.On one side: a transactional database handling operational workloads — the writes, the loo...
- 79 Views
- 1 replies
- 2 kudos
- 2 kudos
Great write up and felt useful. Thanks for sharing the real experience.!
- 2 kudos
- 2299 Views
- 3 replies
- 0 kudos
Resolved! Download all pages of a multi-page dashboard
Hi,I have created a multi-page dashboard in databricks. I want to download all the pages of the dashboard as a single pdf file. But when i export the dashboard I get it only in .json format. Is there a way to download all the pages as a pdf file?
- 2299 Views
- 3 replies
- 0 kudos
- 0 kudos
Dashboard provides a Download as PDF capability for published dashboards. You can distribute a multi-page dashboard as a PDF with all pages & configure a scheduled email subscription and include all dashboard pages in the generated PDF.You can follow...
- 0 kudos
- 138 Views
- 2 replies
- 2 kudos
Resolved! Genie code Customization
Hi,I use Genie code extensively for research , plan and development for building ETL scripts and code migrations.As per my knowledge Databricks manages the backend LLM models for Genie code agent.I wanted to try Genie code with Frontier models for my...
- 138 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Mailendiran, From what’s publicly documented, Genie Code already uses frontier models behind the scenes, but it isn’t exposed as a bring-your-own-model or manual model-selection experience. Databricks describes Genie Code as an agentic system tha...
- 2 kudos
- 103 Views
- 1 replies
- 0 kudos
How to change a field when instancing cluster defined as variable?
I define all clusters as variable in separate files, so I can re-use them. Then I am accessing them in jobs as: The issue is that I want to change just the custom_tags in the cluster when instancing it for a job, cause my tags are different for each ...
- 103 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, you can achieve this seamlessly, but not by overriding the custom_tags inside the cluster variable. Instead, you define your specific tags at the Job level, and Databricks automatically merges them with your cluster variable's tags.Because compl...
- 0 kudos
- 160 Views
- 3 replies
- 1 kudos
Best practice to log Autoloader UNKNOWN_FIELD_EXCEPTION
Hi, When schema evolution is detected, Auto Loader throws an UNKNOWN_FIELD_EXCEPTION, and the error message includes schema information along with other related details. However, when I log the full message, it is too long and contains information th...
- 160 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @bi_123, I would avoid parsing the full rendered UNKNOWN_FIELD_EXCEPTION message. Databricks explicitly notes in the error-handling documentation that the rendered and parameterised messages are not stable across releases, so any logic that depend...
- 1 kudos
- 111 Views
- 1 replies
- 0 kudos
ANALYZE TABLE ... COMPUTE STATISTICS FOR ALL COLUMNS silently does not update column stats on DBR 15
Hi Databricks Support,We need help confirming whether this is a known DBR 15.4 LTS bug or an unsupported/configuration-specific behavior.SummaryOn a Databricks Runtime 15.4.40 Photon job cluster, `ANALYZE TABLE <catalog>.<schema>.<table> COMPUTE STAT...
- 111 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @sd1700092, From what I can verify, this looks more like a DBR 15.4 job-cluster issue than expected behaviour. The public ANALYZE TABLE documentation is clear that ANALYZE TABLE ... COMPUTE STATISTICS FOR ALL COLUMNS applies to both Databricks Run...
- 0 kudos
- 202 Views
- 1 replies
- 5 kudos
My experience replacing a Postgres → Kafka → DMS → S3 pipeline with Lakeflow Connect
Sharing my hands-on experience with Lakeflow Connect for anyone evaluating it for database ingestion. I recently moved data from PostgreSQL on AWS RDS into Databricks, and it replaced a painful legacy pipeline. Keeping this simple and practical.What ...
- 202 Views
- 1 replies
- 5 kudos
- 5 kudos
Great article!
- 5 kudos
- 108 Views
- 1 replies
- 0 kudos
Can we do a shallow clone on top of a shallow clone!
Case 1 Can we do a shallow clone on top of a shallow clone? If I do so, What would be the result Case 2 when ever the table over writes with new data how does it work.Case 3 when a the table is dropped and loaded then how does this scenario effect ...
- 108 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @damodhargandha, A good way to think about a shallow clone is that it copies the table’s metadata but still points to the source table's data files rather than copying them. Databricks explains that behaviour in the Clone a table " section of the ...
- 0 kudos
- 11662 Views
- 9 replies
- 1 kudos
Error when reading delta lake files with Auto Loader
Hi,When reading Delta Lake file (created by Auto Loader) with this code: df = ( spark.readStream .format('cloudFiles') .option("cloudFiles.format", "delta") .option("cloudFiles.schemaLocation", f"{silver_path}/_checkpoint") .load(bronz...
- 11662 Views
- 9 replies
- 1 kudos
- 1 kudos
I've been having similar issues with reading Delta Lake files and I think the solution lies in adjusting the format option. When working with Delta tables, it's essential to use the correct format to avoid compatibility issues. This reminds me of the...
- 1 kudos
- 164 Views
- 1 replies
- 0 kudos
databricks-connect library for python and pandas 3
Hello,databricks-connect is pinning pandas during the installation. Since we're moving towards pandas 3 can you please add the support for the newest version as well?
- 164 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Albertino , how are you doing today?Thanks for calling this out. as per my understanding, at the moment, the pandas pin is there intentionally. Databricks Connect release notes say supported pandas versions are currently limited to 1.0.5<=pandas<...
- 0 kudos
- 156 Views
- 1 replies
- 1 kudos
Databricks apps
I have multiple Databricks Apps running, but their usage is not fixed or predictable. Some apps are used only occasionally, while others may remain idle for long periods.Since Databricks Apps need to stay up and continue consuming resources even when...
- 156 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @anmolhhns, I couldn't find any public documentation showing that Databricks Apps supports automatic idle shutdown or usage-based scale-to-zero for the app runtime itself. The current documented lifecycle is that an app can be Running, Stopped, De...
- 1 kudos
- 168 Views
- 1 replies
- 0 kudos
From RAG Demo to Production on Databricks: 7 Things Teams Should Validate First
From RAG Demo to Production on Databricks: 7 Things Teams Should Validate FirstBy Naveen AyallaMany teams can build a RAG demo quickly.Upload documents, create embeddings, connect a model, ask a question, and show an answer.But production is differen...
- 168 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for reading. I’m especially interested in hearing from people who have worked on real RAG or GenAI workflows.Which one has been the biggest challenge for your team?1. Choosing the right source data2. Access control and governance3. Improving r...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
