- 1386 Views
- 1 replies
- 0 kudos
Databricks & Bigquery
Databricks is packaging a old version of big-query jar(Databricks also repackaged and created a fat jar), and our application needs a latest jar. Now the latest jar depends on spark-bigquery-connector.properties file for a property scala.binary.vers...
- 1386 Views
- 1 replies
- 0 kudos
- 0 kudos
Classloader shadowing in Databricks Runtime (DBR) is a common headache when the built-in connector JARs take precedence over cluster-installed libraries.Instead of manually stripping and rebuilding the JAR, try setting spark.driver.userClassPathFirst...
- 0 kudos
- 144 Views
- 1 replies
- 0 kudos
Your Thoughts on the Modern MI/BI/Engineer Role Description?
Hi all,First post for me….……….we are building the dream that is value enablement through Databricks and are around 6 months in to our journey. I am going to hire in someone to compliment and build upon the team who are already driving us through the ...
- 144 Views
- 1 replies
- 0 kudos
- 0 kudos
Welcome, Mark. My view is that the hybrid exists, but I would be wary of writing it up as one job. The blend that seems most realistic to me is a BI developer who is strong in SQL and picks up a few engineering habits: version control, tests on the t...
- 0 kudos
- 385 Views
- 4 replies
- 8 kudos
Memory Profiling
How do you debug OOM on long-running streaming jobs?Hi All,One of my Structured Streaming jobs has started dying with OutOfMemoryError. Runs fine for about approx 40 hours, then falls over, restarts, and does the same thing again. I had introduced fe...
- 385 Views
- 4 replies
- 8 kudos
- 8 kudos
Hi Kiran,That's a really thorough round of checks. A flat heap floor, flat RocksDB and a small state store rule out the usual suspects, which narrows things down a lot.The one thing I'd grab next is the exact OutOfMemoryError line from the driver log...
- 8 kudos
- 114 Views
- 1 replies
- 0 kudos
- 114 Views
- 1 replies
- 0 kudos
- 0 kudos
It's a green pigment in leaves performing photosynthesis
- 0 kudos
- 320 Views
- 1 replies
- 1 kudos
Questions about databrick voucher
Hi,I have already received a Databricks exam voucher. Can I still join the current Learning Festival and qualify for another voucher?In addition, how can I check which courses count toward the festival requirements? I completed some of the courses du...
- 320 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @cakkitleung @, good questions. Here's what the event rules (https://community.databricks.com/t5/learning-events/databricks-advanced-learning-festival-november-10-december-08/ev-p/172328) say: Which courses count: you need to finish every module i...
- 1 kudos
- 79 Views
- 0 replies
- 0 kudos
Enagement Manager and Technical ProgramManager
Can someone throw inputs on the Interview process of databricks ? what sites are there for preparation ? like for Amazon they say its Loop on Leadership principles and there are sites which guide...are there any such for Databricks ?Anyone has experi...
- 79 Views
- 0 replies
- 0 kudos
- 300 Views
- 3 replies
- 2 kudos
Resolved! Best approach for migrating SAP BW to Databricks?
Hey folks,We are scoping out a migration to move our legacy SAP BW setup over to Databricks.Our main concern is the amout of logic tied up in old ADSOs, infpCubes and custom ABAP transformations routines. Manually rewriting everything into PySpark se...
- 300 Views
- 3 replies
- 2 kudos
- 2 kudos
We did a similar BW to Databricks move recently. Here's what worked for us:1) Data models: Dont copy InfoCube star schemas 1 to 1. Land raw Adso data in Bronze and flatten the business logic into Silver/Gold Delta tables.2) ABAP & metadata: Rewriting...
- 2 kudos
- 425 Views
- 2 replies
- 3 kudos
Unity AI Gateway Observability: Models, MCP Tools, and Coding Agents
Databricks Unity AI Gateway provides a unified way to monitor and understand AI activity through its built-in Workspace AI Gateway Usage Analytics dashboard and the underlying system.ai_gateway.usage system table.Some useful capabilities include:Mode...
- 425 Views
- 2 replies
- 3 kudos
- 3 kudos
Good catch on the initialize and tools/list rows. If you want to handle it in the query instead of by eye, the MCP rows carry an mcp_metadata struct with tool_name, server_type and json_rpc_method, so you can filter on the method column to keep only ...
- 3 kudos
- 1162 Views
- 1 replies
- 0 kudos
Seeking Advice on Optimizing Spark Scripts for Efficient Data Processing
Hello community!I'm currently working on Spark scripts for data processing and facing some performance challenges. Any tips or suggestions on optimizing code for better efficiency? Your expertise is highly appreciated! paybyplatema Thanks.
- 1162 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Andrew I would suggest starting with the Spark UI to identify which stages are taking the most time. Large data shuffles and uneven partitions can often slow down Spark jobs. Try filtering data early, selecting only the columns you need, and check...
- 0 kudos
- 4500 Views
- 6 replies
- 1 kudos
Best practices for optimizing Spark jobs
What are some best practices for optimizing Spark jobs in Databricks, especially when dealing large datasets? Any tips or resources would be greatly appreciated! I’m trying to analyze data on restaurant menu prices so that insights would be especiall...
- 4500 Views
- 6 replies
- 1 kudos
- 1 kudos
Answer to this overall depends on your design and data model, some good practices can be followed but you might have to determine what best fits your design. Quick tips -File Format & Sizing: Use columnar formats (Parquet/ORC) with zstd or snappy com...
- 1 kudos
- 1404 Views
- 3 replies
- 2 kudos
Resolved! eMudhra CA key leak: Databricks LTS runtimes still trust the affected root.
Background. In October 2026, eight eMudhra intermediate CA keys turned out to be recoverable from their public certificates. Browsers blocked them within days. Non-browser clients that carry their own trust store keep trusting the chain until that st...
- 1404 Views
- 3 replies
- 2 kudos
- 2 kudos
Fix 1: certifi, per job: Either put it in your wheel's dependencies.Fix 2: pip, before libraries installHelped us to overcome the Vulnerabilty!
- 2 kudos
- 546 Views
- 4 replies
- 2 kudos
Resolved! Best approach for moving Alteryx workflows to Databricks ?
Hey folks, We're planning to migrate our ETL pipelines from Alteryx to Databricks.Many of our workflows use iterative macros and complex joins, so manually rewriting everything into PySpark/Spark SQL will take a while.For those who have made this tra...
- 546 Views
- 4 replies
- 2 kudos
- 2 kudos
Hey @Abhishek20,I think @Valeria_Koz_DBX has done a great job of explaining lots of things ! Depending on your Alteryx Estate, there might be more complexity. Most Alteryx customers have moved from "legacy" Alteryx to Alteryx One. Assets have broaden...
- 2 kudos
- 121 Views
- 0 replies
- 0 kudos
Exploring Databrick UI
Starting a career in databricks?No problem let me help you with this tutorialson exploring databrick Uihttps://www.youtube.com/watch?v=4gluJWguwqE
- 121 Views
- 0 replies
- 0 kudos
- 456 Views
- 3 replies
- 0 kudos
Resolved! Best approach and tools for migrating from Oracle to Databricks?
Hey everyone,We're planning to migrate our data warehouse from Oracle to Databricks. We have a mix of standard tables and a good amount of legacy PL/SQL code to deal with.What is the best way to approach this migration? specifically:1) How do you usu...
- 456 Views
- 3 replies
- 0 kudos
- 0 kudos
Migrating an Oracle data Warehouse with heavy PL/SQL logic to Databricks is tough. Here is a practical, direct way to approach it:1) Converting PL/SQL: Standard SQL queries migrate easily to Databricks SQL, but procedural code doesn't translate nativ...
- 0 kudos
- 243 Views
- 0 replies
- 0 kudos
Databricks Customer Academy Log In Issues
I purchased 1 year subscription on Aug 7, 2026 - Databricks Academy Labs Courses.Everytime, after OTP, it gives wrong credentials error.Earlier, I was able to login but not for last 4-5 weeks, I am not able to login for last 4-5 weeks.I have raised t...
- 243 Views
- 0 replies
- 0 kudos
