- 129 Views
- 1 replies
- 1 kudos
Serverless Scala JAR: Scala UDFs that read `Row` input fail
### Setup- Serverless jar task, environment version 4 (Databricks Connect 17.3.2, Scala 2.13.16, JDK 17). Also reproduced on version 5 (Databricks Connect 18.0.0).- JAR built with databricks-connect_2.13 as provided- Structured Streaming from a Unity...
- 129 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @ZZX , First, thank you for the quality of this write-up. The pass/fail matrix, the captured cause chain, and the signature comparison across Apache Spark 4.0.0 and the published Databricks Connect clients make this one of the cleaner bug report...
- 1 kudos
- 2212 Views
- 3 replies
- 1 kudos
how does Job cluster auto scaling work
Can you share the metrics databricks uses during job cluster auto scaling?Is Databricks looking at queued tasks, slot utilization etc or just looking at CPU utilizations?The autoscaling docuemnt https://docs.databricks.com/aws/en/compute/configure?u...
- 2212 Views
- 3 replies
- 1 kudos
- 1 kudos
"cannot scale down to zero workers." That is wrong. It can scale down to zero.
- 1 kudos
- 181 Views
- 3 replies
- 0 kudos
Streaming Considerations for Daily Batch Ingestion
Hey everyone, First time posting looking to complete my research for a Databricks implementation. My current data pipeline consists of Azure Data Factory Pipelines that load data from ADLS into a SQL database. This happens once a day on schedule sinc...
- 181 Views
- 3 replies
- 0 kudos
- 0 kudos
@nye-d You don’t necessarily need a full paradigm shift, Databricks works best when you separate procedural orchestration from incremental data processing. Here are some insights on the doubts raised.1) Auto Loader SolutionThe biggest value is removi...
- 0 kudos
- 244 Views
- 1 replies
- 6 kudos
Building a Delivery Assurance Agent: Predicting $8.4M in Penalty Risk with Databricks Genie
Delivery Assurance Agent: AI-Powered Risk Intelligence for Delivery TeamsThe ProblemDelivery teams track thousands of tasks but struggle to answer:Which client commitments will miss, and what should we do?Traditional project tracking gives you task-l...
- 244 Views
- 1 replies
- 6 kudos
- 6 kudos
Excellent article
- 6 kudos
- 20705 Views
- 22 replies
- 4 kudos
spotify API get token - raw_input was called, but this frontend does not support input requests.
hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...
- 20705 Views
- 22 replies
- 4 kudos
- 4 kudos
Modern digital applications focus on delivering convenient, accessible, and user-friendly experiences. Cloudstream Apk is one example of an application users may explore while learning about evolving digital solutions. Such platforms reflect the grow...
- 4 kudos
- 340 Views
- 5 replies
- 2 kudos
Resolved! What is the difference between a managed table and an external table in Databricks?
Hi everyone,I’m trying to understand the difference between managed tables and external tables in Databricks.I understand that both can store data in Delta format, but I’m not clear about how their storage and lifecycle are different.When should we u...
- 340 Views
- 5 replies
- 2 kudos
- 2 kudos
hi @gowri_databrick A managed table is the default for new Databricks workloads, but most of the real-world projects are built using external tables since data comes from various other sources. So try practising each sentence below to make sense Use ...
- 2 kudos
- 160 Views
- 3 replies
- 0 kudos
Lakebase Postgre updating Delta Table.
I am using Postgre for OLTP processing for POS application.Lag is reduced a lot, however when there is updation on Postgre table, I need to sync back to delta table. There is one way from delta table sync table (read only). Any design pattern sas to ...
- 160 Views
- 3 replies
- 0 kudos
- 0 kudos
As @balajij8 and @srini_ve @mentioned, Lakebase CDF is the way to do it. You’ll have the change data flowing to lake house as SCD2 so you have a record of all the transactions happening in lakebase stored in your lake house. I have a LinkedIn post wr...
- 0 kudos
- 486 Views
- 3 replies
- 2 kudos
Serverless Scala JAR: foreachBatch fails with RST_STREAM PROTOCOL_ERROR
Hi everyone,We're migrating a Scala Structured Streaming application (Kinesis → Delta) from classic compute to Databricks Serverless Jobs for cost optimization. During the migration, we're consistently seeing what appears to be a Spark Connect / gRPC...
- 486 Views
- 3 replies
- 2 kudos
- 2 kudos
On (1): the docs don't gate Scala foreachBatch on serverless. Limitations with Databricks Connect for Scala lists streaming foreachBatch as unavailable only on Databricks Runtime 13.3 LTS and below, and serverless limitations names only Trigger.Proce...
- 2 kudos
- 452 Views
- 6 replies
- 0 kudos
Resolved! Streaming tables fail with DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_SCHEMA_LOG after adding a
Environment:Databricks Runtime: Current channel, Photon enabledEdition: ProPlatform: AzureDescription:My setup consists of:A managed ingestion pipeline for a SQL Server database that ingests raw data into bronze Delta tables (SCD Type 1), consisting ...
- 452 Views
- 6 replies
- 0 kudos
- 0 kudos
Thanks everyone for your responses.@ShamenParis TBLPROPERTIES ("pipelines.reset.allowed" = "true") seems to be set by default.@Satyasai a second full refresh would fix the issue but we want to avoid doing that especially in production
- 0 kudos
- 427 Views
- 3 replies
- 0 kudos
Lakeflow connect
When can we expect lakeflow connect for (mysql,oracle,postgres) will be in GA from preview mode
- 427 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey there @gowri_databrick The query-based Lakeflow Connect connectors for MySQL, Oracle, and PostgreSQL are generally available: Release Notes The CDC versions are presently at different preview stages. MySQL and PostgreSQL are in Public Preview, ...
- 0 kudos
- 13945 Views
- 14 replies
- 10 kudos
Resolved! What are powerfull data quality tools/libraries to build data quality framework in Databricks ?
Dear Community Experts,I need your expert advice and suggestions on development of data quality framework. What are powerfull data quality tools or libraries are good to go for development of data quality framework in Databricks ? Please guide team.R...
- 13945 Views
- 14 replies
- 10 kudos
- 10 kudos
@shubham_007 Can re-iterate the same as many mentioned as there are few good options to build DQ in Databricks but the right choice depends on whether you want something native, library-based or more platform driven frameworks.Lakeflow Declarative Pi...
- 10 kudos
- 187 Views
- 2 replies
- 0 kudos
Resolved! Thoughts on Using Remix for Data-Focused Applications
I’ve been exploring different approaches for building web applications that work with data-heavy workflows, and I recently came across Remix as an interesting option.What I like about Remix is its focus on server-side data loading, forms, and handlin...
- 187 Views
- 2 replies
- 0 kudos
- 0 kudos
@ThiamLee I’ve actually worked on a similar use case in one of my projects, where we used:React → FastAPI → Databricks SQL → Unity Catalog (Delta tables)From my experience, this worked quite well for a data-heavy application. React handled the UI, Fa...
- 0 kudos
- 694 Views
- 3 replies
- 2 kudos
internship
I am a data science aspiring Student i am very much interested in databricks and i am looking for internships. if anyone knows how to apply please help
- 694 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi! I’m also very interested in internship opportunities at Databricks. I’m currently building my skills in Python, SQL, Data Analytics, ETL, and Data Engineering, and I’m actively working on related projects. If anyone has information about internsh...
- 2 kudos
- 420 Views
- 6 replies
- 4 kudos
create_auto_cdc_from_snapshot_flow Python session resolution fails if having multiple snapshot flows
When a pipeline contains more than one create_auto_cdc_from_snapshot_flow flow (each driven by a custom Python next_snapshot_and_version function), flow resolution fails intermittently/consistently with: RuntimeError: The original Spark session is be...
- 420 Views
- 6 replies
- 4 kudos
- 249 Views
- 4 replies
- 0 kudos
Best practices for data quality in lakeflow
Hi everyone,What are the recommended best practices for implementing data quality checks in Lake flow Spark Declarative Pipelines?Should data quality expectations be applied mainly in the Bronze layer, Silver layer, or both?Thanks!
- 249 Views
- 4 replies
- 0 kudos
- 0 kudos
@gowri_databrick ,In my experience, the best approach to data quality in DBR Lakeflow is to treat it as a continuous process rather than a one-time validation step.A practical pattern is:Bronze – Observe: Keep the raw data as close to the source as p...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
