- 272 Views
- 4 replies
- 0 kudos
Databricks SDP (Spark declarative Pipelins) Overwrite table
1) Consider I have orders folder which orders_1.csv file and it is loaded to orders tables orders/ ============> load to order table orders_1.csv 2) next day new file (orders_2.csv) arrived orders/ =======...
- 272 Views
- 4 replies
- 0 kudos
- 0 kudos
Streaming tables are append-only by design, and a materialized view recomputes over everything in the folder. Neither gives you "replace the table with just the newest file" out of the box, so you need a small pattern on top. Here are the two I'd co...
- 0 kudos
- 307 Views
- 7 replies
- 0 kudos
What makes a Databricks data platform truly AI-ready?
As more teams start connecting AI agents and AI/BI workloads to Databricks, I’ve been thinking about what actually makes an enterprise data platform “AI-ready.”From my experience working in enterprise data and analytics at Polestar Analytics, the cha...
- 307 Views
- 7 replies
- 0 kudos
- 0 kudos
I see AI-readiness as more than making data accessible to an AI/agent. The foundation is trusted data + governed access + business context.In Databricks, I would build this around Unity Catalog for governance, lineage and discoverability, strong data...
- 0 kudos
- 327 Views
- 7 replies
- 4 kudos
Guidance Required: Scheduling a Biweekly Databricks Job
I’m reaching out to seek your guidance on one of our use cases where we need to schedule a biweekly Databricks job to run every alternate Wednesday.I explored both the standard Databricks scheduling options and cron-based scheduling, but I was unable...
- 327 Views
- 7 replies
- 4 kudos
- 4 kudos
Greetings @pawanswami, I did some digging and here is what I found. Satyasai and ivy125cordes have the root cause right: Databricks schedules run on Quartz cron, and Quartz has no "every N weeks" concept. The question that decides your fix is whether...
- 4 kudos
- 305 Views
- 3 replies
- 1 kudos
Question about MySQL Integrated‑CDC pipeline on Classic compute workspace
Hi everyone,I am trying to set up an Integrated‑CDC pipeline for a MySQL RDS instance on our Databricks workspace. Our workspace only uses Classic compute; Serverless compute is not available.According to the documentation, Integrated‑CDC for MySQL s...
- 305 Views
- 3 replies
- 1 kudos
- 1 kudos
Yes, workspace-level preview access or specific feature flags may be required for MySQL Integrated-CDC depending on your Databricks account tier, and while API-only deployments via the Databricks SDK are fully supported for production, you must ensur...
- 1 kudos
- 4517 Views
- 11 replies
- 0 kudos
Resolved! Bundled Wheel Task with Serverless Compute
I am trying to run a wheel task as part of a bundle on serverless compute. My databricks.yml includes an artifact being constructed:artifacts: nexusbricks: type: whl build: python -m build path: .I then am trying to set up a job to cons...
- 4517 Views
- 11 replies
- 0 kudos
- 16532 Views
- 17 replies
- 7 kudos
Resolved! PDF Attachment on an Alert
Currently my Alert is an HTML table using data pointing to an SQL query.I was wondering if it is possible to attach the resulting table from this SQL query as a PDF to the alert email.If anyone has successfully implemented this, please let me know! T...
- 16532 Views
- 17 replies
- 7 kudos
- 7 kudos
Ok understood the concern, so basically the issue is with PDF rendering as much I understood. Let me know if I am wrong. Let me see if there is any improvement by our engineering team on this front.
- 7 kudos
- 295 Views
- 3 replies
- 2 kudos
Genie Use Cases & easy adaptibility
Recently I started socializing the use of Genie and Databricks to even a laymen (and ofcourse laywomen) who have never heard what coding is in their life. I am proud that in my company nearly 200 people as of now are using Databricks for some or othe...
- 295 Views
- 3 replies
- 2 kudos
- 2 kudos
its really good to see that genie is being used beyond data engineering teams. One thing I think becomes particularly important as adoption grows is separating experimentation from production workloads and the genie makes it much easier for non-techn...
- 2 kudos
- 190 Views
- 2 replies
- 0 kudos
I built an open-source check that stops AI agents from inventing column names in SQL.
I work on data pipelines. The most common way I see Claude Code and Cursor break SQL is dull. They write against the schema they think exists. The README is stale, or they copy `customer_id` from an old query when the real column is `customerid`. The...
- 190 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks for sharing this. In high level, it is a tool for the agent which will validate the schema and take an action. One question how do we maintain the freshness of the schema, for an example if there is any change in the column name or data type, ...
- 0 kudos
- 292 Views
- 2 replies
- 1 kudos
Resolved! Lakeflow connect
Spoiler What is extractor_sql-server-conn_cdc_sink in the Ingestion gateway pipeline in lakeflow connect and is it crreated by deafualta and how does it work?
- 292 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks for the answer..it's now clear..I also followed the doc and understood!
- 1 kudos
- 25196 Views
- 6 replies
- 7 kudos
Resolved! Window function using last/last_value with PARTITION BY/ORDER BY has unexpected results
Hi, I'm wondering if this is the expected behavior when using last or last_value in a window function? I've written a query like this:select col1, col2, last_value(col2) over (partition by col1 order by col2) as column2_last from values ...
- 25196 Views
- 6 replies
- 7 kudos
- 7 kudos
The behavior of LAST_VALUE with PARTITION BY and ORDER BY can be confusing when the window frame affects which row is considered the last value. I’ve found that checking the window frame definition carefully helps make these results easier to underst...
- 7 kudos
- 248 Views
- 2 replies
- 0 kudos
Resolved! Analyze
How many types of analyze do we have for a UC delta table ? And how to check which type of ANALY
- 248 Views
- 2 replies
- 0 kudos
- 0 kudos
How many types of analyze do we have for a UC delta table ? And how to check which type of ANALYZE ran for what table? i think total questions was not printed correctly
- 0 kudos
- 3547 Views
- 5 replies
- 2 kudos
Python logger.info() not showing inside applyInPandas (but print() works) — why?
Problem: In Databricks, logs from an external binary (via os.system) show up, but Python logger.info() inside groupBy(...).applyInPandas(...) does not. print(..., flush=True) does show up.Why: applyInPandas runs your function as a pandas UDF. That c...
- 3547 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi Ben, The stdout logging-handler trick works on a classic/standard cluster, but on serverless it will not surface in the notebook cell, and that is expected rather than a problem with the helper. Here is what is going on. applyInPandas runs your Py...
- 2 kudos
- 24355 Views
- 18 replies
- 17 kudos
How to fix plots and image color rendering on Notebooks?
I am currently running dark mode for my Databricks Notebooks, and am using the "new UI" released a few days ago (May 2023) and the "New notebook editor."Currently all plots (like matplotlib) are showing wrong colors. For example, denoting:```... p...
- 24355 Views
- 18 replies
- 17 kudos
- 17 kudos
HTML outputs in dark mode was recently improved. Please see https://docs.databricks.com/aws/en/notebooks/notebook-ui#html-outputs-in-dark-mode and try it out!
- 17 kudos
- 4080 Views
- 8 replies
- 12 kudos
Future of Movie Discovery: How I Built an AI Movie Recommendation Agent on Databricks Free Edition
As a data engineer deeply passionate about how data and AI can come together to create real-world impact, I’m excited to share my project for the Databricks Free Edition Hackathon 2025 — Future of Movie Discovery (FMD). Built entirely on Databricks F...
- 4080 Views
- 8 replies
- 12 kudos
- 12 kudos
@Brahmareddy I haven't used this exact pattern at multi-million-row scale, but I agree that the tradeoffs start to change as the dataset and query volume grow.I would probably keep Delta as the source of truth and separate the serving layer based on ...
- 12 kudos
- 513 Views
- 5 replies
- 4 kudos
AUTO CDC SCD Type 2 with late-arriving deletes: edge cases the docs don't cover
Hi everyone,I'm designing a Lakeflow Declarative Pipeline that processes customer profile changes from a CDC feed into an SCD Type 2 Silver table. Events can arrive out of order, sometimes up to 24 hours late. The planned flow:CREATE OR REFRESH STREA...
- 513 Views
- 5 replies
- 4 kudos
- 4 kudos
Hello @LiresaFerizaj , I took a look at both internal and external documentation and here is what I found. You've already had solid answers from Thomaz, Nitesh, data_pulse, and Islam, so I'll tie the thread together, separate what's documented from w...
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
