Is there an update to the databricks extension for VS code that will allow me to work with "genie agents" (aka spaces)?I'm having a really hard time with the editing of these agents in the Azure workspace environment. There are lots of things missin...
Hi everyone,We're evaluating an architecture pattern and would appreciate any guidance or recommendations. Has anyone implemented a similar cross-cloud pattern? Specifically:Can AWS Glue be used to access Azure Databricks data without copying it?Are ...
Thank you so much for all your help and guidance. I truly appreciate your responses and insights. I’ll definitely keep this in mind and follow up with the source team first.
Is see that kafka for zerobus is still in beta. So maybe something will change. Can someone tell me whether they are aware of an authentication handshake with this API that might AVOID the callback?Here is the python example, which uses a sophistica...
Hello @DB1To3, I took a look at both internal and external documentation and here is what I found. Straight answer: there's no non-callback auth mode and no unauthenticated mode for the Zerobus Kafka API today. @DoTA has that right. The endpoint only...
Hi Team,We have noticed a change in the behavior of our Lakeflow pipelines.Previously, when we deleted a DLT pipeline, the associated resources, such as materialized views and streaming tables, were also deleted along with the pipeline.However, recen...
Yes, this is expected behavior for Unity Catalog pipelines using the default publishing mode. Deleting the pipeline now retains the materialized views, streaming tables, and views by default, and you can opt back into cascade deletion through the AP...
1) Consider I have orders folder which orders_1.csv file and it is loaded to orders tables orders/ ============> load to order table orders_1.csv 2) next day new file (orders_2.csv) arrived orders/ =======...
Streaming tables are append-only by design, and a materialized view recomputes over everything in the folder. Neither gives you "replace the table with just the newest file" out of the box, so you need a small pattern on top. Here are the two I'd co...
As more teams start connecting AI agents and AI/BI workloads to Databricks, I’ve been thinking about what actually makes an enterprise data platform “AI-ready.”From my experience working in enterprise data and analytics at Polestar Analytics, the cha...
I see AI-readiness as more than making data accessible to an AI/agent. The foundation is trusted data + governed access + business context.In Databricks, I would build this around Unity Catalog for governance, lineage and discoverability, strong data...
I’m reaching out to seek your guidance on one of our use cases where we need to schedule a biweekly Databricks job to run every alternate Wednesday.I explored both the standard Databricks scheduling options and cron-based scheduling, but I was unable...
Greetings @pawanswami, I did some digging and here is what I found. Satyasai and ivy125cordes have the root cause right: Databricks schedules run on Quartz cron, and Quartz has no "every N weeks" concept. The question that decides your fix is whether...
I am currently designing a Disaster Recovery (DR) strategy for our Databricks workspace to recover from system failures and human errors. Our main requirement is to restore both the data and the pipelines reliably if a workspace goes down or gets cor...
Short answer: don't try to back up or migrate ST/MV state and checkpoints at all. On Serverless you can't reach them, and they aren't designed to be portable. Design for rebuild and replay instead: pipeline definitions in Git, replayable sources rep...
Hi everyone,I am trying to set up an Integrated‑CDC pipeline for a MySQL RDS instance on our Databricks workspace. Our workspace only uses Classic compute; Serverless compute is not available.According to the documentation, Integrated‑CDC for MySQL s...
Yes, workspace-level preview access or specific feature flags may be required for MySQL Integrated-CDC depending on your Databricks account tier, and while API-only deployments via the Databricks SDK are fully supported for production, you must ensur...
I am trying to run a wheel task as part of a bundle on serverless compute. My databricks.yml includes an artifact being constructed:artifacts: nexusbricks: type: whl build: python -m build path: .I then am trying to set up a job to cons...
Currently my Alert is an HTML table using data pointing to an SQL query.I was wondering if it is possible to attach the resulting table from this SQL query as a PDF to the alert email.If anyone has successfully implemented this, please let me know! T...
Ok understood the concern, so basically the issue is with PDF rendering as much I understood. Let me know if I am wrong. Let me see if there is any improvement by our engineering team on this front.
Hi Everyone, One of our big ingestion tasks is lakeflow connect CDC ingestion of ERP data from a SQL server database. I am deploying the pipelines and jobs for this via DABs. We are ingesting about 80 tables from the database, a handful of which are ...
Hello @dbernstein_tp , I took a look at the comments here and did more digging based on what I've read. Good questions from @lmcorreahdb, and the 21/60 made me think. Your current design already fits your SLAs. The gateway runs continuously on small ...
Recently I started socializing the use of Genie and Databricks to even a laymen (and ofcourse laywomen) who have never heard what coding is in their life. I am proud that in my company nearly 200 people as of now are using Databricks for some or othe...
its really good to see that genie is being used beyond data engineering teams. One thing I think becomes particularly important as adoption grows is separating experimentation from production workloads and the genie makes it much easier for non-techn...
I'm having problems with using jobs together with GIT.I have created a job that hasType - NotebookSource - Git provider (main),Path "jobs/base/INIT_PROCESSING"I can access the jupyter notebook without any problems. Each push to the repo can be read i...
I believe this could be related to the sparse_checkout configuration. If the notebook is importing modules from directories outside the `jobs/base` path, those files may not be available in the job’s Git checkout.As a first step, I would suggest eith...
I work on data pipelines. The most common way I see Claude Code and Cursor break SQL is dull. They write against the schema they think exists. The README is stale, or they copy `customer_id` from an old query when the real column is `customerid`. The...
Thanks for sharing this. In high level, it is a tool for the agent which will validate the schema and take an action. One question how do we maintain the freshness of the schema, for an example if there is any change in the column name or data type, ...
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
