All board posts in Databricks Community

Re: Lakebase Data API – First Request Times Out After 8 Seconds When Compute Is Offline

KeatOoi — Fri, 04 Sep 2026 00:30:16 GMT

Below are the server-timing headers metrics that i collected:

Metric	Successful request	Failed request
JWT	0.2 ms	25.9 ms
Parse	0.0 ms	0.1 ms
Plan	0.0 ms	0.1 ms
Transaction	3,844.8 ms	8,004.6 ms
Response	0.3 ms	0.0 ms
Result	SUCCESS	57014 TIMEOUT
Authenticator timeout	8 seconds	8 seconds

So far, what I can confirm is that if a query executed via the Lakebase Data API takes more than 8 seconds, the query will be cancelled.

Therefore, turning off scale-to-zero is not going to solve the issue if the query itself takes longer than 8 seconds.

{

"code": "57014",

"message": "canceling statement due to statement timeout",

"details": null,

"hint": null

}

🌟 Community Pulse: Your Weekly Roundup! August 24 – 30, 2026

Advika — Thu, 03 Sep 2026 22:20:37 GMT

The Weekly Digest • August 24 – 30

Community Pulse

A new week, a fresh wave of articles, answers, and discussions. Here's your weekly catch-up. 🗞💬

🏅 This Week's Community Stars

The members who showed up and made a difference this week 👏:

@balajij8

@pradeep_singh

+ Top Brickster contributor:

@AbhilashNagilla

Note: Names are listed in no particular order.

📖 Knowledge Hub

✍️ Community & Lakebase Articles

📃 MVP Articles

📌 Technical Blogs

Technical Blog

Building an Autonomous Supply Chain Engine on Databricks Read article →

Lakebase Blog

Moving Beyond Manual Changes: A Guide to Shipping Lakebase Schema with Bundles Read article →

Active Discussions

Threads that kept the community talking this week 👀

🔥

Distinguishing Runs Related to an All-Purpose Cluster

↗

🔥

INVALID_PARAMETER_VALUE.LOCATION_OVERLAP after catalog deleted

↗

🔥

Declarative Automation Bundle - How to handle Managed resources?

↗

🔥

databricks SQL UDF in select statement

↗

These are just the highlights — head to the community to see everything that's buzzing.

Genie-Powered App Showcase

From idea to app – see the Genie-powered projects our community built during this period ✨ :

Genie App Early Warning Databricks app: giving a school principal Monday morning back Explore →	Genie App Money Performance and customer Insights Explore →	Genie App Resilient-Sync: AI-Powered Supply Chain Risk & Performance Intelligence Explore →
Genie App Genie Business Use case : Promo Genie Explore →	Genie App Le Greffier - A Genie Agent Powered Game Explore →	Genie App Building TileGenie : a Databricks App Where Genie Is the Entire Backend Explore →
Genie App Prove It: the query is the lesson — a Genie-powered data-literacy game Explore →	Genie App Building an End-to-End Store Order Forecasting System on Databricks: From Zero to Automated Pipeline Explore →	Genie App The Exfiltration: a detective game where the only witness is a Genie agent Explore →
Genie App Beyond FIFO with Databricks Genie :Resolving the UK Pensions Dashboards Missing Valuation Bottleneck Explore →	Genie App Genie-Powered App Challege: Bushfire Exposure Across Victoria's Powerline Network Explore →	Genie App Gridiron Genie: an NFL matchup room powered by Databricks Genie Agent Explore →
Genie App Building a Databricks Solutions Architect Genie Agent Explore →	Genie App Building a Delivery Assurance Agent: Predicting $8.4M in Penalty Risk with Databricks Genie Explore →

Questions That Found Their Answer

A few of the threads that wrapped up with a solution this period:

✔

How to show full query results in a SQL alert's notification?

↗

✔

databricks SQL UDF in select statement

↗

✔

Title: Oracle CDC pipeline (Lakeflow Connect) never terminates

↗

✔

Best practice for dbutils with CICD

↗

These are just the highlights – head to the community to see everything that's buzzing.

That's a Wrap – For Now 🙏

Thank you for being part of this community – every post, reply, and article you share makes it better.

See you next week! 👋

Re: Lakaflow connect Data Ingestion gatway pipeline fail often and restarts automatically

ivanvyd — Thu, 03 Sep 2026 20:46:14 GMT

@srikanthp24 the 15-minute delay is consistent with Lakeflow Connect’s automatic exponential-backoff retry. A successful ingestion pipeline run does not confirm that the gateway is healthy; it may only be processing data already written to staging.

Open the failed gateway update under Jobs & Pipelines. Use View event log for the structured error, then open Update details -> Logs and inspect the driver `stdout`. Search for `SOURCE_CONNECTION_FAILURE` and follow the `Caused by` chain to the first specific JDBC, DNS, TCP, TLS, or authentication error.

SELECT timestamp, level, event_type, message, error, details FROM event_log('<gateway-pipeline-id>') WHERE level IN ('ERROR', 'WARN') ORDER BY timestamp DESC LIMIT 100;

I would let the gateway recover automatically while checking whether the failures coincide with SQL Server maintenance, firewall changes, DNS issues, or network timeouts.

Please, let us know what you found.

Lakaflow connect Data Ingestion gatway pipeline fail often and restarts automatically

srikanthp24 — Thu, 03 Sep 2026 19:50:13 GMT

Hello Team,

I have designed Lakeflow connect data ingestion pipeline using managed connector SQL Server. The source is SQL Server created the connection and Change Tracking enabled at the source level. Also, I have give required permission and everything. I am able to create the pipeline successfully which consists one gateway pipeline to capture the changes (Source > Staging layer) and ingestion pipeline(Staging > destination tables). It is running perfectly fine capture the continous changes and update the target table using ingestion pipeline. But My Gateway pipeline fails after running for some hours and it will restart automatically but not immediately, It will continouslu try up and run during that also it will fail multiple time. Some times the error says hostname and port is wrong.

[DLT ERROR CODE: INGESTION_GATEWAY_CONNECTION_ERROR.SOURCE_CONNECTION_FAILURE]

[DLT ERROR CODE: INGESTION_GATEWAY_CONNECTION_ERROR.SOURCE_CONNECTION_FAILURE]
After some time it will up. I am unable to understand why it is happening. How we can fix this issue. We don't have any notification option also keep track of the failure for gateway pipeline. Even though gateway pipeline fails. My Ingestion pipeline will run successfully that is running interval of every 30 minutes.

Re: Issue with auto complete in the datarbicks notebook

ivanvyd — Thu, 03 Sep 2026 18:45:30 GMT

@leahlithe green and red changes with Accept and Reject are consistent with Genie Code’s inline edit or Quick Fix interface. Standard autocomplete produces the grey suggestions you saw before and accepts them with Tab or Enter.

Open your profile menu and go to Settings -> Developer. Keep Autocomplete as you type enabled if you still want the grey suggestions. If your workspace shows a setting for automatic inline Quick Fix, disable that setting.

Databricks treats autocomplete, inline suggestions, and automatic Quick Fix as separate prefeerences. Disabling standard autocomplete may therefore have no effect on the red and green proposals.

P.s. a rejected suggestion should close, and honeslty I could not find a documented feature that recreates deleted cells. Test the behavior in a new SQL notebook. If it continues, ask your workspace administrator whether a Genie code or editor preference changed. Since your colleagues began seeing it during the same week, this may be a shared workspace change or smth.

Issue with auto complete in the datarbicks notebook

leahli — Thu, 03 Sep 2026 17:28:32 GMT

I'm using sql to run a lot of queries in the notebook. Since this week it started to pop up with the Suggested code every time I made an adjustment to the code and my cursor keeps changing between the green and red highlighted lines (previously it only suggested code in the grey font size, and it wouldn't interrupt my coding process. Now it just became very annoying, even when I deleted the cell, it will come back immediately and highlighted in red. I cannot make adjustments at once I have to click accept or rejected multiple times. The same thing happens to my coworkers.

Re: Migrating Talend ETL Jobs to Databricks – Best Practices & Challenges

sbbagal13 — Thu, 03 Sep 2026 17:06:01 GMT

this page is no longer available
https://blog.aidetic.in/replaced-multiple-etl-tools-like-talend-with-databricks-5841e87bdaea
Do you have any other resources

Re: Improvement in DAB for alert_task

emma_s — Thu, 03 Sep 2026 16:50:42 GMT

Hi,

This is a known limitation today. The official documentation for the SQL alert task confirms it under the Limitations section:

"SQL alert tasks do not support parameters. If you need to use parameterised queries, consider using a SQL task instead."

So the behaviour you are seeing is expected, not a misconfiguration on your side. The alert_task schema in DABs only supports alert_id, subscribers, warehouse_id, and workspace_path. There is no parameters field available yet.

The docs suggest using a sql_task as a workaround, but that is not really equivalent. A sql_task runs a query but does not evaluate alert conditions or send subscriber notifications with custom templates (custom_summary, custom_description, {{QUERY_RESULT_VALUE}}). So it would not give you the same behaviour as your alert.

Realistic options for now:

Create separate alerts per task name rather than one generic parameterised alert. Not ideal if you have many task names, but it works today with DABs.
Use the UI to create the job with alert task parameter overrides (as you have already found this works), and manage that specific job outside of DABs for now.
Submit product feedback directly from your workspace. Click your username icon in the top navigation bar, then click "Send feedback". Describing this gap between the UI and DABs for alert task parameters will help the product team prioritise it. There is a good chance this is already being built but all feedback helps!

Hope that helps!

resources:
  jobs:
    test_alert:
      name: test notif
      tasks:
        - task_key: alert_gen_ko
          sql_task:
            warehouse_id: ${var.warehouse_id}
            query:
              query_id: <your_query_id>
            parameters:
              task: OVERRIDE TASK NAME
      queue:
        enabled: true

You would need to handle the notification/alerting logic separately (e.g. via job-level notifications or a downstream condition task).

Re: Partitioning vs Liquid Clustering (per-table):

emma_s — Thu, 03 Sep 2026 16:23:56 GMT

Hi, as others have said they can't coexist. The docs are clear: "Clustering is not compatible with partitioning or ZORDER." You pick one or the other.

If you have existing partitioned tables, you can convert them (DBR 18.1+):

ALTER TABLE my_table REPLACE PARTITIONED BY WITH CLUSTER BY (col1, col2);

Note: this conversion command doesn't work for streaming tables or materialised views in Lakeflow pipelines. For those, you update the pipeline definition directly. See Convert a partitioned table to liquid clustering.

Does PARTITION BY hurt materialised view refreshes?

Not directly. Incremental refresh cares more about whether deletion vectors, row tracking, and change data feed are enabled on source tables. See Incremental refresh for materialized views. But partitioning is rigid, and over-partitioning creates lots of small files which adds metadata overhead.

Is Liquid Clustering redundant if MVs read incrementally?

No. Even with incremental refresh, Liquid Clustering still helps because:

File compaction: OPTIMIZE keeps files right-sized, reducing metadata overhead for all readers including MV refreshes. I've seen in customer scenarios especially where a lot of files are being ingestion without this your pipeline updates over time can grind to a halt.
Data skipping: when a full refresh does happen (fallback), clustered data skips far more effectively
Flexibility: you can change clustering keys without rewriting data, unlike partitions
MVs themselves can be clustered: Databricks recommends Liquid Clustering "for all new tables, including streaming tables and materialized views." See Use liquid clustering for tables.

References

Databricks Implementation / BI Modernization Opportunities

nafikazi — Thu, 03 Sep 2026 16:07:11 GMT

Hello everyone,

I’m a Toronto-based BI & Analytics professional with 12 years of experience across enterprise analytics and financial services. I recently completed the Databricks Certified Data Engineer Associate certification.

My experience includes Power BI, Databricks, Alteryx, BI modernization, data governance, analytics solution design, business requirements, and vendor/stakeholder management.

I’m looking to join a Databricks implementation or consulting team where I can help bridge business requirements and technical delivery, particularly on BI modernization and analytics transformation projects.

If your organization or project needs someone with this combination of BI, business, and Databricks knowledge, I’d appreciate a connection or referral.

Based in Toronto | Open to consulting/implementation opportunities
Thanks
Nafi

How Can I Become a Databricks Partner Champion Through Technical Expertise?

ChiranjeeviKudu — Thu, 03 Sep 2026 16:06:28 GMT

Hello Databricks Community,

I am actively building my technical expertise in the Databricks ecosystem and would like guidance on the roadmap to becoming a Databricks Partner Champion, with a strong focus on technical skills and hands-on implementation.

I understand that Databricks has different recognition programs, including Community Champions. However, my question is specifically about becoming a technically recognized Databricks Partner Champion through certifications, partner learning, solution architecture, and real-world implementation experience.

I would appreciate guidance on the following:

What are the current eligibility requirements for the Databricks Partner Champion Program?
Which Databricks Partner Academy courses or learning paths should I complete?
Are there specific instructor-led courses or badges, such as Solution Architect Essentials, that are required?
Which paid Databricks certifications are recommended or required for the Partner Champion journey?
Should I focus on Data Engineering, Data Analytics, Machine Learning, Generative AI, or Solution Architecture certifications?
How much hands-on customer or implementation experience is expected?
Is nomination by my organization or a Databricks Partner Solutions Architect required before joining the program?
After completing the required courses and certifications, what is the official process for applying or being nominated?
Is there an official roadmap or checklist that explains the complete journey from Databricks Partner learning to Partner Champion recognition?

My goal is not specifically to become a monthly Community Champion. I am primarily interested in developing strong technical Databricks expertise, gaining relevant certifications, working on real-world implementations, and following the correct path toward the Databricks Partner Champion Program.

If anyone from the Databricks team or experienced Partner Champions can share the latest roadmap, prerequisites, recommended certifications, and nomination process, I would greatly appreciate your guidance.

Thank you in advance for your support! Looking forward to learning from the Databricks Community.

Re: Are you part of a Databricks user group in your city/region?

Rishabh_Tiwari — Thu, 03 Sep 2026 15:58:36 GMT

Hi @guptem ,

Thank you for reaching out. Please drop an email to usergroups@databricks.com for queries related to regional or interest Usergroup.

Thanks
Rishabh

CUSTOMER STORY | PAR Technology builds modern AI-powered intelligence on Databricks Genie

Tushar_Parekar — Thu, 03 Sep 2026 15:43:32 GMT

"PAR Intelligence is where PAR Retail’s deep industry data and AI strategy come together. Databricks gives us a modern foundation to build faster, govern smarter and deliver insights customers can act on." - Amine Errazi, Product Manager, AI, PAR Technology

PAR Technology is using the Databricks Data + AI Platform to power PAR Intelligence, a conversational analytics experience for convenience and fuel retailers. By bringing data, AI development, governance and deployment into one foundation, PAR is helping customers move from analyst-dependent questions to actionable insights in minutes.

Key highlights:

Weeks to minutes: Questions that once depended on analysts and could take up to two weeks can now be explored through Genie Agents in minutes.
8 Databricks capabilities unified: PAR replaced a fragmented tool stack with one platform for data preparation, model evaluation, agent development, serving, monitoring and orchestration.
100% natural-language queries: Retailers can ask questions about loyalty and performance data in plain English, without writing SQL.
Built for retail context: PAR grounds its AI experiences in specialized convenience and fuel retail data so insights are relevant to the operators using them.
Governance across the lifecycle: Unity Catalog, Managed MLflow and Model Serving help PAR manage access, evaluation, deployment and ongoing improvement in one environment.

🔗 Check out the full story 👈

Re: Possible Incorrect Answer Marking in Databricks Academy Assessment - Agent Evaluation on Databri

Dhivakar_sathya — Thu, 03 Sep 2026 15:36:02 GMT

i got the following message from Databricks Support Team

Bajar costos de lectura en databricks

elopez95 — Thu, 03 Sep 2026 15:30:58 GMT

Hola colegas me gustaria abrir un pequeño debate de como bajar los costos de usar los sql warehouse en databricks que al final del dia son los que mas nos suman en la factura, cuales serian las buenas practicas para ir bajando estos costos.

Improvement in DAB for alert_task

TPI_PARIS — Thu, 03 Sep 2026 15:25:06 GMT

Hello,

I'm facing with a difference between DAB and the UI concerning alert_task on job ressource.

I have an alert with a SQL query using some parameters, installed by DAB. When I want to use it un job ressource in DAB we can't use "parameters" in alert_task so we can't override these parameters. But when I create a job ressource with the UI using the same alert I can override parameters.

Any help or improvement in DAB ?

Exemple :

simple alert yaml

resources:
  alerts:
    generic_alert_notification:
      display_name: generic_alert_notification
      warehouse_id: ${var.warehouse_id}
      query_text: SELECT :task as TACHE
      parameters:
        - name: task
        value : DEFAULT TASK
      evaluation:
        comparison_operator: IS_NOT_NULL
        source:
          name: TACHE
        notification:
          subscriptions:
            - user_email: ${var.mail_alert_1}
      schedule:
        pause_status: PAUSED
        quartz_cron_schedule: "0 0 9 * * ?"
        timezone_id: "Europe/Paris"
      custom_summary: "[${var.domain_name}][${bundle.target}] Notes on {{QUERY_RESULT_VALUE}}"
      custom_description: |
        <b>Domaine : </b><br>
        <b>Environnement : ${bundle.target}</b><br>
        <b>Tâche : {{QUERY_RESULT_VALUE}}</b><br>
        <br>
        <b>------------------------------------------</b><br>

simple job using alert with parameters (doesn't work)

resources:
  jobs:
    test_alert:
      name: test notif
      tasks:
        - task_key: alert_gen_ko
          alert_task:
          alert_id: ${resources.alerts.generic_alert_notification.id}
          parameters:
            task: OVERIDE TASK NAME
   queue:
     enabled: true

Re: Lakebase Data API – First Request Times Out After 8 Seconds When Compute Is Offline

AbhilashNagilla — Thu, 03 Sep 2026 15:17:19 GMT

The Data API connects as the authenticator role, which then assumes the requesting user's identity. PostgreSQL loads a role's configuration defaults only at login; SET ROLE and SET SESSION AUTHORIZATION do not load them, and a role-level setting outranks a database-level one (ALTER ROLE). So your ALTER DATABASE change is overridden by the statement_timeout loaded for authenticator, a setting on the assumed user role is never loaded, and authenticator itself cannot be altered.

PostgreSQL counts statement_timeout from when the query reaches the server, so SQLSTATE 57014 confirms a cancellation but does not prove that reactivation consumed the eight seconds.

No Data API setting changes this timeout, so I would:

Enable Server-Timing headers, reproduce the failure, and keep whatever headers appear on the failed response.
Not retry on 57014 alone, since it also covers ordinary cancellation. Retry once only when the request follows a known suspension and is safe to replay, and make writes idempotent first. The scale-to-zero guidance recommends retry logic during reactivation.
For latency-sensitive calls, warm compute with an earlier lightweight read, lengthen the inactivity period, or turn off scale to zero. A longer period reduces how often compute suspends but cannot keep it always active (startup latency).
If it keeps recurring, have an authorized contact open a support case with the project, branch, endpoint, timestamps, the SQLSTATE and message, pg_roles.rolconfig, and any Server-Timing headers.

Re: Is it possible to embed a Genie Space inside a Power BI report?

fjrodriguez — Thu, 03 Sep 2026 15:01:26 GMT

Thanks every one ! Was a great one :9

Re: MuleGraph Investigator: Uncovering Money Mule Networks with Databricks Genie

LingeshK — Thu, 03 Sep 2026 14:11:26 GMT

This is pretty cool @niteshm . I like how it starts with one alert and tracing the hidden network!

Re: Aprender Data Engineer

balajij8 — Thu, 03 Sep 2026 13:54:11 GMT

@DavidSandoval Vuelve el Databricks Advanced Learning Festival. 50 % de descuento en el cupón de examen. Más información aquí: Del 16 de septiembre al 14 de octubre de 2026. 50 % de descuento en cupones para exámenes de certificación . Una gran oportunidad para invertir en tus #habilidades y obtener la certificación de Databricks.