- 35 Views
- 3 replies
- 0 kudos
Power BI refresh job task
I have tried Databricks job task to refresh power bi dataset and I have found 2 issues.1. I set up tables in Power BI Desktop using Import mode. After deploying the model to Power BI Service, I was able to download it as an Import mode model. However...
- 35 Views
- 3 replies
- 0 kudos
- 0 kudos
Can you send a screenshot of the refresh power BI task in the jobs UI within Databricks please?
- 0 kudos
- 2214 Views
- 3 replies
- 0 kudos
unity catalog system.access.audit lag
Hello,We have unity catalog enabled workspace. To get the completion time of a pipeline that runs multiple times a day, I am checking system.access.audit table. Comparing the completion time of the pipeline compared to other pipeline time I am creat...
- 2214 Views
- 3 replies
- 0 kudos
- 0 kudos
@angel_ba - This is expected/designed behaviour.Audit logs are ingested into the system tables asynchronously. Databricks batches these events befor surfacing them in UC system tables. Alternate (prhaps) the best way is to use Job API for start/compl...
- 0 kudos
- 29 Views
- 1 replies
- 0 kudos
AI/BI Genie
I was working with genie and started using Research agent, and now I want to make the genie as a sharable Infrastructure as Code where I can manage all the changes through GitHub and so does anyone have any suggestions how to do this in a best optimi...
- 29 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Gaganmjain_012 Hi once in github then if you want to deploy via asset bundles then https://github.com/databricks/cli/issues/3008 looks like an open request/
- 0 kudos
- 20 Views
- 1 replies
- 0 kudos
integrating linear app with databricks
I want to integrate linear app with databricks .. the moment any job fails i want to create a linear task with the error in the description . can you guide me to implement this
- 20 Views
- 1 replies
- 0 kudos
- 0 kudos
@hidden Configure a Generic Webhook notification destination in Databricks that points to your own small service (AWS Lambda)In that service, parse the payload, detect job failure events, extract the relevant error/stack trace, and call the Linear Gr...
- 0 kudos
- 60 Views
- 1 replies
- 2 kudos
Resolved! Databricks Data Engineer Associate Exam Dec 2025
Hi, I am prepping for the Databricks DEA exam. I am seeing some conflicts with DLT/DP or declarative pipeline syntax. I am trying to see if the most up to date syntax is going to be on the exam as opposed to the legacy DLT. For example, current DP sy...
- 60 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello, @rc10000 You are correct to prioritize the current syntax. The most up-to-date Databricks Certified Data Engineer Associate exam will focus on the declarative syntax within Lakeflow Spark Declarative Pipelines. You should prepare for CREATE OR...
- 2 kudos
- 110 Views
- 2 replies
- 0 kudos
Jackson version conflict
Hi,I am trying to implement the Spark TableProvider api and i am experiencing a jar conflict (I am using the 17.3 runtime). com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.15.2 requires Jackson Databind version >= 2.15.0 and < 2.1...
- 110 Views
- 2 replies
- 0 kudos
- 0 kudos
For now we are trying to contact Databricks, In worst case scenario we were planning to shade the dependencies we need.would love to hear what has worked for you.Best,Shimon
- 0 kudos
- 51 Views
- 1 replies
- 0 kudos
DLT File Level Deduplication
I want to create a DLT pipeline that incrementally processes csv files arriving daily. However, some of those files are duplicate - they have the same names and data but are in different directories. What is the best way to handle this? I'm assuming ...
- 51 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @dgahram , Auto Loader tracks ingestion progress by persisting discovered file metadata in a RocksDB store within the checkpoint, which provides “exactly-once” processing for discovered files.Doc: https://docs.databricks.com/aws/en/ingestion...
- 0 kudos
- 84 Views
- 4 replies
- 0 kudos
Extract Snowflake data based on environment
Hi all, In the development workspace, I need to extract data from a table/view in Snowflake development environment. Example table is called as VD_DWH.SALES.SALES_DETAILWhen we deploy the code into production, it needs to extract data from a table/vi...
- 84 Views
- 4 replies
- 0 kudos
- 0 kudos
Option 1 — Notebook widgets (simple & reliable)Use a Databricks widget to pass the environment (e.g., via Jobs/Workflows parameter), then build the table name dynamically.# Databricks notebook dbutils.widgets.dropdown("env", "dev", ["dev", "prod"], "...
- 0 kudos
- 2133 Views
- 4 replies
- 5 kudos
Resolved! Permission Denied when Creating External Tables Using Workspace Default Credential
I’m building out schemas, volumes, and external Delta tables in Unity Catalog via Terraform. The schemas and volumes are created successfully, but all external tables are failing.The error message from Terraform doesn't highlight what the issue is bu...
- 2133 Views
- 4 replies
- 5 kudos
- 5 kudos
Feature or bug, I discovered the same. I couldn't create tables with the default credential. To test, I assigned the default credential and a custom credential the same access rights to the storage container that is the target of the external locatio...
- 5 kudos
- 130 Views
- 3 replies
- 3 kudos
Spark structured streaming- calculate signal, help required! 🙏
Hello everyone!I’m very very new to Spark Structured Streaming, and not a data engineer I would appreciate guidance on how to efficiently process streaming data and emit only changed aggregate results over multiple time windows.Input Stream:Source: A...
- 130 Views
- 3 replies
- 3 kudos
- 3 kudos
I would implement stateful streaming by using transformWithStateInPandas to keep the state and implement the logic there. I would avoid doing stream-stream JOINs.
- 3 kudos
- 2599 Views
- 12 replies
- 2 kudos
Seeking Guidance on Migrating Informatica PowerCenter Workflows to Databricks using Lakebridge
Hi everyone,I hope you're doing well.I'm currently exploring options to migrate a significant number of Informatica PowerCenter workflows and mappings to Databricks. During my research, I came across Lakebridge, especially its integration with BladeB...
- 2599 Views
- 12 replies
- 2 kudos
- 2 kudos
Hi Chirag. At Kanerika Inc,, we've built a migration accelerator that automates 80% of the Informatica to Databricks migration process, saving you significant time, effort, and resources. You can check out the demo video of the same here - https://ww...
- 2 kudos
- 54 Views
- 1 replies
- 0 kudos
Data Ingestion - Missing Permission
Hi, I would like to use Data Ingestion through fivetran connectors to get data from external data source to databricks but I am getting missing permission error. I already have admin permission. I kindly ask your help regarding to this situation.Look...
- 54 Views
- 1 replies
- 0 kudos
- 0 kudos
@bercaakbayir - 2 areas to look at for permissions.Unity Catalog PermissionDestination‑level permissionsPlease Check,UC enabled for your workspace. [Metastore Admin, not workspace Admin]CREATE permissions on the target catalog - User or SP should hav...
- 0 kudos
- 2844 Views
- 7 replies
- 0 kudos
Triggering DLT Pipelines with Dynamic Parameters
Hi Team,We have a scenario where we need to pass a dynamic parameter to a Spark job that will trigger a DLT pipeline in append mode. Can you please suggest an approach for this?Regards,Phani
- 2844 Views
- 7 replies
- 0 kudos
- 0 kudos
found a working example -databricks pipelines update <pipeline_id> --json @new_config.jsondatabricks pipelines start-update <Pipelineid>where in use JSON for passing parameters.. every run update the parameters with new json file
- 0 kudos
- 827 Views
- 7 replies
- 5 kudos
Resolved! EXCEL_DATA_SOURCE_NOT_ENABLED Excel data source is not enabled in this cluster
I want to read an Excel xlsx file on DBR 17.3. On the Cluster the library dev.mauch:spark-excel_2.13:4.0.0_0.31.2 is installed. V1 Implementation works fine:df = spark.read.format("dev.mauch.spark.excel").schema(schema).load(excel_file) display(df)V2...
- 827 Views
- 7 replies
- 5 kudos
- 5 kudos
I reached out to Databricks support and they fixed it with December 2025 maintenance update. Now the open source excel reader and the new build in should work.https://learn.microsoft.com/en-gb/azure/databricks/query/formats/excel
- 5 kudos
- 124 Views
- 1 replies
- 0 kudos
Lakebase error logs
Anyone know where to see any logs related to Lakebase/Postgres? I have a Tableau Prep flow that is failing but the error is not clear and I'm trying to find out what the database is capturing.
- 124 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @pdiamond ,You can try to use Lakebase monitoring tools to capture query generated by Tableau Prep.Monitor | Databricks on AWSAlternatively, it seems that you can also use external monitoring tools. So you can connect to your lakebase instance usi...
- 0 kudos
| User | Count |
|---|---|
| 1630 | |
| 790 | |
| 512 | |
| 349 | |
| 287 |
