Hi team!I have a Notebook (notebook A) in workspace and I'd like to execute it with %run command from another Notebook (notebook B). It works perfect with command: %run /workspace/path/to/notebook/ANow, i want to specify above path in a variable, an...
Hi!I have a workflow job to trigger a refresh of an online table. How can I get the update_id with this specific refresh?Also, is it possible to get the status from this specific update_id?Thanks!
Another qq: Since online table has 3 sync mode: Snapshot, Triggered, and Continuous. when refreshing the online table with w.pipelines.start_update(pipeline_id='{pipeline_id}', full_refresh=True) which sync mode is used by default?
Hi I got this error "com.databricks.common.client.DatabricksServiceHttpClientException: 403: Invalid access token" during the run of a workflow job. It has been working for a while without error. Nothing has changed in regards to code or cluster. And...
Hi @skanapuram, The error "com.databricks.common.client.DatabricksServiceHttpClientException: 403: Invalid access token" typically occurs when the access token used for authentication has expired or is invalid. Have you reviewed the driver logs of th...
There are few fields in the source CSV file which has spaces in the field names. When I tried to create the table like `SOURCE 1 NAME` as string for few fields, I got an error message like 'INVALID_COLUMN_NAME_AS_PATH' error. Runtime version is 10.2...
Hi there,I’m working on a POC to connect a C# application to query tables from Unity Catalog using the ODBC connector. Currently, I’m testing this locally using Visual Studio. I followed the steps in the ODBC documentation, but I’m encountering the f...
The error message you encountered, "ERROR [ODBC Driver Manager] Data source name not found and no default driver specified," typically indicates that the ODBC driver manager cannot find the specified data source name (DSN) or that no default driver i...
Hi there!I’d like to use Liquid Clustering in a Spark Streaming process with foreachBatch(upsert). However, I’m not sure of the correct approach.The Databricks documentation suggests using .clusterBy(key) when writing streaming data. In my case, I'm ...
Hello Community,I have a question about migrating data from PostgreSQL to Databricks. My PostgreSQL database receives new data every hour, and I want to synchronize these hourly inserts with the bronze layer in my Databricks catalog.Currently, I’m us...
Hello Walter,Thank you for your help - you're amazing. I wanted to explain my current challenge in more detail:We have a platform that stores data in PostgreSQL, with a pipeline ingesting millions of rows every hour. We're trying to migrate this data...
Hello Community,I have a basic question that I’ve been thinking about lately. Is it better to use DLT Pipelines or CDF Delta Tables for handling a medallion architecture?I understand that DLT Pipelines offer some shortcuts, but are they a good choice...
Hi @jeremy98, When deciding between using Delta Live Tables (DLT) Pipelines and Change Data Feed (CDF) Delta Tables for handling a medallion architecture, there are several factors to consider. DLT Pipelines: Automation and Management: DLT Pipeli...
Hi Team, Good Day! Recently in a Credentials section under Catalog , we have to add the self assume capabilities in the IAM role right.. Is it only for the Roles associated with Unity Catalog or for all the roles?. Thanks, Gowtham
Hi @GowthamR,I believe it's only for the roles associated with UC. I was going through this community post on including self-assume capabilities for AWS IAM roles and it's mentioned that this change does not affect storage credentials that are not cr...
I need to access delta tables with API outside azure using in a reporting tool workiva with using the connector. Can someone able to provide the details on how I can achieve it
Hi @Shivaprasad ,Accessing Delta tables in Databricks from external tools or platforms requires using Databricks REST APIs or JDBC/ODBC connectors. Does your platform supports integration of code snippets?
I'm new to Databricks and, I'm looking to get data from an external database into Databricks and keep it synchronized when changes occur in the source tables. It seems like I may be able to some form of change data capture and the delta live tables. ...
Hey @peritus ,I would suggest using Lakehouse Federation and create a DLT pipeline to read tables as Materialised Views. If you trigger a refresh of that pipeline at a scheduled interval, you should have the SQL server data replicated in Databricks.
Hi Databricks Community,I'm encountering an issue with watermarks in Delta Live Tables that's causing data loss in my streaming pipeline. Let me explain my specific problem:Current SituationI've implemented watermarks for stateful processing in my De...
Dear @VZLA, @Walter_C ,I wanted to take a moment to express my sincere gratitude for your incredibly detailed explanation and thoughtful suggestions. Your guidance has been immensely valuable and has provided us with a clear path forward in addressi...
I've the filename as below and I want to extract the datetime values and convert to datetime data type. This_is_new_file_2024_12_06T11_00_49_AM.csvHere I want to extract only '2024_12_06T11_00_49' and convert to datetime value in new field. I tried S...
Unfortunately I am not able to make it work with SQL functions
Hi there,If I understood correctly, Roland said output SQL task can be used as input to ForEach task in Workflows. I tried that and used the expression sqlTaskName.output.rows, but Databricks rejected that expression. Anyone know how to do that?
Our internal teams has confirmed that this is currently not working on your side as this feature is currently in Private preview we will need to wait for some time until it is fully released.
Good morning, I'm trying to run: databricks bundle run --debug -t dev integration_tests_job My bundle looks: bundle: name: x include: - ./resources/*.yml targets: dev: mode: development default: true workspace: host: x r...
Hello, Reopening this ticket in hopes that either of you had some luck in resolving your bug. I am currently facing the same issue where I can deploy an asset bundle via the local CLI without issue (by deploy I mean the bundle code is written to my ...
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group| User | Count |
|---|---|
| 1608 | |
| 763 | |
| 345 | |
| 286 | |
| 251 |
