Regarding data governance best practices: I have jobs created by a user who has left the organization, and I need to change the job creator to a service principal. Currently, it seems the only option is to clone the job and update it. Is this the rec...
I want to raise a feature request as follows.Currently, in the Automatic schema evolution for merge when a column is not present in the source dataset it is not changed in the target dataset. For e.g.%sql CREATE OR REPLACE TABLE edw_nprd_aen.bronze.t...
I am referring to this link of databricks documentation. Here is a screenshot of the same According to the documentation the UPDATE command should work when the target table doesn't have the column but it is present in source. I tried the same with ...
Hi @iyashk-DBThanks for the response, it will help in resolving the issue.But, can you mark it as a bug and report it? Because specifying just the column without the table name is a little risky.
getting below error-Failure starting repl. Try detaching and re-attaching the notebook. while executing notebook and can see cluster have all installed lib.
Hi ,this issue might be related to hive metastore migration related.PFBi can see driver log having this connection issue which matching the host in below url-IP CIDR For Azure Databricks Hive Metastore - Databricks Community - 133401Caused by: java.s...
According to this https://iceberg.apache.org/docs/latest/spark-structured-streaming/ , we can stream from iceberg tables. I have ensured that my source table is Iceberg version 3, but no matter what I do, I get Iceberg does not streaming reads. Looki...
Hi,Iceberg streaming is possible in Databricks. One does not need to change to Delta Lake. In my previous attempt, I used "load" while reading the source iceberg table. One should instead use "table". Load apparently seems to take a path and not a ta...
Hello,I am currently trying to implement OBO auth in a streamlit db app but I'm getting the following error message:requests.exceptions.HTTPError: 400 Client Error: PERMISSION_DENIED: User does not have USE CATALOG on Catalog '...'. Config: host=, a...
Hi @rcatelli Here's a quick example https://docs.databricks.com/aws/en/dev-tools/databricks-apps/auth#user-authorization https://docs.databricks.com/aws/en/dev-tools/databricks-apps/auth#example-query-with-user-authorization Get the user token from...
Hi, can we pass a parameter into job activity from ADF side to change the environment inside the job cluster configuration?
Hello @Naren1 , Yes — you can pass parameters from ADF to a Databricks Job run, but you generally can’t use those parameters to change the job cluster configuration (node type, Spark version, autoscale, init scripts, etc.) for that run.In an ADF Data...
Hello everyone,I would like to clarify some doubts regarding how Databricks Pipelines (DLT) behave when using serverless pipelines with incremental updates.In general, incremental processing is enabled and works as expected. However, I have observed ...
Hi, I am trying to trigger an alert through a job, and the issue I'm experiencing is that we have the same alert name in our dev/test/pre/prod environments but they will all have different alert IDs. And I have to input an alert id within the job yam...
@halsgbs your notebook task workaround failed as Databricks Jobs expect a static alert_id at the time the job is submitted or created, not a dynamic variable evaluated during the run.The best way to deal with this is Asset Bundle (DAB) where you get ...
I'm trying to perform a simple example using structured streaming on a directory created as a Volume. The use case is purely educational; I am investigating various forms of triggers. Basic info:Catalog: "dev_catalog"Schema: "stream"Volume: "streamin...
I am looking for any prebuilt governance dashboards. I see in some demo videos a governance portal. Is that or similar available to load to our environments. I am aware of the data quality and profiling, but a single view of key indicators would be a...
@MoJaMa This looks beautiful and insightful.@turagittech We are currently building a governance dashboard separately in Power BI, let me know if you want some KPIs or how to start with that.
I am developing a Databricks application using the Streamlit package. I was able to get a "hello world" app deployed successfully, but now I am trying to pass data that exists in the dbfs on the same instance. I try to read a csv saved to the dbfs bu...
I used Azure Blob Storage, and this resolved the issue. Our app now contains Python files in the Databricks environment app that have access to Azure Blob Storage using Azure credentials.
Hello Team,I am facing a consistent issue when streaming a large table (~53 million rows) from a Databricks SQL Warehouse using Python (databricks-sql-connector) with OAuth authentication.I execute a single long-running query and fetch data in batche...
Hi @Rose_15 53M rows via streaming looks big number. What is the use case, if the use case is streaming possibly, you need find a way to extract small batches ( like incremental) and load the data. The use case more look like a batch load. Please pro...
Hi Community,I’m working with Databricks Apps (FastAPI) and invoking the API from external IoT devices.Currently, the recommended approach is to authenticate using a Bearer token generated via a Databricks Apps Service Principal (Client ID + Client S...
I am on Databricks Run Time LTE 14.3 Spark 3.5.0 Scala 2.12 and mongodb-spark-connector_2.12:10.2.0. Trying to connect to Document DB using the connector and all I get is a connection timeout. I tried using PyMongo, which works as expected and I can ...
Hi @rijin-thomas - Can you please allow the CIDR block for databricks account VPC from aws document db sg ( Executor connectivity stated by@bianca_unifeye ) .
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 517 | |
| 349 | |
| 287 |
