- 856 Views
- 5 replies
- 2 kudos
Resolved! Updating Job Creator to Service Principal
Regarding data governance best practices: I have jobs created by a user who has left the organization, and I need to change the job creator to a service principal. Currently, it seems the only option is to clone the job and update it. Is this the rec...
- 856 Views
- 5 replies
- 2 kudos
- 2 kudos
I agree with @nayan_wylde , for auditing, creator is important and it should in immutable by nature.
- 2 kudos
- 13434 Views
- 6 replies
- 0 kudos
Can I pass parameters to a Delta Live Table pipeline at running time?
I need to execute a DLT pipeline from a Job, and I would like to know if there is any way of passing a parameter. I know you can have settings in the pipeline that you use in the DLT notebook, but it seems you can only assign values to them when crea...
- 13434 Views
- 6 replies
- 0 kudos
- 572 Views
- 2 replies
- 1 kudos
Resolved! BUG: Merge with schema evolution doesn't work in update clause
I am referring to this link of databricks documentation. Here is a screenshot of the same According to the documentation the UPDATE command should work when the target table doesn't have the column but it is present in source. I tried the same with ...
- 572 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @iyashk-DBThanks for the response, it will help in resolving the issue.But, can you mark it as a bug and report it? Because specifying just the column without the table name is a little risky.
- 1 kudos
- 1583 Views
- 2 replies
- 1 kudos
Resolved! Structured streaming for iceberg tables
According to this https://iceberg.apache.org/docs/latest/spark-structured-streaming/ , we can stream from iceberg tables. I have ensured that my source table is Iceberg version 3, but no matter what I do, I get Iceberg does not streaming reads. Looki...
- 1583 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi,Iceberg streaming is possible in Databricks. One does not need to change to Delta Lake. In my previous attempt, I used "load" while reading the source iceberg table. One should instead use "table". Load apparently seems to take a path and not a ta...
- 1 kudos
- 408 Views
- 1 replies
- 0 kudos
OBO auth implementation in Streamlit not working
Hello,I am currently trying to implement OBO auth in a streamlit db app but I'm getting the following error message:requests.exceptions.HTTPError: 400 Client Error: PERMISSION_DENIED: User does not have USE CATALOG on Catalog '...'. Config: host=, a...
- 408 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @rcatelli Here's a quick example https://docs.databricks.com/aws/en/dev-tools/databricks-apps/auth#user-authorization https://docs.databricks.com/aws/en/dev-tools/databricks-apps/auth#example-query-with-user-authorization Get the user token from...
- 0 kudos
- 401 Views
- 1 replies
- 1 kudos
Resolved! CLuster Config
Hi, can we pass a parameter into job activity from ADF side to change the environment inside the job cluster configuration?
- 401 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @Naren1 , Yes — you can pass parameters from ADF to a Databricks Job run, but you generally can’t use those parameters to change the job cluster configuration (node type, Spark version, autoscale, init scripts, etc.) for that run.In an ADF Data...
- 1 kudos
- 218 Views
- 1 replies
- 0 kudos
Resolved! Alert ID within job yaml file - different environments
Hi, I am trying to trigger an alert through a job, and the issue I'm experiencing is that we have the same alert name in our dev/test/pre/prod environments but they will all have different alert IDs. And I have to input an alert id within the job yam...
- 218 Views
- 1 replies
- 0 kudos
- 0 kudos
@halsgbs your notebook task workaround failed as Databricks Jobs expect a static alert_id at the time the job is submitted or created, not a dynamic variable evaluated during the run.The best way to deal with this is Asset Bundle (DAB) where you get ...
- 0 kudos
- 822 Views
- 2 replies
- 1 kudos
Resolved! Databricks Governance Dashboards
I am looking for any prebuilt governance dashboards. I see in some demo videos a governance portal. Is that or similar available to load to our environments. I am aware of the data quality and profiling, but a single view of key indicators would be a...
- 822 Views
- 2 replies
- 1 kudos
- 1 kudos
@MoJaMa This looks beautiful and insightful.@turagittech We are currently building a governance dashboard separately in Power BI, let me know if you want some KPIs or how to start with that.
- 1 kudos
- 614 Views
- 2 replies
- 2 kudos
Resolved! Salesforce Connector SCD2 - Get new record with isDeleted = true on deletion
Hi all,I'm using the Databricks Salesforce connector to ingest tables with history tracking enabled (SCD Type 2).When records are deleted in Salesforce, the connector closes the existing record by setting the end date. The isDeleted flag remains fals...
- 614 Views
- 2 replies
- 2 kudos
- 2 kudos
Greetings @AcrobaticMonkey , I put on my researcher hat and dug into our internal docs. Here is what I found: Short answer: this isn’t configurable today. The connector’s SCD Type 2 behavior “closes” a record by setting __END_AT and does not emit a ...
- 2 kudos
- 1061 Views
- 5 replies
- 3 kudos
Resolved! Warehouse ID specified in job yaml file for sql tasks
My goal is to trigger an alert I have through a job, and it seems I have to specify the warehouse id within the job yaml file itself. We have different environments with different warehouse ids, and the issue is that if I specify the warehouse id in ...
- 1061 Views
- 5 replies
- 3 kudos
- 3 kudos
Thank you! looks like the alert_id also needs to be parametised, and I was wondering if its possible to use a job parameter to do so? If I can use the alert name then that would be great but I believe it has to be the alert id, which will be differen...
- 3 kudos
- 599 Views
- 6 replies
- 0 kudos
Sql server setup for lakeflow sql server connector to create ingestion
When i am executing below command change instance is getting created but without lakeflow as prefix, i read the documentation and it mentioned that to track schema evolution we need to have prefix, can I please get some assistance.Command Used:EXEC d...
- 599 Views
- 6 replies
- 0 kudos
- 0 kudos
and when i altered the table got below warning : WARNING: Table [dbo].[test_table] has a pre-existing capture instance named 'dbo_test_table' that was not created by lakeflow. Lakeflow will preserve this instance and create its own instance alongside...
- 0 kudos
- 584 Views
- 6 replies
- 3 kudos
Resolved! Result Difference Between View and Manually Run View Query
I am experiencing an issue where a view does not display the correct results, but running the view query manually in either a new notebook or the SQL Editor displays different, correct results. I have tried switching the compute resource in the noteb...
- 584 Views
- 6 replies
- 3 kudos
- 3 kudos
There are 2 fixes that I can think off Option A: Make first_value deterministic first_value(Customer_ID, true) OVER ( PARTITION BY customer_name ORDER BY submitted ASC, event_id ASC ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING ) U...
- 3 kudos
- 428 Views
- 1 replies
- 1 kudos
Resolved! Unable to update DLT-based materialized view if clustering key is missing
If we set up a materialized view with a clustering key, and then update the definition such that this key is no longer part of the table, Databricks complains:Run ALTER TABLE ... CLUSTER BY ... to repair Delta clustering metadata.But this is not poss...
- 428 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @Malthe , Currently, there is no supported way to repair broken clustering metadata in Delta materialised views if you remove the clustering key from the definition, other than dropping and recreating the materialised view. Additionally, a full...
- 1 kudos
- 324 Views
- 1 replies
- 3 kudos
Resolved! DLT Pipeline issue
Hello Everyone, I have setup a DLT pipeline and while running it first time, I am getting the below issue:key not found: test_bronze_dltNoSuchElementExceptionkey not found: test_bronze_dlt test_bronze_dlt - this my DLT table name that is expected to...
- 324 Views
- 1 replies
- 3 kudos
- 3 kudos
Hello @manjeetgahlawat , NoSuchElementException: key not found: test_bronze_dlt occurs when the table/view in the pipeline references a LIVE dataset named test_bronze_dlt, but DLT cannot find a dataset with that exact name in the pipeline graph. (So ...
- 3 kudos
- 9531 Views
- 4 replies
- 2 kudos
Resolved! How can I check that column on a delta table has a "NOT NULL" constraint or not?
Title pretty much says it all - I'm trying to determine whether or not a column on my existing delta table was defined as NOT NULL or not. It does not show up in any of the metadata (describe detail, describe history, show tblproperties). Thanks in...
- 9531 Views
- 4 replies
- 2 kudos
- 2 kudos
@muki , you can run SHOW CREATE TABLE <catalog>.<schema>.<table> and in that also you can also see the constraints applied.
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
