- 491 Views
- 2 replies
- 2 kudos
Resolved! How to use dlt.expect to validate table level constraints?
I know how to validate the column level constraint, like checking whether the specified column value is larger than target value.Can I validate some table level constraints? For example, validate whether the total records count of a table is larger t...
- 491 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @guangyi ,Unfortunately, there is no out of the box solution for this requirement in dlt. But as a workaround you can add an additional view/table to your pipeline that defines an expectation in similar way to below: CREATE OR REFRESH MATERIALIZED...
- 2 kudos
- 903 Views
- 6 replies
- 2 kudos
AnalysisException: is not a Delta table. but that table is Delta table
When running a Databricks notebook,an error occurs stating that SOME_TABLE is not a Delta table.However, after executing the describe detail command and checking the format,the table is shown as Delta.Without taking any specific actions, re-running t...
- 903 Views
- 6 replies
- 2 kudos
- 2 kudos
Another thing you could check is how the underlying data looks like. Maybe the actual writer of the data, messed it up.
- 2 kudos
- 735 Views
- 3 replies
- 0 kudos
Resolved! Migrate delta live tables from a non-uc to uc enabled databricks workspace
Hello Team,I have a usecase where I need to migrate Delta Live Tables from a non-Unity Catalog Databricks workspace to a Unity Catalog enabled Databricks workspace. What will be the efficient approach to achieve this scenario, please suggest?
- 735 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Fnazar ,You can try to use UCX by Databricks Labs. It automates whole process. databrickslabs/ucx: Automated migrations to Unity Catalog (github.com)
- 0 kudos
- 2514 Views
- 2 replies
- 1 kudos
Databricks Jobs For Each Functionality
Hi Team, I have two questions, please clarigy.1. can we use multiple tasks inside a for each for databricks jobs2. is nested for each functionality available?
- 2514 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @varshini_reddy,While running multiple tasks within a for each loop is not available, there is a simple workaround.Workaround: create a main notebook to run multiple child notebooks:Suppose you need to run notebook1, notebook2, and notebook3 withi...
- 1 kudos
- 408 Views
- 2 replies
- 1 kudos
Resolved! Can we add two corn expression in workflows
I wan to add two cron expression to run my job on each Friday and 1 day of each month.how it is possible 0 15 1 1 * ? *0 15 1 ? * 6 *
- 408 Views
- 2 replies
- 1 kudos
- 2942 Views
- 4 replies
- 0 kudos
issue with azure databricks workspace after we disable public network access
Hi All,We had azure databricks workspaces created thru terraform with public network access enabled to true and everything was working great. recently we have disabled the public network access and started to face issues.terraform is uanble to add us...
- 2942 Views
- 4 replies
- 0 kudos
- 0 kudos
Do you got any solution for this issue? I am also facing the same issue.
- 0 kudos
- 574 Views
- 4 replies
- 4 kudos
Dashboard use case - order of bars
On a spark dataframe, is there any smart way to set the order of a categorical feature explicitly, equivalent to Categorical(ordered=list) in Pandas? The use case here is a dashboard in Databricks, and I want the bars to be arranged in certain order....
- 574 Views
- 4 replies
- 4 kudos
- 188 Views
- 0 replies
- 0 kudos
foreach execution faulty with number of partitions >= worker cores
In order to download multiple wikipedia dumps, I collected the links in the list and wanted to use foreach method to iterate over those links and apply a UDF that downloads the data in the previously created volume structure. However, I ran into an i...
- 188 Views
- 0 replies
- 0 kudos
- 2845 Views
- 2 replies
- 0 kudos
DLT: Waiting for resources took a long time
Hi Team,I have a DLT pipeline running in Production for quite some time now. When I check the pipeline, a couple of jobs took longer than expected. Usually, 1 job only took 10-15 minutes to complete with 2 to 3 mins to provision a resource. Then I ha...
- 2845 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello,To troubleshoot performance issues in your DLT pipeline, review pipeline logs, profile the pipeline, optimize data transformations, increase parallelism, and adjust resource allocation. Identify and address resource provisioning delays, data vo...
- 0 kudos
- 3369 Views
- 4 replies
- 2 kudos
Resolved! Data lineage graph is not working
Hi Team,The issue - Data lineage graph is not working (16-feb, 17-18 Feb) – I created the below tables but when I click the lineage graph not able to see the upstream or downstream table .... the + sign goes away after a few sec but not able to clic...
- 3369 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi Kaniz,We're encountering the same issue where the lineage is not getting populated for a few tables. Could you let us know if a fix has been implemented in any runtime?"We are uaing job cluster 12.2.x .
- 2 kudos
- 17889 Views
- 9 replies
- 7 kudos
How to make the job fail via code after handling exception
Hi , We are capturing the exception if an error occurs using try except. But we want the job status to be failed once we got the exception. Whats the best way to do that. We are using pyspark.
- 17889 Views
- 9 replies
- 7 kudos
- 7 kudos
you can do some hack arround dbutils = get_dbutils(spark) tables_with_exceptions = [] for table_config in table_configs: try: process(spark, table_config) except Exception as e: exception_detail = f"Error p...
- 7 kudos
- 690 Views
- 3 replies
- 1 kudos
/api/2.1/jobs/runs/get-output api response
/api/2.1/jobs/runs/get-output what are the possbile status or state values of the api?I am trying to check the status of my job run based on run id. but there is not enough detail on the response body which contains the status of the run.
- 690 Views
- 3 replies
- 1 kudos
- 1 kudos
@monil It's documented well in the API documentation.https://docs.databricks.com/api/workspace/jobs/getrunoutput
- 1 kudos
- 8818 Views
- 4 replies
- 1 kudos
PowerApps connection to Azure Databricks
Hello,​I would like to read Databricks delta table to show the data on the screen using PowerApps gallery and insert new records to the same table also. What is the best method to achieve an efficient connection and perform above?​​Cheers
- 8818 Views
- 4 replies
- 1 kudos
- 1 kudos
Anyone find a solution to this yet? I'm currently investigating the same issue. Currently the only one I can find is paying for a third-party tool to set it up. Thanks,
- 1 kudos
- 1711 Views
- 6 replies
- 1 kudos
Resolved! Adding extra libraries to databricks (rosbag)
HelloI have interesting challenge, I am required to install few libraries which are part of rosbag packages, for allowing some data deserialization tasks.While creating cluster I do use init_script that install this software using apt sudo apt upd...
- 1711 Views
- 6 replies
- 1 kudos
- 1 kudos
@PiotrU did adding the path to sys.path resolve all of your ModuleNotFoundErrors? i'm trying to do something similar and adding the path to the sys.path resolved ModuleNotFoundError for rclpy, but i continue to see others related to ros
- 1 kudos
- 6088 Views
- 3 replies
- 0 kudos
Control query caching using SQL statement execution API
I want to execute this statement using databricks SQL Statement Execution API. curl -X POST -H 'Authorization: Bearer <access-token>' -H 'Content-Type: application/json' -d '{"warehouse_id": "<warehouse_id>", "statement": "set us...
- 6088 Views
- 3 replies
- 0 kudos
