Hi,I am using UCX for unity catalog migration and up to the step of table migration. When I run the command databricks labs ucx create-table-mapping, it returns the error message 'ERROR [src/databricks/labs/ucx.create-table-mapping] ValueError: Pleas...
I am trying to load on prem transaction log table and update a databricks table.In Source table,the column reqdetails table hold all the information and expect ProductID remaining columns are dynamic(not all columns exists in the request) all time th...
Hi RucHow are you doing today?, as per my understanding, Since your source table has a ReqDetails column with dynamic key-value pairs and only ProductID is fixed, you can parse the ReqDetails column dynamically and merge it into the target table usin...
while accessing event logs created by dlt pipeline, i am getting below error.[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the correspondin...
Hi Pawan,How are you doing today? As per my understanding, This error likely indicates an internal issue with Databricks, Delta Live Tables (DLT), or a Spark plugin. First, check if there are any known issues in Databricks release notes or community ...
Hi All,User uploads an excel file from power apps interface, excel file has to store in databricks folder and insert the excel data into sql table, storing excel file in databricks and inserting data has to go through power automate. Please suggest ...
Hi @Sridhark How are you doing today? As per my understanding, You can achieve this by setting up a Power Automate flow that triggers when a user uploads an Excel file in Power Apps, storing it in OneDrive or SharePoint first. Then, use Power Automat...
Hello Experts,We have data bricks on Azure. We need to provide a user interface to users so that some customizing table entries can be entered by end users which in turn are saved in the Delta table. Is there any feature in Databricks or tools that w...
I achieved this using textbox widget and asked user to enter and run dashboard, in python code validating and inserting data in databricks.2nd method , created excel vba application, using odbc connection to insert delete and update data in delta tab...
I have to convert Vertica queries in Databricks SQLs, so that I can run them in databricks environment. So I want to know the list of all keywords, functions or anything that is different in databricks SQL.
Hello @vidya_kothavale, Please refer to: https://docs.databricks.com/aws/en/sql/language-manual https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-reserved-words
Hi All,I have a Streamlit app running via Databricks Apps. The app is fairly simple and displays data from a small handful of lightweight database queries (each running in less than 1 second).As of a few days ago, this app was running great. But as o...
Hello @brianr, Have you validated the SQL warehouse you are referring in your code? I would be useful to identify the delayed queries and check it's status from "Query History" in Databricks UI.
Hi, i'm using the medallion architecture and the bronze (autoloader with outputmode append) has the full history. So i decided to user the silver zone to dedup the bronze using the 'change data feed'. But when i this to do the upsert i got the messag...
Hi @rafaelcavalcant How are you doing today? As per my understanding, The error happens because _change_type is appearing twice in your schema, likely due to how you're processing the Change Data Feed (CDF). This can happen if SELECT * is pulling in ...
Created a ADF pipeline, having databricks notebook activity. In that using python code. Have a scenario to use some classes from azure devops repos. As part of that , have to clone the repo into databricks workpsace. I have used managed identity to c...
Hi @amitDE, Could you please review this documentation and make sure setup looks fine? https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/use-ms-entra-sp-with-devops
Error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)> python --versionPython 3.10.4This error seems to be coming from the thrift backend. I suspect but have not confirmed that t...
You can set up an SSL context that skips certificate verification with the following command:import ssl ssl._create_default_https_context = ssl._create_unverified_contextI have followed some steps from this source.
Hi, Is there a way to read a sql table using jdbc as a streaming source? If not what is the best approach to read incremental data from a sql table and implement SCD Type 1 while loading into silver and gold layer. Can this be implemented using dlt p...
hi,You can use ADF to extract data incrementally into the bronze layer, and use DLT to load this data into silver with SCD1 or SCD2.You will need to store somewhere information about last extracted ID and read it in ADF pipeline to get only new recor...
Hi Folks,I am trying to retrieve table properties of delta table in Databricks dashboard data editor but getting error message.It seems like describe and show command not working with dashboard editor as Query is working fine in sql editor and notebo...
Hello @Anmol_Chauhan! Dashboard Data Editor is designed for querying and working with datasets to create dashboard visualizations. However, outputs from commands like show, describe, or explain are not valid as datasets, which is why the query does n...
When I try to fetch system.information_schema.schema_tags, it shows up but when I'm trying to fetch system.information_schema.table_tags it's not showing up and returns an empty df. Is there anything I am missing?assert os.getenv('DATABRICKS_WAREHOUS...
Hi @shekharshukla How are you doing today? As per understanding, It looks like the system.information_schema.table_tags query is returning an empty DataFrame, which could be due to a couple of reasons. First, make sure that there are actually tags as...
Hello everyone,We recently installed UCX and were able to access the UCX Assessment Dashboard successfully. However, we’re now seeing an error stating: "The warehouse was not found." I suspect that someone may have accidentally deleted the warehouse ...
Hi @Faizan_khan8171 How are you doing today? As per my understanding, It looks like the warehouse linked to your UCX Assessment Dashboard was deleted, which is likely causing the error. You can try checking under SQL Warehouses to see if it's still t...
Hi, I regularly use the repair job option to rerun a subset of a larger data pipeline job. However, this option is disabled if I start this job from another "orchestration job" via Job Run tasks. The "repair" button is disabled in my case. This restr...
Hi @carolsun08 How are you doing today?, Yeah, it's a bit frustrating that the repair job option is disabled when running jobs through Job Run tasks in an orchestration job. While there's no official confirmation on when this might change, it would d...
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 484 | |
| 349 | |
| 287 |
