- 816 Views
- 1 replies
- 0 kudos
Keywords and Functions supported in Vertica SQL but not in Databricks SQL.
I have to convert Vertica queries in Databricks SQLs, so that I can run them in databricks environment. So I want to know the list of all keywords, functions or anything that is different in databricks SQL.
- 816 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @vidya_kothavale, Please refer to: https://docs.databricks.com/aws/en/sql/language-manual https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-reserved-words
- 0 kudos
- 2096 Views
- 2 replies
- 0 kudos
Databricks Apps - Streamlit Performance Hangups
Hi All,I have a Streamlit app running via Databricks Apps. The app is fairly simple and displays data from a small handful of lightweight database queries (each running in less than 1 second).As of a few days ago, this app was running great. But as o...
- 2096 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @brianr, Have you validated the SQL warehouse you are referring in your code? I would be useful to identify the delayed queries and check it's status from "Query History" in Databricks UI.
- 0 kudos
- 1621 Views
- 2 replies
- 0 kudos
Resolved! Autoloader with cdf not ignoring reserved columns
Hi, i'm using the medallion architecture and the bronze (autoloader with outputmode append) has the full history. So i decided to user the silver zone to dedup the bronze using the 'change data feed'. But when i this to do the upsert i got the messag...
- 1621 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @rafaelcavalcant How are you doing today? As per my understanding, The error happens because _change_type is appearing twice in your schema, likely due to how you're processing the Change Data Feed (CDF). This can happen if SELECT * is pulling in ...
- 0 kudos
- 1136 Views
- 1 replies
- 0 kudos
Cloning to data bricks from azure dev ops
Created a ADF pipeline, having databricks notebook activity. In that using python code. Have a scenario to use some classes from azure devops repos. As part of that , have to clone the repo into databricks workpsace. I have used managed identity to c...
- 1136 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @amitDE, Could you please review this documentation and make sure setup looks fine? https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/use-ms-entra-sp-with-devops
- 0 kudos
- 34665 Views
- 6 replies
- 0 kudos
How can I resolve this SSL error which occurrs when calling databricks-sql-connector/databricks.sql.connect() from my python app?
Error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)> python --versionPython 3.10.4This error seems to be coming from the thrift backend. I suspect but have not confirmed that t...
- 34665 Views
- 6 replies
- 0 kudos
- 0 kudos
You can set up an SSL context that skips certificate verification with the following command:import ssl ssl._create_default_https_context = ssl._create_unverified_contextI have followed some steps from this source.
- 0 kudos
- 634 Views
- 1 replies
- 0 kudos
Incremental load from a sql table
Hi, Is there a way to read a sql table using jdbc as a streaming source? If not what is the best approach to read incremental data from a sql table and implement SCD Type 1 while loading into silver and gold layer. Can this be implemented using dlt p...
- 634 Views
- 1 replies
- 0 kudos
- 0 kudos
hi,You can use ADF to extract data incrementally into the bronze layer, and use DLT to load this data into silver with SCD1 or SCD2.You will need to store somewhere information about last extracted ID and read it in ADF pipeline to get only new recor...
- 0 kudos
- 938 Views
- 1 replies
- 0 kudos
Query with SHOW and Describe command not working in Databricks Dashboard
Hi Folks,I am trying to retrieve table properties of delta table in Databricks dashboard data editor but getting error message.It seems like describe and show command not working with dashboard editor as Query is working fine in sql editor and notebo...
- 938 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Anmol_Chauhan! Dashboard Data Editor is designed for querying and working with datasets to create dashboard visualizations. However, outputs from commands like show, describe, or explain are not valid as datasets, which is why the query does n...
- 0 kudos
- 824 Views
- 1 replies
- 0 kudos
Not able to access Table_tags in Databricks Apps:
When I try to fetch system.information_schema.schema_tags, it shows up but when I'm trying to fetch system.information_schema.table_tags it's not showing up and returns an empty df. Is there anything I am missing?assert os.getenv('DATABRICKS_WAREHOUS...
- 824 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @shekharshukla How are you doing today? As per understanding, It looks like the system.information_schema.table_tags query is returning an empty DataFrame, which could be due to a couple of reasons. First, make sure that there are actually tags as...
- 0 kudos
- 1108 Views
- 1 replies
- 0 kudos
UCX Assessment Dashboard Error: "The warehouse was not found"
Hello everyone,We recently installed UCX and were able to access the UCX Assessment Dashboard successfully. However, we’re now seeing an error stating: "The warehouse was not found." I suspect that someone may have accidentally deleted the warehouse ...
- 1108 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Faizan_khan8171 How are you doing today? As per my understanding, It looks like the warehouse linked to your UCX Assessment Dashboard was deleted, which is likely causing the error. You can try checking under SQL Warehouses to see if it's still t...
- 0 kudos
- 2166 Views
- 2 replies
- 0 kudos
Resolved! Missing Delta-live-Table in hive-metastore catalog
Hi experts,I defined my delta table in an external location as following:%sqlCREATE OR REFRESH STREAMING TABLE pumpdata (Body string,EnqueuedTimeUtc string,SystemProperties string,_rescued_data string,Properties string)USING DELTALOCATION 'abfss://md...
- 2166 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey @BobCat62 , This might helpdlt will be in direct publishingmode by default. If you select hive_metstore you must specify the default schema in the dlt pipeline setting. If not done there. At the time of defining the dlt table pass the schema_name...
- 0 kudos
- 1557 Views
- 1 replies
- 0 kudos
500 Error on /ajax-api/2.0/fs/list When Accessing Unity Catalog Volume in Databricks
We are encountering an issue with volumes created inside Unity Catalog. We are using AWS and Terraform to host Databricks, and our Unity Catalog structure is as follows:• Catalog: catalog_name• Schemas: raw, bronze, silver, gold (all with external l...
- 1557 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @MrFi How are you doing today?As per my understanding, It looks like the Unity Catalog UI might have trouble handling external volumes, even though dbutils works fine. Try running SHOW VOLUMES IN catalog_name.raw; to check if the volume is properl...
- 0 kudos
- 2801 Views
- 8 replies
- 0 kudos
inconsistent behavior with serverless sql: user is not an owner of table error with views
We get the following error with some basic views and not others when using serverless compute (from a notebook or from SQL Editor or from the Catalog Explorer). Views are simple select * from table x and underlying schemas/tables are using managed m...
- 2801 Views
- 8 replies
- 0 kudos
- 0 kudos
@ceceliac just a quick check, if you rerun the same query after it has initially failed, will it go through or still fail? if it runs fine, wait another 10-15mins and rerun it and share the outcome. So: 1.- Run it once, it will fail. 2.- Rerun it inm...
- 0 kudos
- 1380 Views
- 1 replies
- 0 kudos
RESTORE deletes part of the delta table's history
Having a delta table with the history of 15 versions (see screenshot). After running the command:RESTORE TABLE hive_metastore.my_schema.my_table TO VERSION AS OF 6;And then running DESCRIBE HISTORY (see screenshot) it seems that a new version (RESTOR...
- 1380 Views
- 1 replies
- 0 kudos
- 0 kudos
it's not. I haven't observed this behavior. According to the delta lake documentation "Using the restore command resets the table’s content to an earlier version, but doesn’t remove any data. It simply updates the transaction log to indicate that cer...
- 0 kudos
- 4296 Views
- 2 replies
- 1 kudos
Load data from Aurora to Databricks directly
Hi,Does anyone know how to link Aurora to Databricks directly and load data into Databricks automatically on a schedule without any third-party tools in the middle?
- 4296 Views
- 2 replies
- 1 kudos
- 1 kudos
AWS Aurora supports PostgreSQL or MySQL, did you try to connect using JDBC?url = f"jdbc:postgresql://{database_host}:{database_port}/{database_name}"remote_table = (spark.read.format("jdbc").option("driver", driver).option("url", url).option("dbtable...
- 1 kudos
- 5883 Views
- 4 replies
- 0 kudos
Resolved! Writing a small pyspark dataframe to a table is taking a very long time
My experience with Databricks pyspark up to this point has always been to execute a SQL query against existing Databricks tables, then write the resulting pyspark dataframe into a new table. For the first time, I am now getting data via an API which ...
- 5883 Views
- 4 replies
- 0 kudos
- 0 kudos
After reading the suggested documentation, I tried using the "Parse nested XML (from_xml and schema_of_xml)". I used this code from the doc: df = spark.createDataFrame([(8, xml_data)], ["number", "payload"]) schema = schema_of_xml(df.select("payload"...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
