Hi, we recently made an upgrade to our Databricks warehouse, transitioning from SQL Classic to SQL PRO. However, we encountered the following error message when attempting to execute the "INSERT INTO" table query with a "REPLACE WHERE" predicate that...
HiI am using the databricks extension on VSCode and am running against an issue since two days, prior it worked fine. I receive an error when I want to use Pandas-on-Spark during debugging.from databricks.connect import DatabricksSession spark = Data...
Additional info:It seems that the issue comes from the 1.1.0 version of the databricks extension in VSCode.Downgrading to 1.0.0 solves my issue.
Getting this error message on our production cluster when I run a notebook that uses the SQL expr function that call the LEN() funciton example code:df = df.withColumn("POL", expr("CASE WHEN SRC_SYSTEM = 'X' THEN CONCAT('08' , SUBSTRING(POL, 3, LEN(P...
@PhillT There's no "LEN" function. You should use "LENGTH" instead.https://spark.apache.org/docs/2.3.0/api/sql/index.html#length
Hi,I've large Delta Table for IoT data for over 10K different sensors with timestamp, sensor name and value columns at 1 second precision.Query pattern is usually random 5-100 sensors at a time. But typically involves specific year/month/day interval...
@numersoz did you z-order on the timestamp column or on less granular columns, like Year, Month, or Day. timestamp column is very granular (high cardinality) since it also includes hour, minute, second...
Hi all, A couple of weeks ago I noticed that whenever I use the intellisense to autocomplete my column names in the sql editor, it adds a second alias. For example if i have a table (Table1) and I alias it in my query (i.e. Table1 as a) if I use the...
And how do I fix it?
Databricks notebook is connecting ADLS Gen2 using Service principal authentication and the setup is working fine. The notebook is able to read/write files to ADLS Gen2. However, occasionally, we are seeing below error in the production environment:ja...
Hello everyone!I have a pipeline in Data Factory full of notebooks. In one of the notebooks I have the following (inside a function take writes a query to a table):"try:df = spark.sql(query)df.write.format(format).mode(method).saveAsTable(table)excep...
I have two clusters A, B under the same azure databricks workspace. Under cluster A, inside my notebook, I tried to query a table: `SELECT * FROM some_table LIMIT 5`. It shows some permission errors. Under cluster B, if I run the same sql query, it ...
Check for any spark config or init script differences in the two clusters.
Hey team,Need your help.I am trying to run the below python code in a data bricks notebook, which is part of parsing an XML file, exploding the element. This works great for all the other elements with no numbers and elements not starting with a zero...
Have a couple of questions wrt Streaming tables, Kindly help us with this.1)Can we create streaming tables without a DLT pipeline?2)Can we create streaming tables in Databricks SQL?3)What we observe in Streaming tables, it support Kafka and event lo...
HiPlease refer to the document: https://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-create-streaming-table.html#create-streaming-tableI think this should help you answer your questions.
Hi all,I have a postgres database that contains two tables: A and B.Also, I have 2 delta tables, called C and D. My task is to push the data from A to C and B to D - and if something fails, then leave everything as is.With python it is easy. Set up t...
Hi @thomasthomas We haven't heard from you since the last response from @daniel_sahal , and I was checking back to see if her suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful to oth...
How to manage different workspaces’ clusters permissions?
Which {request_object_type} to use for setting permissions for a cluster? "cluster", "clusters", "compute" does not work.
I'm not able to login databricks community edition from last 5 days, getting Invalid email address or password issue even though its correct credentials. i tried to reset password but unable to reset password also. what is the issue. can you guys pl...
im getting mails saying that if issue is fixed then mark one as a accepted solution but unfortunately issue is still not yet fixed , can someone look into it and fix issue please ??
Hi there,Anyone here experienced any problems with formatiing python code inside databricks notebook cells? I'm constantly having this weird problem, both with very short/simple and complex code, in different notebooks. Sometimes it somehow works, an...
that is interesting.Can you check if you you use a version > 11.2. If not you have to install black and tokenize-rt.Can you check if you use a custom pyproject.toml file? It might be related to that.(You also need edit permission on the notebook bu...
Hi Community, I'm using Databricks E2, and need to convert pptx files to pdf files.This can be done in either a python or an R notebook using #LibreofficeTo achieve this I'd have to download LibreOffice; I'm not too sure on how to do that. Would I ha...
I suppose by Libreoffice you mean the sdk, without the frontend?You will have to install the jar as a library on the compute cluster.From that moment on, you can use the classes in your code.If you cannot run the jar from a command line, it might be ...
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 489 | |
| 349 | |
| 287 |
