So.. I have Databricks and I'm using dbt-databricks adapter to run dbt. Everything works fine but for some reasons I need to run some stuff inside of notebook. That still worked fine for some time but after upgrading to dbt-databricks 1.5.5 something...
Hello, I am trying to launch a serverless data warehouse, it used to work fine before but for some reason it no longer works. I tried creating a brand new serverless cluster and I get the same result. I am the creator of the clusters, and a workspace...
For anyone interested in the status page link : https://status.azuredatabricks.net/
Dear Databricks community,I am using Spark Structured Streaming to move data from silver to gold in an ETL fashion. The source stream is the change data feed of a Delta table in silver. The streaming dataframe is transformed and joined with a couple ...
It's not recommended to have AQE on a Streaming query for the same reason you shared in the description. It has been documented here
The Databricks on AWS docs claim that 30G + 150G EBS drives are mounter to every node by default. But if I use instance type like r5d.2xlarge, it already has local disk so I want to avoid mounting the 150G EBS drive to it. Is there a way to do it?We ...
Hey Ivan, did you find a way to do this?
0I have a database with two columns: name (str) and probability (float).I am running this command:df[['name','probability']].groupby('name').prod()on a Databricks (runtime 7.3) notebook and df is a koalas dataframe.The error I get is:PandasNotImpleme...
Hi, we recently made an upgrade to our Databricks warehouse, transitioning from SQL Classic to SQL PRO. However, we encountered the following error message when attempting to execute the "INSERT INTO" table query with a "REPLACE WHERE" predicate that...
HiI am using the databricks extension on VSCode and am running against an issue since two days, prior it worked fine. I receive an error when I want to use Pandas-on-Spark during debugging.from databricks.connect import DatabricksSession spark = Data...
Additional info:It seems that the issue comes from the 1.1.0 version of the databricks extension in VSCode.Downgrading to 1.0.0 solves my issue.
Getting this error message on our production cluster when I run a notebook that uses the SQL expr function that call the LEN() funciton example code:df = df.withColumn("POL", expr("CASE WHEN SRC_SYSTEM = 'X' THEN CONCAT('08' , SUBSTRING(POL, 3, LEN(P...
@PhillT There's no "LEN" function. You should use "LENGTH" instead.https://spark.apache.org/docs/2.3.0/api/sql/index.html#length
Hi,I've large Delta Table for IoT data for over 10K different sensors with timestamp, sensor name and value columns at 1 second precision.Query pattern is usually random 5-100 sensors at a time. But typically involves specific year/month/day interval...
@numersoz did you z-order on the timestamp column or on less granular columns, like Year, Month, or Day. timestamp column is very granular (high cardinality) since it also includes hour, minute, second...
Hi all, A couple of weeks ago I noticed that whenever I use the intellisense to autocomplete my column names in the sql editor, it adds a second alias. For example if i have a table (Table1) and I alias it in my query (i.e. Table1 as a) if I use the...
And how do I fix it?
Databricks notebook is connecting ADLS Gen2 using Service principal authentication and the setup is working fine. The notebook is able to read/write files to ADLS Gen2. However, occasionally, we are seeing below error in the production environment:ja...
Hello everyone!I have a pipeline in Data Factory full of notebooks. In one of the notebooks I have the following (inside a function take writes a query to a table):"try:df = spark.sql(query)df.write.format(format).mode(method).saveAsTable(table)excep...
I have two clusters A, B under the same azure databricks workspace. Under cluster A, inside my notebook, I tried to query a table: `SELECT * FROM some_table LIMIT 5`. It shows some permission errors. Under cluster B, if I run the same sql query, it ...
Check for any spark config or init script differences in the two clusters.
Hey team,Need your help.I am trying to run the below python code in a data bricks notebook, which is part of parsing an XML file, exploding the element. This works great for all the other elements with no numbers and elements not starting with a zero...
Have a couple of questions wrt Streaming tables, Kindly help us with this.1)Can we create streaming tables without a DLT pipeline?2)Can we create streaming tables in Databricks SQL?3)What we observe in Streaming tables, it support Kafka and event lo...
HiPlease refer to the document: https://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-create-streaming-table.html#create-streaming-tableI think this should help you answer your questions.
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
