- 2893 Views
- 2 replies
- 3 kudos
Job aborted due to stage failure. Relative path in absolute URI
Hello Team we have frequently data bricks job failure with following message , any help would be appreciated Job aborted due to stage failure. Relative path in absolute URI
- 2893 Views
- 2 replies
- 3 kudos
- 3 kudos
@DB_PROD_Molina One of the reasons this error shows up is due to file path/name containing special characters in it. If that is the case, could you rename your file to have the special characters removed.
- 3 kudos
- 6234 Views
- 3 replies
- 3 kudos
Resolved! PARSE_SYNTAX_ERROR creating view from CSV
Hello, if i run this code: %sqlCREATE OR REPLACE VIEW esprosilver.xxx.encuestas_talleresASSELECT * FROM CSV.`abfss://landing@esproanalyticscenterdl.dfs.core.windows.net/oracle-dwh/encuestas_talleres/encuestas_talleres.csv` It creates the view in unit...
- 6234 Views
- 3 replies
- 3 kudos
- 3702 Views
- 0 replies
- 0 kudos
Running dbt macro inside Databricks notebook as a part of Workflow
So.. I have Databricks and I'm using dbt-databricks adapter to run dbt. Everything works fine but for some reasons I need to run some stuff inside of notebook. That still worked fine for some time but after upgrading to dbt-databricks 1.5.5 something...
- 3702 Views
- 0 replies
- 0 kudos
- 3124 Views
- 2 replies
- 1 kudos
Resolved! Serverless clusters not starting
Hello, I am trying to launch a serverless data warehouse, it used to work fine before but for some reason it no longer works. I tried creating a brand new serverless cluster and I get the same result. I am the creator of the clusters, and a workspace...
- 3124 Views
- 2 replies
- 1 kudos
- 1 kudos
For anyone interested in the status page link : https://status.azuredatabricks.net/
- 1 kudos
- 12056 Views
- 6 replies
- 3 kudos
Resolved! Enabling Adaptive Query Execution and Cost-Based Optimizer in Structured Streaming foreachBatch
Dear Databricks community,I am using Spark Structured Streaming to move data from silver to gold in an ETL fashion. The source stream is the change data feed of a Delta table in silver. The streaming dataframe is transformed and joined with a couple ...
- 12056 Views
- 6 replies
- 3 kudos
- 3 kudos
It's not recommended to have AQE on a Streaming query for the same reason you shared in the description. It has been documented here
- 3 kudos
- 3500 Views
- 1 replies
- 2 kudos
Is there a way to avoid using EBS drives on workers with local NVMe SSD?
The Databricks on AWS docs claim that 30G + 150G EBS drives are mounter to every node by default. But if I use instance type like r5d.2xlarge, it already has local disk so I want to avoid mounting the 150G EBS drive to it. Is there a way to do it?We ...
- 3500 Views
- 1 replies
- 2 kudos
- 2 kudos
Hey Ivan, did you find a way to do this?
- 2 kudos
- 2618 Views
- 2 replies
- 0 kudos
Work around for The method `pd.groupby.GroupBy.prod()` is not implemented yet.
0I have a database with two columns: name (str) and probability (float).I am running this command:df[['name','probability']].groupby('name').prod()on a Databricks (runtime 7.3) notebook and df is a koalas dataframe.The error I get is:PandasNotImpleme...
- 2618 Views
- 2 replies
- 0 kudos
- 2638 Views
- 0 replies
- 0 kudos
java.time.LocalDate exception when a date column is used with "IN" operator in replace where clause
Hi, we recently made an upgrade to our Databricks warehouse, transitioning from SQL Classic to SQL PRO. However, we encountered the following error message when attempting to execute the "INSERT INTO" table query with a "REPLACE WHERE" predicate that...
- 2638 Views
- 0 replies
- 0 kudos
- 2257 Views
- 1 replies
- 0 kudos
Databricks-connect VSCode debugging pandas_api not working
HiI am using the databricks extension on VSCode and am running against an issue since two days, prior it worked fine. I receive an error when I want to use Pandas-on-Spark during debugging.from databricks.connect import DatabricksSession spark = Data...
- 2257 Views
- 1 replies
- 0 kudos
- 0 kudos
Additional info:It seems that the issue comes from the 1.1.0 version of the databricks extension in VSCode.Downgrading to 1.0.0 solves my issue.
- 0 kudos
- 5095 Views
- 1 replies
- 2 kudos
SQL expr undefined function 'LEN'
Getting this error message on our production cluster when I run a notebook that uses the SQL expr function that call the LEN() funciton example code:df = df.withColumn("POL", expr("CASE WHEN SRC_SYSTEM = 'X' THEN CONCAT('08' , SUBSTRING(POL, 3, LEN(P...
- 5095 Views
- 1 replies
- 2 kudos
- 2 kudos
@PhillT There's no "LEN" function. You should use "LENGTH" instead.https://spark.apache.org/docs/2.3.0/api/sql/index.html#length
- 2 kudos
- 6880 Views
- 3 replies
- 5 kudos
Resolved! Z-Ordering Timestamp Column
Hi,I've large Delta Table for IoT data for over 10K different sensors with timestamp, sensor name and value columns at 1 second precision.Query pattern is usually random 5-100 sensors at a time. But typically involves specific year/month/day interval...
- 6880 Views
- 3 replies
- 5 kudos
- 5 kudos
@numersoz did you z-order on the timestamp column or on less granular columns, like Year, Month, or Day. timestamp column is very granular (high cardinality) since it also includes hour, minute, second...
- 5 kudos
- 1697 Views
- 1 replies
- 0 kudos
Databricks intellisence adds a second aliad
Hi all, A couple of weeks ago I noticed that whenever I use the intellisense to autocomplete my column names in the sql editor, it adds a second alias. For example if i have a table (Table1) and I alias it in my query (i.e. Table1 as a) if I use the...
- 1697 Views
- 1 replies
- 0 kudos
- 0 kudos
And how do I fix it?
- 0 kudos
- 712 Views
- 0 replies
- 0 kudos
ava.net.SocketTimeoutException - ReadTimeOut
Databricks notebook is connecting ADLS Gen2 using Service principal authentication and the setup is working fine. The notebook is able to read/write files to ADLS Gen2. However, occasionally, we are seeing below error in the production environment:ja...
- 712 Views
- 0 replies
- 0 kudos
- 5612 Views
- 0 replies
- 0 kudos
How to abort/fail a notebook in databricks (and in datafactory)
Hello everyone!I have a pipeline in Data Factory full of notebooks. In one of the notebooks I have the following (inside a function take writes a query to a table):"try:df = spark.sql(query)df.write.format(format).mode(method).saveAsTable(table)excep...
- 5612 Views
- 0 replies
- 0 kudos
- 5406 Views
- 4 replies
- 0 kudos
Why can't I query a table from a cluster, but can query from another cluster in the same workspace
I have two clusters A, B under the same azure databricks workspace. Under cluster A, inside my notebook, I tried to query a table: `SELECT * FROM some_table LIMIT 5`. It shows some permission errors. Under cluster B, if I run the same sql query, it ...
- 5406 Views
- 4 replies
- 0 kudos
- 0 kudos
Check for any spark config or init script differences in the two clusters.
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
