- 3504 Views
- 1 replies
- 0 kudos
Run driver on spot instance
The traditional advice seems to be to run the driver on "on demand", and optionally the workers on spot. And this is indeed what happends if one chooses to run with spot instances in Databricks. But I am interested in what happens if we run with a dr...
- 3504 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for your answer @Retired_mod ! Good overview, and I understand that "driver on-demand and the rest on spot" is a good generall advice. But I am still considering using spot instances for both, and I am left with two concrete questions:1: Can w...
- 0 kudos
- 3849 Views
- 2 replies
- 2 kudos
SQL Warehouse - Table does not support overwrite by expression:
I'm copying data from a foreign catalog using a replace where logic in the target table, this work fine for two other tables. But for a specific one, I keep getting this error:Table does not support overwrite by expression: DeltaTableV2(org.apache.sp...
- 3849 Views
- 2 replies
- 2 kudos
- 2 kudos
Thank you for the checklist @Retired_mod,> Review and validate the replace where expression.I was using dateadd() with a pipeline parameter, dateadd() returns a timestamp, which was being compared against a date column which threw the error.
- 2 kudos
- 8878 Views
- 2 replies
- 1 kudos
Error while creating delta table with partitions
Hi All,I am unable to create delta table with partitioning option, can someone please correct me what I am missing and help me with updated query CREATE OR REPLACE TABLE invoice USING DELTA PARTITION BY (year(shp_dt), month(shp_dt)) LOCATION '/ta...
- 8878 Views
- 2 replies
- 1 kudos
- 1 kudos
@Retired_mod Hi. Is that not exactly what I suggested before? Sorry for stupid questions, but I am learning rules or earning kudos and getting solutions approved, therefore suggestions from your end would be appreciated. Thank you.
- 1 kudos
- 8733 Views
- 1 replies
- 0 kudos
Delta Partition File on Azure ADLS Gen2 Migration
Hello,I am working on a migration project and I am facing issue while migrating delta tables from Azure ADLS Gen1 to Gen2.So, as per the Microsoft migration pre-requisites:File or directory names with only spaces or tabs, ending with a ., containing ...
- 8733 Views
- 1 replies
- 0 kudos
- 8752 Views
- 8 replies
- 6 kudos
Resolved! Cannot spin up a cluster
HiWhen I try to spin up a cluster, it gives me a bootstrap timeout error{ "reason": { "code": "BOOTSTRAP_TIMEOUT", "parameters": { "databricks_error_message": "[id: InstanceId(i-00b2b7acdd82e5fde), status: INSTANCE_INITIALIZING, workerEnv...
- 8752 Views
- 8 replies
- 6 kudos
- 6 kudos
Thanks guys. It's indeed a network issue on the AWS side. It's resolved now
- 6 kudos
- 6760 Views
- 3 replies
- 0 kudos
transform a dataframe column as concatenated string
Hello,I have a single column dataframe and I want to transform the content into a stringEG df=abcdefxyzToabc, def, xyz Thanks
- 6760 Views
- 3 replies
- 0 kudos
- 0 kudos
sure: %pythonfrom pyspark.sql.functions import from_json, col, concat_wsfrom pyspark.sql.types import *schema = StructType([StructField('meterDateTime', StringType(), True), StructField('meterId', LongType(), True), StructField('meteringState', Strin...
- 0 kudos
- 11213 Views
- 2 replies
- 0 kudos
Resolved! How to use the variable haiving set of values in a spark.sql?
Hi, I have a set of values to be searched from a table, for which i was trying to assign them to a variable first and then trying to use the variable in spark.sql, but i'm unable to fetch the records. Please see the image attached and correct my code...
- 11213 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, One way to address the example provided in your screenshot is by using a combination of a python f-string and a Common Table Expression like shown below. This is assuming that in reality the two tables are different unlike in the provided screens...
- 0 kudos
- 2309 Views
- 3 replies
- 1 kudos
Incorrect dropped rows count in DLT Event log
Hello, I'm using a DLT pipeline with expectationsexpect_or_drop(...) To test it, I added files that contain records that should be dropped, and indeed when running the pipeline I can see some rows were dropped.However when looking at the DLT Event lo...
- 2309 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @erigaud The issue appears to be related to the details.flow_progress.data_quality.dropped_records field always being 0, despite records being dropped. This might be because the expect_or_drop operator isn't updating the dropped_records field ...
- 1 kudos
- 11433 Views
- 3 replies
- 0 kudos
Bigquery - Databricks integration issue.
I am trying to get the Bigquery data to Databricks using Notebooks. Following the steps based on this https://docs.databricks.com/external-data/bigquery.html. I believe I am making some mistake with this step and getting the below error. I tried givi...
- 11433 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi!Did you get the problem solved?I am facing the same issueplease guide
- 0 kudos
- 4842 Views
- 1 replies
- 1 kudos
how to stream change feed from delta table when its schema is changed?
Hi Team,I would like to know how we can continue streaming change data feed from a delta table when its schema is changed ( non-additive schema changes like drop/rename column / schema migration ).I came across schemaTrackingLocation in readStream bu...
- 4842 Views
- 1 replies
- 1 kudos
- 2755 Views
- 2 replies
- 0 kudos
Converting TSQL datepart(week) to Databricks SQL
Hi!I'm converting an existing TSQL script into Databricks SQL.In TSQL, the below script returns 1select datepart(WEEK,'2022-01-01')In Databricks SQL, then below script returns 52.select date_part('week','2022-01-01') Does Databricks SQL have somethin...
- 2755 Views
- 2 replies
- 0 kudos
- 0 kudos
This worked for me.SELECT your_date_column, CASE WHEN DAYOFYEAR(your_date_column) <= 7 THEN 'Week 1' ELSE 'Week ' || CAST(CEIL((DAYOFYEAR(your_date_column) - 7) / 7.0) + 1 AS STRING) END AS WeekNumberFROM your_date_table;
- 0 kudos
- 4907 Views
- 2 replies
- 1 kudos
Resolved! DLT AutoLoader S3 Access Denied Using File Notification mode
Hi all,I'm attempting to switch our DLT pipeline using Auto Loader from Directory Listing to File Notification mode, and running in to S3 Access Denied issues with very little detail. I have followed all the instructions here and here to set up File ...
- 4907 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks @Retired_mod, we ended up abandoning this route due to limitations imposed by the Shared compute access mode enforced by DLT's, and opted for a standard Spark Structured Streaming Job (using Kafka) in the end.
- 1 kudos
- 4872 Views
- 2 replies
- 1 kudos
Resolved! Got Failure: com.databricks.backend.common.rpc.SparkDriverExceptions$ReplFatalException error
Job is scheduled on interactive cluster, and it failed with below error and in the next scheduled run it ran fine. I want to why this error occurred and how can I prevent from occurring this again.How to debug these types of error? com.databricks.b...
- 4872 Views
- 2 replies
- 1 kudos
- 1 kudos
@JKR Could you try setting the configurations below at the cluster level and retry the job?spark.databricks.python.defaultPythonRepl pythonshellspark.databricks.pyspark.py4j.pinnedThread false
- 1 kudos
- 2597 Views
- 2 replies
- 0 kudos
Mount Workspace to Docker container
Is there a way to mount Workspace folder (WSFS) to the Docker container if I'm using the Databricks Container Services ofr running a general purpose cluster?If I create a cluster without a Docker image, the `!ls` command in Databricks notebook return...
- 2597 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello:Thanks for contacting Databricks Support! I'm afraid that mounting the WSFS directly into a Docker container isn't directly supported. The Databricks workspace is a specialized environment and isn't directly analogous to a regular filesystem. W...
- 0 kudos
- 4877 Views
- 9 replies
- 3 kudos
Databricks Certified Associate Developer for Apache Spark 3.0
Databricks Certified Associate Developer for Apache Spark 3.0
- 4877 Views
- 9 replies
- 3 kudos
- 3 kudos
Hey I am looking for sample papers for the above exam other than the one provided by databricks do any one have any idea about it
- 3 kudos
| User | Count |
|---|---|
| 1613 | |
| 770 | |
| 349 | |
| 286 | |
| 253 |
