cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Faisal
by • Contributor
  • 8567 Views
  • 2 replies
  • 1 kudos

Error while creating delta table with partitions

Hi All,I am unable to create delta table with partitioning option, can someone please correct me what I am missing and help me with updated query  CREATE OR REPLACE TABLE invoice USING DELTA PARTITION BY (year(shp_dt), month(shp_dt)) LOCATION '/ta...

Data Engineering
Reply
  • 8567 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Emil_Kaminski
Contributor
  • 1 kudos

@Retired_mod Hi. Is that not exactly what I suggested before? Sorry for stupid questions, but I am learning rules or earning kudos and getting solutions approved, therefore suggestions from your end would be appreciated. Thank you.

  • 1 kudos
1 More Replies
hold_my_samosa
by • New Contributor II
  • 7659 Views
  • 1 replies
  • 0 kudos

Delta Partition File on Azure ADLS Gen2 Migration

Hello,I am working on a migration project and I am facing issue while migrating delta tables from Azure ADLS Gen1 to Gen2.So, as per the Microsoft migration pre-requisites:File or directory names with only spaces or tabs, ending with a ., containing ...

Data Engineering
azure
datalake
delta
dtabricks
Reply
  • 7659 Views
  • 1 replies
  • 0 kudos
View Replies
BWong
by • New Contributor III
  • 7957 Views
  • 8 replies
  • 6 kudos

Resolved! Cannot spin up a cluster

HiWhen I try to spin up a cluster, it gives me a bootstrap timeout error{ "reason": { "code": "BOOTSTRAP_TIMEOUT", "parameters": { "databricks_error_message": "[id: InstanceId(i-00b2b7acdd82e5fde), status: INSTANCE_INITIALIZING, workerEnv...

Data Engineering
Reply
  • 7957 Views
  • 8 replies
  • 6 kudos
View Replies
Latest Reply
BWong
New Contributor III
  • 6 kudos

Thanks guys. It's indeed a network issue on the AWS side. It's resolved now

  • 6 kudos
7 More Replies
geertvanhove
by • New Contributor III
  • 6181 Views
  • 3 replies
  • 0 kudos

transform a dataframe column as concatenated string

Hello,I have a single column dataframe and I want to transform the content into a stringEG df=abcdefxyzToabc, def, xyz Thanks

Data Engineering
Reply
  • 6181 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
geertvanhove
New Contributor III
  • 0 kudos

sure: %pythonfrom pyspark.sql.functions import from_json, col, concat_wsfrom pyspark.sql.types import *schema = StructType([StructField('meterDateTime', StringType(), True), StructField('meterId', LongType(), True), StructField('meteringState', Strin...

  • 0 kudos
2 More Replies
Daniel3
by • New Contributor II
  • 10580 Views
  • 2 replies
  • 0 kudos

Resolved! How to use the variable haiving set of values in a spark.sql?

Hi, I have a set of values to be searched from a table, for which i was trying to assign them to a variable first and then trying to use the variable in spark.sql, but i'm unable to fetch the records. Please see the image attached and correct my code...

Data Engineering
Reply
  • 10580 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
brockb
Databricks Employee
  • 0 kudos

Hi, One way to address the example provided in your screenshot is by using a combination of a python f-string and a Common Table Expression like shown below. This is assuming that in reality the two tables are different unlike in the provided screens...

  • 0 kudos
1 More Replies
erigaud
by • Honored Contributor
  • 2066 Views
  • 3 replies
  • 1 kudos

Incorrect dropped rows count in DLT Event log

Hello, I'm using a DLT pipeline with expectationsexpect_or_drop(...) To test it, I added files that contain records that should be dropped, and indeed when running the pipeline I can see some rows were dropped.However when looking at the DLT Event lo...

Data Engineering
Reply
  • 2066 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Priyanka_Biswas
Databricks Employee
  • 1 kudos

Hello @erigaud  The issue appears to be related to the details.flow_progress.data_quality.dropped_records field always being 0, despite records being dropped. This might be because the expect_or_drop operator isn't updating the dropped_records field ...

  • 1 kudos
2 More Replies
ekar-databricks
by • New Contributor II
  • 10364 Views
  • 3 replies
  • 0 kudos

Bigquery - Databricks integration issue.

I am trying to get the Bigquery data to Databricks using Notebooks. Following the steps based on this https://docs.databricks.com/external-data/bigquery.html. I believe I am making some mistake with this step and getting the below error. I tried givi...

image
Data Engineering
Reply
  • 10364 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Wundermobility
New Contributor II
  • 0 kudos

Hi!Did you get the problem solved?I am facing the same issueplease guide

  • 0 kudos
2 More Replies
raghav99
by • New Contributor II
  • 4519 Views
  • 1 replies
  • 1 kudos

how to stream change feed from delta table when its schema is changed?

Hi Team,I would like to know how we can continue streaming change data feed from a delta table when its schema is changed ( non-additive schema changes like drop/rename column / schema migration ).I came across schemaTrackingLocation in readStream bu...

Data Engineering
Reply
  • 4519 Views
  • 1 replies
  • 1 kudos
View Replies
Shahfik
by • New Contributor II
  • 2366 Views
  • 2 replies
  • 0 kudos

Converting TSQL datepart(week) to Databricks SQL

Hi!I'm converting an existing TSQL script into Databricks SQL.In TSQL, the below script returns 1select datepart(WEEK,'2022-01-01')In Databricks SQL, then below script returns 52.select date_part('week','2022-01-01') Does Databricks SQL have somethin...

Data Engineering
Reply
  • 2366 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
TimFrazer
New Contributor II
  • 0 kudos

This worked for me.SELECT  your_date_column,  CASE    WHEN DAYOFYEAR(your_date_column) <= 7 THEN 'Week 1'    ELSE 'Week ' || CAST(CEIL((DAYOFYEAR(your_date_column) - 7) / 7.0) + 1 AS STRING)  END AS WeekNumberFROM your_date_table;

  • 0 kudos
1 More Replies
samhollenbach
by • New Contributor III
  • 4439 Views
  • 2 replies
  • 1 kudos

Resolved! DLT AutoLoader S3 Access Denied Using File Notification mode

Hi all,I'm attempting to switch our DLT pipeline using Auto Loader from Directory Listing to File Notification mode, and running in to S3 Access Denied issues with very little detail. I have followed all the instructions here and here to set up File ...

Data Engineering
Auto Loader
Delta Live Tables
File Notification
Reply
  • 4439 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
samhollenbach
New Contributor III
  • 1 kudos

Thanks @Retired_mod, we ended up abandoning this route due to limitations imposed by the Shared compute access mode enforced by DLT's, and opted for a standard Spark Structured Streaming Job (using Kafka) in the end.

  • 1 kudos
1 More Replies
JKR
by • Contributor
  • 4409 Views
  • 2 replies
  • 1 kudos

Resolved! Got Failure: com.databricks.backend.common.rpc.SparkDriverExceptions$ReplFatalException error

Job is scheduled on interactive cluster, and it failed with below error and in the next scheduled run it ran fine. I want to why this error occurred and how can I prevent from occurring this again.How to debug these types of error?   com.databricks.b...

Data Engineering
Reply
  • 4409 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Tharun-Kumar
Databricks Employee
  • 1 kudos

@JKR Could you try setting the configurations below at the cluster level and retry the job?spark.databricks.python.defaultPythonRepl pythonshellspark.databricks.pyspark.py4j.pinnedThread false

  • 1 kudos
1 More Replies
ivanychev
by • Contributor II
  • 2267 Views
  • 2 replies
  • 0 kudos

Mount Workspace to Docker container

Is there a way to mount Workspace folder (WSFS) to the Docker container if I'm using the Databricks Container Services ofr running a general purpose cluster?If I create a cluster without a Docker image, the `!ls` command in Databricks notebook return...

Data Engineering
Docker
Mount
Workspace
Reply
  • 2267 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
User16539034020
Databricks Employee
  • 0 kudos

Hello:Thanks for contacting Databricks Support! I'm afraid that mounting the WSFS directly into a Docker container isn't directly supported. The Databricks workspace is a specialized environment and isn't directly analogous to a regular filesystem. W...

  • 0 kudos
1 More Replies
Smitha1
by • Valued Contributor II
  • 4382 Views
  • 9 replies
  • 3 kudos

Databricks Certified Associate Developer for Apache Spark 3.0

Databricks Certified Associate Developer for Apache Spark 3.0

Data Engineering
Reply
  • 4382 Views
  • 9 replies
  • 3 kudos
View Replies
Latest Reply
Shivam_Patil
New Contributor II
  • 3 kudos

Hey I am looking for sample papers for the above exam other than the one provided by databricks do any one have any idea about it

  • 3 kudos
8 More Replies
abhaigh
by • New Contributor III
  • 10317 Views
  • 1 replies
  • 0 kudos

Resolved! Azure Shared Clusters - P4J Security Exception on non-whitelisted classes

Hi allHaving some fun trying to run a notebook on a shared UC-aware, shared cluster - I keep on running into this error:py4j.security.Py4JSecurityException: Method public static org.apache.spark.sql.SparkSession org.apache.sedona.spark.SedonaContext....

Data Engineering
Reply
  • 10317 Views
  • 1 replies
  • 0 kudos
View Replies
210573
by • New Contributor
  • 9237 Views
  • 4 replies
  • 1 kudos

Internal error. Attach your notebook to a different cluster or restart the current cluster.

Started getting this error while running all the scripts. All the scripts were running fine before. I tried de-attaching and also restart nothing seems to work.Internal error. Attach your notebook to a different cluster or restart the current cluste...

Data Engineering
Reply
  • 9237 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
tieu_quyen
New Contributor II
  • 1 kudos

Hi @210573 (Customer)​ ,I got the same error, tried to restart and create a new cluster but the solution does not work. What I did to fix the issue: Instead of putting in function, break the code out to run line by line. I just want to see where the ...

  • 1 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels