cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

646901
by New Contributor II
  • 968 Views
  • 1 replies
  • 1 kudos

What is the local-ssd used for in databricks?

What is the use-case for local-ssd's in databricks clusters? I noticed some clusters have many Tb's worth and some have no local ssd's.What are the pro's and con's of changing the disk size bigger and smaller? According to the docs:> The disk cache i...

  • 968 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @646901 , Local SSDs in Databricks clusters serve several purposes and can impact performance, cost, and scalability.    Let’s delve into the details:   Use Cases for Local SSDs: Low-Latency Storage: Local SSDs provide fast, low-latency storage th...

  • 1 kudos
nyck33
by New Contributor II
  • 2295 Views
  • 1 replies
  • 0 kudos

snowflake python connector import error

```--------------------------------------------------------------------------- ImportError Traceback (most recent call last) File <command-1961894174266859>:1 ----> 1 con = snowflake.connector.connect( 2 user=USER, 3 password=SNOWSQL_PWD, 4 account=A...

  • 2295 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @nyck33 , It appears that you’re encountering an ImportError related to the snowflake-connector-python package.    Let’s troubleshoot this issue:   Module Not Found: The error message indicates that Python cannot find the library snowflake-connect...

  • 0 kudos
AndyM
by New Contributor II
  • 7317 Views
  • 1 replies
  • 0 kudos

DAB wheel installation job fails, user error Library from /Workspace not allowed

Hi Community!I am getting started with DABs and just recently ran into a following error after deployment trying to run my bundle that has a wheel installation job. Error: failed to reach TERMINATED or SKIPPED, got INTERNAL_ERROR: Task main_task fail...

  • 7317 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @AndyM ,  The error message you encountered indicates that the library installation failed due to a restriction related to the Unity Catalog. Let’s break down the issue and explore the solution: Unity Catalog and Workspace: Unity Catalog is a dat...

  • 0 kudos
Shahfik
by New Contributor II
  • 1317 Views
  • 2 replies
  • 0 kudos

Converting TSQL datepart(week) to Databricks SQL

Hi!I'm converting an existing TSQL script into Databricks SQL.In TSQL, the below script returns 1select datepart(WEEK,'2022-01-01')In Databricks SQL, then below script returns 52.select date_part('week','2022-01-01') Does Databricks SQL have somethin...

  • 1317 Views
  • 2 replies
  • 0 kudos
Latest Reply
TimFrazer
New Contributor II
  • 0 kudos

This worked for me.SELECT  your_date_column,  CASE    WHEN DAYOFYEAR(your_date_column) <= 7 THEN 'Week 1'    ELSE 'Week ' || CAST(CEIL((DAYOFYEAR(your_date_column) - 7) / 7.0) + 1 AS STRING)  END AS WeekNumberFROM your_date_table;

  • 0 kudos
1 More Replies
samhollenbach
by New Contributor III
  • 2715 Views
  • 4 replies
  • 1 kudos

Resolved! DLT AutoLoader S3 Access Denied Using File Notification mode

Hi all,I'm attempting to switch our DLT pipeline using Auto Loader from Directory Listing to File Notification mode, and running in to S3 Access Denied issues with very little detail. I have followed all the instructions here and here to set up File ...

Data Engineering
Auto Loader
Delta Live Tables
File Notification
  • 2715 Views
  • 4 replies
  • 1 kudos
Latest Reply
samhollenbach
New Contributor III
  • 1 kudos

Thanks @Kaniz_Fatma, we ended up abandoning this route due to limitations imposed by the Shared compute access mode enforced by DLT's, and opted for a standard Spark Structured Streaming Job (using Kafka) in the end.

  • 1 kudos
3 More Replies
JKR
by Contributor
  • 2584 Views
  • 2 replies
  • 1 kudos

Resolved! Got Failure: com.databricks.backend.common.rpc.SparkDriverExceptions$ReplFatalException error

Job is scheduled on interactive cluster, and it failed with below error and in the next scheduled run it ran fine. I want to why this error occurred and how can I prevent from occurring this again.How to debug these types of error?   com.databricks.b...

  • 2584 Views
  • 2 replies
  • 1 kudos
Latest Reply
Tharun-Kumar
Honored Contributor II
  • 1 kudos

@JKR Could you try setting the configurations below at the cluster level and retry the job?spark.databricks.python.defaultPythonRepl pythonshellspark.databricks.pyspark.py4j.pinnedThread false

  • 1 kudos
1 More Replies
ivanychev
by Contributor
  • 1439 Views
  • 2 replies
  • 0 kudos

Mount Workspace to Docker container

Is there a way to mount Workspace folder (WSFS) to the Docker container if I'm using the Databricks Container Services ofr running a general purpose cluster?If I create a cluster without a Docker image, the `!ls` command in Databricks notebook return...

Data Engineering
Docker
Mount
Workspace
  • 1439 Views
  • 2 replies
  • 0 kudos
Latest Reply
User16539034020
Contributor II
  • 0 kudos

Hello:Thanks for contacting Databricks Support! I'm afraid that mounting the WSFS directly into a Docker container isn't directly supported. The Databricks workspace is a specialized environment and isn't directly analogous to a regular filesystem. W...

  • 0 kudos
1 More Replies
Smitha1
by Valued Contributor II
  • 2833 Views
  • 9 replies
  • 3 kudos

Databricks Certified Associate Developer for Apache Spark 3.0

Databricks Certified Associate Developer for Apache Spark 3.0

  • 2833 Views
  • 9 replies
  • 3 kudos
Latest Reply
Shivam_Patil
New Contributor II
  • 3 kudos

Hey I am looking for sample papers for the above exam other than the one provided by databricks do any one have any idea about it

  • 3 kudos
8 More Replies
abhaigh
by New Contributor III
  • 5332 Views
  • 2 replies
  • 0 kudos

Resolved! Azure Shared Clusters - P4J Security Exception on non-whitelisted classes

Hi allHaving some fun trying to run a notebook on a shared UC-aware, shared cluster - I keep on running into this error:py4j.security.Py4JSecurityException: Method public static org.apache.spark.sql.SparkSession org.apache.sedona.spark.SedonaContext....

  • 5332 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @abhaigh , Certainly! It seems you’re encountering a security issue related to the Py4J framework when running your notebook on a shared cluster.    Let’s address this and explore potential solutions:   Py4J Security Exception: The error message y...

  • 0 kudos
1 More Replies
210573
by New Contributor
  • 7195 Views
  • 4 replies
  • 1 kudos

Internal error. Attach your notebook to a different cluster or restart the current cluster.

Started getting this error while running all the scripts. All the scripts were running fine before. I tried de-attaching and also restart nothing seems to work.Internal error. Attach your notebook to a different cluster or restart the current cluste...

  • 7195 Views
  • 4 replies
  • 1 kudos
Latest Reply
tieu_quyen
New Contributor II
  • 1 kudos

Hi @210573 (Customer)​ ,I got the same error, tried to restart and create a new cluster but the solution does not work. What I did to fix the issue: Instead of putting in function, break the code out to run line by line. I just want to see where the ...

  • 1 kudos
3 More Replies
TaBorjaTa
by New Contributor II
  • 7687 Views
  • 1 replies
  • 2 kudos

Pytest imports of sibling modules when using Databricks for VSCode

Hello all, I am following the Databrick's documentation on unit testing found here: Run tests with pytest for the Databricks extension for Visual Studio Code - Azure Databricks | Microsoft LearnHowever, when taking it a step further I get an ImportEr...

Data Engineering
pytest
VSCode
  • 7687 Views
  • 1 replies
  • 2 kudos
Latest Reply
Trifa
New Contributor II
  • 2 kudos

HelloImport errors happen often with Pytest. To Debug this error you can add this in your "test_myfunction_test.py":import sys # printing all directories for # interpreter to search sys.pathsys.path is a built-in variable within the sys module. I...

  • 2 kudos
AFox
by Contributor
  • 2928 Views
  • 7 replies
  • 0 kudos

databricks-connector: Error: Cluster MASKED is in unexpected state Pending.

Is there a way to make databricks-connector wait for cluster to be running?Details:databricks-connector==13.1.0 and the python minor version of cluster and environment are both 3.10If the cluster is not running this will start it, but any commands af...

  • 2928 Views
  • 7 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @AFox , I want to express my gratitude for your effort in selecting the most suitable solution. It's great to hear that your query has been successfully resolved. Thank you for your contribution. 

  • 0 kudos
6 More Replies
AndyAtINX
by New Contributor III
  • 2164 Views
  • 4 replies
  • 1 kudos

Resolved! Error inviting user to workspace "Failed to add user: A user with email ... or username ... in different cases already exist in the account"

We have 3 workspaces - 1 old version in one AWS account, 2 latest versions in another.We are PAYG full edition, not using SSO.Our admins (existing DBX users in the `admins` group) can invite new users via the Admin Console from the 1 old and 1 new wo...

  • 2164 Views
  • 4 replies
  • 1 kudos
Latest Reply
Schneider-Elect
New Contributor II
  • 1 kudos

We are facing same issue, We are on azure. @AndyAtINX you mean if user exist in workspace with abc@gmail.com we should add the user in workspace2 with abc@gmail.com not ABC@GMAIL.COM. if this the case we tried this and its not working for us.

  • 1 kudos
3 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels