cancel
Showing results for 
Search instead for 
Did you mean: 
Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the conversation to deepen your understanding and maximize your usage of the Databricks platform.
cancel
Showing results for 
Search instead for 
Did you mean: 

Browse the Community

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies with...

9222 Posts

Data Governance

Join discussions on data governance practices, compliance, and security within the Databricks Commun...

396 Posts

Generative AI

Explore discussions on generative artificial intelligence techniques and applications within the Dat...

104 Posts

Machine Learning

Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithm...

829 Posts

Warehousing & Analytics

Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Communi...

490 Posts

Activity in Databricks Platform Discussions

alvaro_databric
by > New Contributor III
  • 1797 Views
  • 2 replies
  • 2 kudos

How to access hard disk attached to cluster?

Hi,I am using the VM family Lasv3, which incorporate a NVMe SSD. I would like to take advantage of this huge amount of space but I cannot find where this disk is mounted. Does someone know where this disk is mounted and if it can be used as local dri...

  • 1797 Views
  • 2 replies
  • 2 kudos
Latest Reply
JosiahJohnston
New Contributor II
  • 2 kudos

Great question; I've been trying to hunt that down also. `/local_disk0` looks like a good candidate, but it has restricted access and I can't confirm or use.Would love to learn a solution someday. This is a big need for hybrid workflows & libraries c...

  • 2 kudos
1 More Replies
Anand4
by > New Contributor
  • 7 Views
  • 1 replies
  • 0 kudos

Delta Table - Partitioning

Created a streaming job with delta table as a target.  The table did not have a partition when created earlier, however i would like to add an existing column as a partition column.I am getting the following error.com.databricks.sql.transaction.tahoe...

  • 7 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @Anand4,Delta Lake does not support altering the partitioning of an existing table directly. Therefore, the way forward is to rewrite the entire table with the new partition column

  • 0 kudos
597581
by > New Contributor
  • 16 Views
  • 2 replies
  • 0 kudos

Run selected text shortcut not working

The keyboard shortcut to run selected text (ctrl + shift + enter) has not been working for me since yesterday (10/31/24). Instead of running the selected text, databricks notebooks are treating it like shift + enter and running the entire cell. I hav...

  • 16 Views
  • 2 replies
  • 0 kudos
Latest Reply
Sean_dbx
Visitor
  • 0 kudos

I have the same issue.

  • 0 kudos
1 More Replies
mvmiller
by > New Contributor III
  • 5106 Views
  • 4 replies
  • 2 kudos

Troubleshooting _handle_rpc_error GRPC Error

I am trying to run the following chunk of code in the cell of a Databricks notebook (using Databricks runtime 14.3 LTS, Apache spark 3.5.0, scala 2.12): spark.sql("CREATE OR REPLACE table sample_catalog.sample_schema.sample_table_tmp AS SELECT * FROM...

  • 5106 Views
  • 4 replies
  • 2 kudos
Latest Reply
kunalmishra9
New Contributor III
  • 2 kudos

Following. Also having this issue, but within the context of pivoting a DF, then aggregating by *

  • 2 kudos
3 More Replies
AndreLIUfr
by > New Contributor
  • 506 Views
  • 12 replies
  • 9 kudos

community edition : "User is not a member of this workspace"

Something very strange has happened. When trying to login to my databricks community edition account. I'm getting the email with my verification code. but after entring that code, I'm getting the error message : "User is not a member of this workspac...

  • 506 Views
  • 12 replies
  • 9 kudos
Latest Reply
Krishnapriya19
  • 9 kudos

Hi Walter,Am also facing the same issue...am actively using my work space on regular basis...i had already dropped a mail to feedback@databricks.com and help@databricks.com..but didn'tget any reply till now...I want my notebooks back..please suggest ...

  • 9 kudos
11 More Replies
rcostanza
by > New Contributor II
  • 126 Views
  • 3 replies
  • 0 kudos

Resolved! Changing git's author field when committing through Databricks

I have a git folder to a Bitbucket repo. Whenever I commit something, the commit uses my Bitbucket username (the unique name) in the field author, making it less readable when I'm reading a list of commits.For example, commits end up like this: commi...

  • 126 Views
  • 3 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Oh, sorry it is internal. If you have an account executive you work with, they can help you with adding the details of the company etc.

  • 0 kudos
2 More Replies
ChristianRRL
by > Contributor III
  • 141 Views
  • 7 replies
  • 3 kudos

DLT Potential Bug: File Reprocessing Issue with "cloudFiles.allowOverwrites": "true"

Hi there, I ran into a peculiar case and I'm wondering if anyone else has run into this and can offer an explanation. We have a DLT process to pull CSV files from a landing location and insert (append) them into target tables. We have the setting "cl...

  • 141 Views
  • 7 replies
  • 3 kudos
Latest Reply
NandiniN
Databricks Employee
  • 3 kudos

Apologies, that could be the internet or networking issue. So, in DLT you will be able to change the DBR but will have to use custom image, it may be tricky if you have not done it earlier.  By default, photon will be used in serverelss. It may be a ...

  • 3 kudos
6 More Replies
DRock
by > New Contributor
  • 112 Views
  • 4 replies
  • 0 kudos

ODBC data source to connect to a Databricks catalog.database via MS Access Not Working

When using an ODBC data source to connect to a Databricks catalog database via Microsoft Access, the tables are not listing/appearing in the MS Access database for selection.However, when using the same ODBC data source to connect to Microsoft Excel,...

  • 112 Views
  • 4 replies
  • 0 kudos
Latest Reply
DRock
New Contributor
  • 0 kudos

The Trace log info is below.  Also, I find it interesting that the log is making reference to hive_metastore (see below), which is not the catalog explicitly stated via the Server Side Properties.    MSACCESS 3818-164c ENTER SQLAllocEnvHENV * 0x00000...

  • 0 kudos
3 More Replies
FabianGutierrez
by > New Contributor II
  • 136 Views
  • 3 replies
  • 1 kudos

Issue with DAB (Databricks Asset Bundle) requesting Terraform files

Hi community,Since recently (2 days ago) we have been receiving the following error when validating and deploying our DAB (Databricks Asset Bundle):"Error: error downloading Terraform: Get "https://releases.hashicorp.com/terraform/1.5.5/index.json": ...

  • 136 Views
  • 3 replies
  • 1 kudos
Latest Reply
FabianGutierrez
New Contributor II
  • 1 kudos

Some update, we cannot get the FW cleared on time so we need to go for the offline optiion, that is download everything form Terraform and DB templated but it is not as clear or intuitive as describe. Using their Container unfortunately not a option ...

  • 1 kudos
2 More Replies
Pingleinferyx
by > New Contributor
  • 237 Views
  • 1 replies
  • 0 kudos

jdbc integration returning header as data for read operation

package com.example.databricks; import org.apache.spark.sql.Dataset;import org.apache.spark.sql.Row;import org.apache.spark.sql.SparkSession; public class DatabricksJDBCApp {     public static void main(String[] args) {        // Initialize Spark Ses...

  • 237 Views
  • 1 replies
  • 0 kudos
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

Can you please try it this way: package com.example.databricks; import org.apache.spark.sql.Dataset; import org.apache.spark.sql.Row; import org.apache.spark.sql.SparkSession; public class DatabricksJDBCApp { public static void main(String[] a...

  • 0 kudos
pjv
by > New Contributor III
  • 248 Views
  • 1 replies
  • 0 kudos

How to ensure pyspark udf execution is distributed across worker nodes

Hi,I have the following databricks notebook code defined: pyspark_dataframe = create_pyspark_dataframe(some input data)MyUDF = udf(myfunc, StringType())pyspark_dataframe = pyspark_dataframe.withColumn('UDFOutput', DownloadUDF(input data columns))outp...

  • 248 Views
  • 1 replies
  • 0 kudos
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

@pjv Can you please try the following, you'll basically want to have more than a single partition: from pyspark.sql import SparkSession from pyspark.sql.functions import udf from pyspark.sql.types import StringType # Initialize Spark session (if not...

  • 0 kudos
VasuKumarT
by > New Contributor
  • 109 Views
  • 1 replies
  • 0 kudos

Larger than Max error :

Hi,We are trying to pass the keys to decrypt a file and receiving the above error as in attached.Please help in case we need to change and configuration or set any options to avoid this error. Thanks. Vasu 

VasuKumarT_0-1728473121954.png
  • 109 Views
  • 1 replies
  • 0 kudos
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

@VasuKumarT can you provide some more details or context? Feel free to replace sensitive data. Where are you getting this? How are you passing the keys to decrypt a file? Is there a move comprehensive stacktrace apart from this message in the image?

  • 0 kudos
Maatari
by > New Contributor III
  • 39 Views
  • 1 replies
  • 0 kudos

What is the behaviour of starting version with spark structured streaming ?

Looking into the followinghttps://docs.databricks.com/en/structured-streaming/delta-lake.html#specify-initial-positionI am unclear as to what is the exact difference (if any) between "startingVersion: The Delta Lake version to start from. Databricks ...

  • 39 Views
  • 1 replies
  • 0 kudos
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

The key difference is that not specifying startingVersion will include the current snapshot of the table in the stream, while setting startingVersion to latest will only process new changes from that point onward.

  • 0 kudos
ChingizK
by > New Contributor III
  • 1922 Views
  • 1 replies
  • 1 kudos

Hyperopt Error: There are no evaluation tasks, cannot return argmin of task losses.

The trials succeed when the cell in the notebook is executed manually:However, the same process fails when executed as a Workflow: The error simply says that there's an issue with the objective function. However how can that be the case if I'm able t...

01.png 02.png
Data Engineering
hyperopt
Workflows
  • 1922 Views
  • 1 replies
  • 1 kudos
Latest Reply
honj
Visitor
  • 1 kudos

I've run in to the same issue using SparkTrials.Runs fine manually.Runs using only Trials in the workflow.Get this error when using SparkTrials.I've tried dropping parallelism right down, making sure there's only one experiment on that cluster.Did yo...

  • 1 kudos
AndrewHess
by > New Contributor
  • 88 Views
  • 4 replies
  • 0 kudos

Unity Group management, Group: Manager role

We would like to have the ability to assign an individual and/or group to the "Group: Manager" role, providing them with the ability to add/remove users without the need to be an account or workspace administrator.  Ideally this would be an option fo...

AndrewHess_0-1730378933657.png
  • 88 Views
  • 4 replies
  • 0 kudos
Latest Reply
AndrewHess
New Contributor
  • 0 kudos

thanks @NandiniN , we have looked through that documentation and still have not been able to get anything to work without the user also being an account or workspace admin.  The way i'm interpreting the documentation (screenshot) is the API currently...

  • 0 kudos
3 More Replies