cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Rajesh_M
by • New Contributor III
  • 4092 Views
  • 3 replies
  • 6 kudos

Resolved! Unable to change the index, when writing to a Azure SQL Data Warehouse

Hi,I have some data in a spark data frame and I am trying to write it to a table in Azure SQL Data Warehouse. If I use df.write.mode(saveMode="overwrite") I get this error:com.microsoft.sqlserver.jdbc.SQLServerException: The statement failed. Column ...

Data Engineering
Reply
  • 4092 Views
  • 3 replies
  • 6 kudos
View Replies
Latest Reply
Rajesh_M
New Contributor III
  • 6 kudos

Thanks @Hubert Dudek​ . Do you know if there is a way to run a create table statement on Azure Synapse/Azure SQL Datawarehouse from Databricks?

  • 6 kudos
2 More Replies
Jeff1
by • Contributor II
  • 4323 Views
  • 5 replies
  • 5 kudos

Resolved! Recommended database when using R in databricks

I'm new to integrating the sparklyr / R interface in databricks. In particular it appears that sparklyr and R commands and functions are dependent upon the type of dataframe one is working with (hive, Spark R etc). Is there a recommend best practice...

Data Engineering
Reply
  • 4323 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 5 kudos

Recommended is delta format in data lake. Here is code example https://docs.databricks.com/delta/quick-start.html#language-r

  • 5 kudos
4 More Replies
anmol_deep
by • New Contributor III
  • 7021 Views
  • 8 replies
  • 10 kudos

Resolved! dbutils.fs.mv taking too long with delta table

I have a folder which contains multiple delta tables and some parquet tables. I want to move that folder to another path. When I use dbutils.fs.mv(), it takes an absurd amount of time.

Data Engineering
Reply
  • 7021 Views
  • 8 replies
  • 10 kudos
View Replies
Latest Reply
anmol_deep
New Contributor III
  • 10 kudos

Thanks @Hubert Dudek​ !Actually I want to delete the folder. But when I try to do that, I get this error: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: This operation is...

  • 10 kudos
7 More Replies
Rk2
by • New Contributor II
  • 624 Views
  • 0 replies
  • 1 kudos

Hi folks, I am trying a particular use case, where I need to schedule a run of three different notebooks(pyspark,sql code) in sequence ​I need to use ...

Hi folks, I am trying a particular use case, where I need to schedule a run of three different notebooks(pyspark,sql code) in sequence ​I need to use date field as a common parameter in all three (date is part of the sql query in each nb where clause...

Data Engineering
Reply
  • 624 Views
  • 0 replies
  • 1 kudos
lizou
by • Contributor II
  • 5638 Views
  • 6 replies
  • 3 kudos

Resolved! Random error related to dynamic variables in SQL - at least one column must be specified (shallowclone table)

Environment: AzureA data transformation template that take s table name as variable. The variable is set in separate notebook %run ../_configShallow clone is used to generate a staging table with exact same columns of target table.A random error occ...

image
Data Engineering
Reply
  • 5638 Views
  • 6 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

@lizou​ - That's great! I'm so glad. Would you be happy to share your solution with us and then mark the answer as best? That will help others find the solution more quickly.

  • 3 kudos
5 More Replies
lizou
by • Contributor II
  • 3656 Views
  • 2 replies
  • 2 kudos

Resolved! How to find the identity column seed value?

How to find the identity column seed value? A seed value is required when we need specifically like start generating new values from a number (most likely we need to keep the original key values when data is reloaded from another source, and any new ...

Data Engineering
Reply
  • 3656 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
lizou
Contributor II
  • 2 kudos

found it, thanks!of course, it will be nice to have a sql function available to query the value.example\"delta.identity.start\":984888,\"delta.identity.highWaterMark\":1004409,\"comment\":\"identity\",\"delta.identity.step\":1}

  • 2 kudos
1 More Replies
Databricks9876
by • New Contributor III
  • 8288 Views
  • 8 replies
  • 3 kudos

Resolved! Clone the data from Prod Workspace to Test workspace

Hi there, I have requirement to deep clone the data from Prod workspace to Test Workspace. I know we can do in the same workspace, but I am not aware of between workspaces.I heard some key is needed and have some firewall issues when linking between ...

Data Engineering
Reply
  • 8288 Views
  • 8 replies
  • 3 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 3 kudos

@swarup muthumula​  Azure workspace has self service migration

  • 3 kudos
7 More Replies
ANOOP_V
by • New Contributor II
  • 2417 Views
  • 3 replies
  • 3 kudos

Resolved! DataBricks Job Orchestration in PROD

Can I suggest customer about databricks Job orchestration (public preview) ? Can we use this feature in Production as well?

Data Engineering
Reply
  • 2417 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 3 kudos

@ANOOP V​ , At present, we don’t have these features. I assume we do plan to include it by Q4 FY2022.What is next for Multitask Jobs?After GA we will be working on some highly-requested features during the private preview:- Job cluster reuse: make it...

  • 3 kudos
2 More Replies
LanceYoung
by • New Contributor III
  • 11701 Views
  • 5 replies
  • 6 kudos

Resolved! Unable to make Databricks API calls from an HTML iframe rendered by a notebook's `displayHTML()` call, due to the browser enforcing CORS policy.

My GoalI want to make my Databricks Notebooks more interactive and have custom HTML/JS UI widgets that guide non-technical people through a business/data process. I want the HTML/JS widget to be able to execute a DB job, or execute some python code t...

Data Engineering
Reply
  • 11701 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 6 kudos

displayHTML content is served in production from an iframe on the domain databricksusercontent.comThis iframe is sandboxed and it seems to prevent any callsSo even by creating a CORS-friendly server in the notebook exposed through the driver-proxy-ap...

  • 6 kudos
4 More Replies
msoczka
by • New Contributor II
  • 3267 Views
  • 3 replies
  • 3 kudos

Resolved! Cluster terminated.Reason:Unexpected launch failure

Using Community Edition. Cluster starts up and immediately terminates with the following error message.Why would that be?MessageCluster terminated.Reason:Unexpected launch failureAn unexpected error was encountered while setting up the cluster. Pleas...

Data Engineering
Reply
  • 3267 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Manav
New Contributor II
  • 3 kudos

Hi, I am getting the same error. Can @DARSHAN BARGAL​  please help?

  • 3 kudos
2 More Replies
HashMan
by • New Contributor III
  • 3370 Views
  • 6 replies
  • 4 kudos

Resolved! Learn Apache Spark

I want to learn Apache Spark for Developer, where do I start and want materials are recommended.

Data Engineering
Reply
  • 3370 Views
  • 6 replies
  • 4 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

If you are a databricks customer (any paid subscription like Azure databricks), please register through https://databricks.com/learn/training/home to Academy using email from subscription.Course there are the best on the internet.If you will not see ...

  • 4 kudos
5 More Replies
Constantine
by • Contributor III
  • 18470 Views
  • 4 replies
  • 5 kudos

Resolved! How to provide UPSERT condition in PySpark

I have a table `demo_table_one` in which I want to upsert the following valuesdata = [   (11111 , 'CA', '2020-01-26'), (11111 , 'CA', '2020-02-26'), (88888 , 'CA', '2020-06-10'), (88888 , 'CA', '2020-05-10'), (88888 , 'WA', '2020-07-10'), ...

Data Engineering
Reply
  • 18470 Views
  • 4 replies
  • 5 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 5 kudos

@John Constantine,​ can you additionally share what data is in demo_table_one? as we have only df (alias update_table) in that example

  • 5 kudos
3 More Replies
sarvesh242
by • Contributor
  • 9853 Views
  • 3 replies
  • 2 kudos

Resolved! java.lang.NoSuchMethodError in databricks

I have created a package in scala. Now, I am calling a method from that package and using it in my notebook. During run time, it throws me an error java.lang.NoSuchMethodError. The method exists in the package but still, I am getting this error. Plea...

Data Engineering
Reply
  • 9853 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
sarvesh242
Contributor
  • 2 kudos

Hi! @Kaniz Fatma​ . I am using scala version 2.11 with spark 2.4.3. According to Apache spark official website https://spark.apache.org/docs/2.4.3/#:~:text=For%20the%20Scala%20API%2C%20Spark,x.) Spark 2.4.3 uses Scala 2.12. (https://spark.apache.org/...

  • 2 kudos
2 More Replies
KC_1205
by • New Contributor III
  • 3266 Views
  • 5 replies
  • 3 kudos

Resolved! Update Databricks at notebook level?

Hi all,Can we update the Databricks from existing to newer at notebook level?I know we can create a newer cluster and attach the note book to update to the newer version. but, can we also update DB at notebook as we update the libraries?if we can't,...

Data Engineering
Reply
  • 3266 Views
  • 5 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey there @Kiran Chalasani​ Just checking in. Glad that you were able to resolve your query. Would you be happy to mark the answer as best so that other members can find the solution more quickly?

  • 3 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels