cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

rbiddle
by New Contributor
  • 10818 Views
  • 3 replies
  • 0 kudos

Specifying a Managed Resource Group name

Azure Databricks provisions a Managed Resource Group when you create your Workspace. Is there a way to specify the name of the Managed Resource Group and its resources during creation?The defaults created by the Workspace violate my company's standar...

  • 10818 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hey there @Robert Biddle​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.Cheers!

  • 0 kudos
2 More Replies
Confused
by New Contributor III
  • 7051 Views
  • 3 replies
  • 3 kudos

Resolved! Dealing with updates to a delta table being used as a streaming source

Hi AllI have a requirement to perform updates on a delta table that is the source for a streaming query.I would like to be able to update the table and have the stream continue to work while also not ending up with duplicates.From my research it se...

  • 7051 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey @Mathew Walters​ Hope you are doing great.Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 3 kudos
2 More Replies
JananiMohan
by New Contributor
  • 8219 Views
  • 4 replies
  • 0 kudos

Resolved! ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

After the new release of numpy 1.22.0 on Dec 31st, Databricks failed with this error for my existing Databricks Notebook Version 10.1 and numpy 1.20.0Qn: Why did the earlier releases after 1.20.0 uptil 1.22.0 did not raise the same exception. ?

  • 8219 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Janani Mohan​ Hope you are doing well.Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!

  • 0 kudos
3 More Replies
KC_1205
by New Contributor III
  • 6283 Views
  • 9 replies
  • 1 kudos

Resolved! Migrating DB from 7.3 LTS to 9.1 LTS

Hi All,I have a code in the dev and production using DB 7.3 LTS. Now, I would like to update the environment to 9.1 LTS as support is going to finish. I have gone through the documentation given in the following link. https://docs.databricks.com/rele...

  • 6283 Views
  • 9 replies
  • 1 kudos
Latest Reply
gmondauto
New Contributor II
  • 1 kudos

@Kiran Chalasani​  Hey Have you ever been able to run 7.3run time with multi_gpus before you migrated to 9.1?

  • 1 kudos
8 More Replies
Mr__E
by Contributor II
  • 3885 Views
  • 1 replies
  • 1 kudos

Databricks dashboard removing order, incorrectly sorted.

I created a table that aggregates data by year and week of year and display this in a chart over time. As part of the query, I order by the year, then week columns. In the visualization on the query (in the SQL editor), I disabled the sort, because i...

  • 3885 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Mr. E (Customer)​ , did you get a chance to check Kaniz's previous comments ? is this issue resolved or do you need any further help here ?

  • 1 kudos
pantelis_mare
by Contributor III
  • 6755 Views
  • 7 replies
  • 10 kudos

Resolved! Delta Upsert performance on empty table

Hello all,I was just wandering, performance wise how does it compare a plain write operation with a merge operation on an EMPTY delta table. Do we really risk to get significant performance drop?The use case would be to have the same pipeline for ini...

  • 6755 Views
  • 7 replies
  • 10 kudos
Latest Reply
pantelis_mare
Contributor III
  • 10 kudos

Hello @Kaniz Fatma​ ,Unfortunately I did not do any further investigation on the subject. Given that the merge on an empty table will only be done once at the creation of a table, it wouldn't really matter to be honest.

  • 10 kudos
6 More Replies
720677
by New Contributor III
  • 2465 Views
  • 1 replies
  • 2 kudos

Resolved! Databricks Clusters on GCP stop working "Environment directory not found" issue - waitForEnvironmentFileSystem

Starting from yesterday 17/5/2022 i start getting errors while running notebooks or jobs on clusters of Databricks GCP. The error is: SparkException: Environment directory not found at /local_disk0/.ephemeral_nfs/cluster_libraries/pythonThe job/noteb...

  • 2465 Views
  • 1 replies
  • 2 kudos
Latest Reply
720677
New Contributor III
  • 2 kudos

Databricks supports detected an issue with the NFS mounts on GCP. Looks like DBR 10.X versions were affected. After several hours they fixed it and now the same clusters are back to normal.

  • 2 kudos
JohanRex
by New Contributor II
  • 8392 Views
  • 3 replies
  • 5 kudos

Resolved! IllegalArgumentException: requirement failed: Result for RPC Some(e100cace-3836-4461-8902-80b3744fcb6b) lost, please retry your request.

I'm using databricks connect to talk to a cluster on Azure. When doing a count on a dataframe I sometimes get this error message. Once I've gotten it once I don't seem to be able to get rid of it even if I restart my dev environment. ----------------...

  • 8392 Views
  • 3 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Johan Rex​ We checked with databricks connect team, this issue can happen when the library is too large to upload, Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no ...

  • 5 kudos
2 More Replies
sparkstreaming
by New Contributor III
  • 8092 Views
  • 4 replies
  • 6 kudos

Resolved! Rest API invocation for databricks notebook fails while invoking from ADF pipeline

In the current implementation a streaming databricks notebook needs to be started based on the configuration passed. Since the rest of databricks notebooks are being invoked by using ADF,it was decided to use ADF for starting these notebooks. Since t...

  • 8092 Views
  • 4 replies
  • 6 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 6 kudos

@Prasanth KP​ ,clearly, the rest call is invalid. What endpoint do you call?Also do not forget to authenticate.May I ask why you use the REST API instead of the available notebook functionality of ADF?

  • 6 kudos
3 More Replies
findinpath
by Contributor
  • 7296 Views
  • 2 replies
  • 3 kudos

Databricks 2.6.25 JDBC driver can't create tables with `GENERATED` columns

I'm using the Databricks JDBC driver recently made available via Maven:https://mvnrepository.com/artifact/com.databricks/databricks-jdbc/2.6.25While trying to create a table with `GENERATED` columns I receive the following exception:Caused by: java.s...

  • 7296 Views
  • 2 replies
  • 3 kudos
Latest Reply
findinpath
Contributor
  • 3 kudos

I was under the impression that this has been recognised as a BUG and is being handled by Databricks.What do I need to do for reporting the issue officially as a BUG?

  • 3 kudos
1 More Replies
ChristianWuerdi
by New Contributor III
  • 15671 Views
  • 4 replies
  • 5 kudos

Resolved! How can I backup my Databricks instance?

We have a Databricks instance on Azure that has somewhat organically grow with dozens of users and hundreds of notebooks. How do I conveniently backup this env so in case disaster strikes the notebooks aren't lost? The data itself is backed by Azure ...

  • 15671 Views
  • 4 replies
  • 5 kudos
Latest Reply
ChristianWuerdi
New Contributor III
  • 5 kudos

@Kaniz Fatma​ All good thanks, combination of CLI + gradually migrating everything to git is a viable solution

  • 5 kudos
3 More Replies
StephanieAlba
by Databricks Employee
  • 8430 Views
  • 2 replies
  • 5 kudos

Resolved! How to add a select all option in a Databricks SQL parameter? I would like to use a query-based drop-down list.

So I want to create a select all button in a parameter. The actual parameter has around 200 options because of the size of the database. However, if I want a general summary where you can see all the options I would have to select one by one and that...

  • 8430 Views
  • 2 replies
  • 5 kudos
Latest Reply
StephanieAlba
Databricks Employee
  • 5 kudos

You could add '--- All Stores ---' to your list. Here is the query I would use to populate the drop-down. S.O. answer hereSELECT store as store_name FROM ( Select Distinct store From Table   UNION ALL   SELECT ...

  • 5 kudos
1 More Replies
pantelis_mare
by Contributor III
  • 6246 Views
  • 4 replies
  • 5 kudos

Resolved! Slow imports for concurrent notebooks

Hello all,I have a large number of light notebooks to run so I am taking the concurrent approach launching notebook runs with dbutils.notebook.run in parallel. The more I increase parallelism the more I see the duration of each notebook increasing.I ...

  • 6246 Views
  • 4 replies
  • 5 kudos
Latest Reply
pantelis_mare
Contributor III
  • 5 kudos

Hello @Kaniz Fatma​ yes it is clear.Following some tests on my side using a ***** notebook that all it does is importing stuff and sleeping for 15 secs (so nothing to do with spark) I figured that even with a 32 cores driver, the fatigue point is clo...

  • 5 kudos
3 More Replies
Anonymous
by Not applicable
  • 2948 Views
  • 3 replies
  • 2 kudos

Resolved! JOB API KEEPS SAYING THE JOB IS RUNNING

I have a library that waits until the job goes in the "TERMINATED" / "SKIPPED" state before continuing. It pools the JOB API.Unfortunately, I'm experiencing cases where the job is terminated on the GUI but the API still keeps saying "RUNNING".There i...

  • 2948 Views
  • 3 replies
  • 2 kudos
Latest Reply
Prabakar
Databricks Employee
  • 2 kudos

@Alessio Palma​ could you please provide the API that you are using? Also share some sample output and logs that would help us with some information.

  • 2 kudos
2 More Replies
Serhii
by Contributor
  • 3518 Views
  • 2 replies
  • 6 kudos

Resolved! DBFS FileStore html document not showing in the browser

hello all! I am using the guide https://docs.databricks.com/data/filestore.html to save folder of static html content to the DBFS FileStore directory (as a sub-directory) and have "enable DBFS web browsing" setting on but still I can't view the web p...

  • 3518 Views
  • 2 replies
  • 6 kudos
Latest Reply
Prabakar
Databricks Employee
  • 6 kudos

@Sergii Ivakhno​ In FileStore you can save files, such as images and libraries, that are accessible within HTML and JavaScript when you call displayHTML. However when you try to access the link it will download the file to your local desktop.

  • 6 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels