cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Itachi_Naruto
by New Contributor II
  • 8278 Views
  • 3 replies
  • 0 kudos

hdbscan package error

I try to import **hdbscan** but it throws this following error/databricks/python_shell/dbruntime/PythonPackageImportsInstrumentation/__init__.py in import_patch(name, globals, locals, fromlist, level) 156 # Import the desired module. ...

  • 8278 Views
  • 3 replies
  • 0 kudos
Latest Reply
Atanu
Databricks Employee
  • 0 kudos

does this help @Rajamannar Aanjaram​ ?

  • 0 kudos
2 More Replies
Reza
by New Contributor III
  • 2824 Views
  • 1 replies
  • 0 kudos

Resolved! Can we order the widgets?

I have two text widgets (dbutils.widgets.text). One is called "start date" and another one is "end date". When I create them, they will be shown in alphabetic order (end_date, start_date). Is there any way that we can set the order when we create the...

  • 2824 Views
  • 1 replies
  • 0 kudos
Latest Reply
Atanu
Databricks Employee
  • 0 kudos

https://docs.databricks.com/notebooks/widgets.html all options available here I think. @Reza Rajabi​  , but we can crosscheck

  • 0 kudos
timothy_uk
by New Contributor III
  • 2454 Views
  • 3 replies
  • 0 kudos

Zombie .Net Spark Databricks Job (CourseGrainedExecutorBackend)

Hi all,Environment:Nodes: Standard_E8s_v3Databricks Runtime: 9.0.NET for Apache Spark 2.0.0I'm invoking spark submit to run a .Net Spark job hosted in Azure Databricks. The job is written in C#.Net with its only transformation and action, reading a C...

  • 2454 Views
  • 3 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Timothy Lin​ ,I will recommend to not use spark.stop() or System.exit(0) in your code because it will explicitly stop the Spark context but the graceful shutdown and handshake with databricks' job service does not happen.

  • 0 kudos
2 More Replies
anthony_cros
by New Contributor
  • 3505 Views
  • 1 replies
  • 0 kudos

How to publish a notebook in order to share its URL, as a Premium Plan user?

Hi,I'm a Premium Plan user and am trying to share a notebook via URL.The link at https://docs.databricks.com/notebooks/notebooks-manage.html#publish-a-notebook states: "If you’re using Community Edition, you can publish a notebook so that you can sha...

  • 3505 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hello @Anthony Cros​ - My name is Piper, and I'm a moderator for Databricks. Welcome and thank you for your question. We will give the members some time to answer your question. If needed, we will circle back around later.

  • 0 kudos
Braxx
by Contributor II
  • 5191 Views
  • 3 replies
  • 3 kudos

Resolved! spark.read excel with formula

For some reason spark is not reading the data correctly from xlsx file in the column with a formula. I am reading it from a blob storage.Consider this simple data set The column "color" has formulas for all the cells like=VLOOKUP(A4,C3:D5,2,0)In case...

image.png image.png
  • 5191 Views
  • 3 replies
  • 3 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 3 kudos

the formula itself isprobably what is actually stored in the excel file.Excel translates this to NA.I only know of setErrorCellsToFallbackValues but I doubt if this is applicable in your case here.You could use a matching function (regexp f.e.) to d...

  • 3 kudos
2 More Replies
chandan_a_v
by Valued Contributor
  • 3322 Views
  • 4 replies
  • 3 kudos

Resolved! Spark Error : RScript (1243) terminated unexpectedly: Cannot call r___RBuffer__initialize().

grid_slice %>% sdf_copy_to(  sc = sc,  name = "grid_slice",  overwrite = TRUE ) %>% sdf_repartition(  partitions = min(n_executors * 3, NROW(grid_slice)),  partition_by = "variable" ) %>% spark_apply(  f = slice_data_wrapper,  columns = c(   variable...

  • 3322 Views
  • 4 replies
  • 3 kudos
Latest Reply
chandan_a_v
Valued Contributor
  • 3 kudos

Hi @Kaniz FatmaDid you find any solution? Please let us know

  • 3 kudos
3 More Replies
RiyazAli
by Valued Contributor
  • 4464 Views
  • 2 replies
  • 3 kudos

Resolved! Where does the files downloaded from wget get stored in Databricks?

Hey Team!All I'm trying is to download a csv file stored on S3 and read it using Spark.Here's what I mean:!wget https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2020-01.csvIf i download this "yellow_tripdata_2020-01.csv" where exactly it wo...

  • 4464 Views
  • 2 replies
  • 3 kudos
Latest Reply
RiyazAli
Valued Contributor
  • 3 kudos

Hi @Kaniz Fatma​ , thanks for the remainder.Hey @Hubert Dudek​ - thank you very much for your prompt response.Initially, I was using urllib3 to 'GET' the data residing in the URL. So, I wanted an alternative for the same. Unfortunately, requests libr...

  • 3 kudos
1 More Replies
GlenLewis
by New Contributor III
  • 3719 Views
  • 3 replies
  • 0 kudos

Resolved! Markup and table of contents is no longer working on Notebooks

Around 2 days ago, Markdown in our notebooks stopped working (the %md tag isn't visible but the headings appear as #Heading1. In addition, there are no longer any table of contents on any of my workbooks. Trying a different instance in Microsoft Az...

  • 3719 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Glen Lewis​ - Thank you for coming to the community with this. Would you be happy to mark your answer as best so other members can find the solution more readily?

  • 0 kudos
2 More Replies
sonali1996
by New Contributor
  • 2708 Views
  • 0 replies
  • 0 kudos

Multithreading in SCALA DATABRICKS

Hi Team, I was trying to call/run multiple notebooks in one notebook concurrent. But the caller notebooks are getting executing one by one whereas I need to run all the caller notebooks concurrently. I have also tried using Threading in Scala Databri...

  • 2708 Views
  • 0 replies
  • 0 kudos
saltuk
by Contributor
  • 1259 Views
  • 0 replies
  • 0 kudos

Using Parquet, passing Partition on Insert Overwrite. Partition parenthesis includes equitation and it gives an error.

I am new on Spark sql, we are migrating our Cloudera to Databricks. there are a lot of SQLs done, only a few are on going. We are having some troubles during passing an argument and using it in an equitation on Partition section. LOGDATE is an argu...

  • 1259 Views
  • 0 replies
  • 0 kudos
Oricus_semicon
by New Contributor
  • 438 Views
  • 0 replies
  • 0 kudos

oricus-semicon.com

Oricus Semicon Solutions is an innovative Semiconductor Tools manufacturing company who, with almost 100 years of collective expertise, craft high tech bespoke tooling solutions for the global Semiconductor Assembly and Test industry.https://oricus-s...

  • 438 Views
  • 0 replies
  • 0 kudos
chaitanya
by New Contributor II
  • 3250 Views
  • 2 replies
  • 4 kudos

Resolved! While loading Data from blob to delta lake facing below issue

I'm calling the stored proc then store into pandas dataframe then creating list while creating list getting below error Databricks execution failed with error state Terminated. For more details please check the run page url: path An error occurred w...

  • 3250 Views
  • 2 replies
  • 4 kudos
Latest Reply
shan_chandra
Databricks Employee
  • 4 kudos

@chaitanya​ , could you please try disabling arrow optimization and see if this resolves the issue?spark.sql.execution.arrow.enabled falsespark.sql.execution.arrow.pyspark.enabled false

  • 4 kudos
1 More Replies
sanjoydas6
by New Contributor III
  • 7598 Views
  • 7 replies
  • 1 kudos

Problem faced while trying to Reset my Community Edition Password

I have forgotten my Databricks Community Edition Password and is trying to Reset the same using the Forgot Password link. It is saying that an Email will be sent with the link to reset the password but the Email is not coming. However Databricks mail...

  • 7598 Views
  • 7 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Sanjoy Das​ - Popping in here to let you know that we've escalated the issue to the team.

  • 1 kudos
6 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels