- 6265 Views
- 3 replies
- 4 kudos
Resolved! Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot modify the value of a Spark config: spark.executor.memory;
I am trying to read a 16mb excel file and I was getting a gc overhead limit exceeded error to resolve that i tried to increase my executor memory with,spark.conf.set("spark.executor.memory", "8g")but i got the following stack :Using Spark's default l...
- 6265 Views
- 3 replies
- 4 kudos
- 4 kudos
On the cluster configuration page, go to the advanced options. Click it to expand the field. There you will find the Spark tab and you can set the values there in the "Spark config".
- 4 kudos
- 10142 Views
- 9 replies
- 8 kudos
Resolved! Getting Null values at the place of data which was removed manually from excel file( solved )
I was reading an excel file with one column,country india India india India indiadataframe i got from this data : df.show()+-------+ |country| +-------+ | india | | India | | india | | India | | india | +-------+In the next step i removed last value ...
- 10142 Views
- 9 replies
- 8 kudos
- 8 kudos
@sarvesh singh​ - Thank you for letting us know. Would you be happy to mark the best answer so others can find the solution easily?
- 8 kudos
- 5584 Views
- 4 replies
- 2 kudos
Resolved! How to transfer ownership of a Databricks cloud standard account?
My email address is the owner of an account in a particular standard plan tenancy. I would like to transfer ownership to another user so they can change billing details, and take admin access going forward. How can this be accomplished?
- 5584 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @David Allen​ To transfer account owner rights, contact your Databricks account representative. This is applicable for both legacy and E2 accounts.https://docs.databricks.com/administration-guide/account-settings/account-console.html#access-the-ac...
- 2 kudos
- 3996 Views
- 4 replies
- 3 kudos
- 3996 Views
- 4 replies
- 3 kudos
- 3 kudos
You may have noticed that the local SQL endpoint is not listed in the options for getting started with APEX. The local SQL endpoint is an extremely useful feature for getting ADO.NET web services started. I say check this uk-dissertation.com review f...
- 3 kudos
- 6552 Views
- 6 replies
- 1 kudos
Hi Guys Is there any documentation on where the /databricks-datasets/ mount is actually served from?We are looking at locking down where our workspace...
Hi GuysIs there any documentation on where the /databricks-datasets/ mount is actually served from?We are looking at locking down where our workspace can reach out to via the internet and as it currently stands we are unable to reach this.I did look ...
- 6552 Views
- 6 replies
- 1 kudos
- 1 kudos
Hello Mat, Thanks for letting us know. Would you be happy to mark your answer as best if that will solve the problem for others? That way, members will be able to find the solution more easily.
- 1 kudos
- 4251 Views
- 2 replies
- 1 kudos
2021-08-Best-Practices-for-Your-Data-Architecture-v3-OG-1200x628
Thanks to everyone who joined the Best Practices for Your Data Architecture session on Getting Workloads to Production using CI/CD. You can access the on-demand session recording here, and the code in the Databricks Labs CI/CD Templates Repo. Posted ...
- 4251 Views
- 2 replies
- 1 kudos
- 1 kudos
Here's the embedded links list!Jobs scheduling and orchestrationBuilt-in job scheduling: https://docs.databricks.com/jobs.html#schedule-a-job Periodic scheduling of the jobsExecute notebook / jar / Python script / Spark-submitMultitask JobsExecute no...
- 1 kudos
- 6198 Views
- 7 replies
- 5 kudos
Resolved! Why adding the package 'org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1' failed in runtime 9.1.x-scala2.12 but was successful using runtime 8.2.x-scala2.12 ?
Using Databricks spark submit job, setting new cluster1] "spark_version": "8.2.x-scala2.12" => OK, works fine2] "spark_version": "9.1.x-scala2.12" => FAIL, with errorsException in thread "main" java.lang.ExceptionInInitializerError at com.databricks...
- 6198 Views
- 7 replies
- 5 kudos
- 5 kudos
this has been resolved by adding the following spark_conf (not thru --conf) "spark.hadoop.fs.file.impl": "org.apache.hadoop.fs.LocalFileSystem"example:------"new_cluster": { "spark_version": "9.1.x-scala2.12", ... "spark_conf": { "spar...
- 5 kudos
- 4347 Views
- 2 replies
- 4 kudos
Resolved! display() function always return connection refused on tunneling despite successfully retrieving the schema
Hi everyone,I am using SSH tunnelling with SSHTunnelForwarder to reach a target AWS RDS PostgreSQL database. The connection got through, however when I tried to display the retrieved data frame it always throws "connection refused" error. Please see ...
- 4347 Views
- 2 replies
- 4 kudos
- 4 kudos
hi @Kurnianto Trilaksono Sutjipto​ ,This seems like a connectivity issue with the url you are trying to connect to. It fails during the display() command because read is a lazy transformation and it will not be executed right away. On the other hand,...
- 4 kudos
- 7439 Views
- 5 replies
- 11 kudos
Resolved! Runtime SQL Configuration - how to make it simple
Hi, I'm running couple of Notebooks in my pipeline and I would like to set fixed value of 'spark.sql.shuffle.partitions' - same value for every notebook. Should I do that by adding spark.conf.set.. code in each Notebook (Runtime SQL configurations ar...
- 7439 Views
- 5 replies
- 11 kudos
- 11 kudos
Hi, Thank you all for the tips. I tried before to set this option in Spark Config but didn't work for some reason. Today I tried again and it's working :).
- 11 kudos
- 5426 Views
- 3 replies
- 5 kudos
Resolved! Delta Tables incremental backup method
Hello,Does anyone tried to create an incremental backup on delta tables? What I mean is to load into the backup storage only the latest parquet files part of the Delta Table and to refresh the _delta_log folder, instead of copying the whole files aga...
- 5426 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @Stefan Stegaru​ ,You can use Delta time travel to query the data that was just added on a specific version. Then like @Hubert Dudek​ mentioned, you can copy over this sub set of data to a new table or a new location. You will need to do a deep...
- 5 kudos
- 5782 Views
- 3 replies
- 5 kudos
Resolved! Can't find or Enable "Files in Repos" feature
Not able to Find or Enable "Files in Repos" feature in the workspace, What could be the reason
- 5782 Views
- 3 replies
- 5 kudos
- 4734 Views
- 4 replies
- 2 kudos
Resolved! Anyone using RAPIDS and cuGraph on a current runtime?
We're in the process of migrating a large graph computation workload to nvidia RAPIDS + cuGraph for GPU acceleration. The package isn't a part of the base runtime and it is available by conda package management only, so can't be installed via init sc...
- 4734 Views
- 4 replies
- 2 kudos
- 2 kudos
Thanks @Prabakar Ammeappin​ , we're looking at this. Strangely, the last commit removed the rapids libraries from the base cuda-images. We're adding them back in.
- 2 kudos
- 7661 Views
- 3 replies
- 0 kudos
How can I extract/get the time, along with the status (Failed or Succeeded) into a table for every time my Databricks job finishes running?
I want to get a mail notification at the end of each day for when my Databricks job has finished running and for that I need to extract the time of it's completion and it's status. How can I achieve that?
- 7661 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Yatharth Kaushik​ you can use the JobsRunList API to get all the information of the job run. You can write a code to extract the information that you need for the table.The are multiple API's in the same doc that you can use to get information a...
- 0 kudos
- 10591 Views
- 4 replies
- 0 kudos
Resolved! SSLCertVerificationError how to disable SSL Certification
Hi, How is that possible to disable SSL Certification.With databricks API I got this error :SSLCertVerificationError SSLCertVerificationError: ("hostname 'https' doesn't match either of '*.numericable.fr', 'numericable.fr'",) MaxRetryError: HTTPS...
- 10591 Views
- 4 replies
- 0 kudos
- 0 kudos
@Bertrand BURCKER​ - Thanks for letting us know your issue is resolved. If @Prabakar Ammeappin​'s answer solved the problem, would you be happy to mark his answer as best so others can more easily find an answer for this?
- 0 kudos
- 21197 Views
- 1 replies
- 1 kudos
How to handle <IPython.core.display.HTML object>
Some libraries have intermediate IPython HTML-objects returned to the notebook cell output.Since this happens during training a machine learning model the statements are typically buried within in the library so I cannot easily interfere. (e.g. in or...
- 21197 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Kaniz Fatma​ ,thanks for showing me the link. This helps if you are in control of the generated html-object. If the html-content comes from a library, that is where the problems start, because I cannot wrap displayHTML().(I can of course look for...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 561 | |
| 349 | |
| 287 |
