- 4814 Views
- 7 replies
- 5 kudos
Resolved! Why adding the package 'org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1' failed in runtime 9.1.x-scala2.12 but was successful using runtime 8.2.x-scala2.12 ?
Using Databricks spark submit job, setting new cluster1] "spark_version": "8.2.x-scala2.12" => OK, works fine2] "spark_version": "9.1.x-scala2.12" => FAIL, with errorsException in thread "main" java.lang.ExceptionInInitializerError at com.databricks...
- 4814 Views
- 7 replies
- 5 kudos
- 5 kudos
this has been resolved by adding the following spark_conf (not thru --conf) "spark.hadoop.fs.file.impl": "org.apache.hadoop.fs.LocalFileSystem"example:------"new_cluster": { "spark_version": "9.1.x-scala2.12", ... "spark_conf": { "spar...
- 5 kudos
- 3439 Views
- 2 replies
- 4 kudos
Resolved! display() function always return connection refused on tunneling despite successfully retrieving the schema
Hi everyone,I am using SSH tunnelling with SSHTunnelForwarder to reach a target AWS RDS PostgreSQL database. The connection got through, however when I tried to display the retrieved data frame it always throws "connection refused" error. Please see ...
- 3439 Views
- 2 replies
- 4 kudos
- 4 kudos
hi @Kurnianto Trilaksono Sutjipto​ ,This seems like a connectivity issue with the url you are trying to connect to. It fails during the display() command because read is a lazy transformation and it will not be executed right away. On the other hand,...
- 4 kudos
- 5320 Views
- 5 replies
- 11 kudos
Resolved! Runtime SQL Configuration - how to make it simple
Hi, I'm running couple of Notebooks in my pipeline and I would like to set fixed value of 'spark.sql.shuffle.partitions' - same value for every notebook. Should I do that by adding spark.conf.set.. code in each Notebook (Runtime SQL configurations ar...
- 5320 Views
- 5 replies
- 11 kudos
- 11 kudos
Hi, Thank you all for the tips. I tried before to set this option in Spark Config but didn't work for some reason. Today I tried again and it's working :).
- 11 kudos
- 4318 Views
- 3 replies
- 5 kudos
Resolved! Delta Tables incremental backup method
Hello,Does anyone tried to create an incremental backup on delta tables? What I mean is to load into the backup storage only the latest parquet files part of the Delta Table and to refresh the _delta_log folder, instead of copying the whole files aga...
- 4318 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @Stefan Stegaru​ ,You can use Delta time travel to query the data that was just added on a specific version. Then like @Hubert Dudek​ mentioned, you can copy over this sub set of data to a new table or a new location. You will need to do a deep...
- 5 kudos
- 4896 Views
- 3 replies
- 5 kudos
Resolved! Can't find or Enable "Files in Repos" feature
Not able to Find or Enable "Files in Repos" feature in the workspace, What could be the reason
- 4896 Views
- 3 replies
- 5 kudos
- 3779 Views
- 4 replies
- 2 kudos
Resolved! Anyone using RAPIDS and cuGraph on a current runtime?
We're in the process of migrating a large graph computation workload to nvidia RAPIDS + cuGraph for GPU acceleration. The package isn't a part of the base runtime and it is available by conda package management only, so can't be installed via init sc...
- 3779 Views
- 4 replies
- 2 kudos
- 2 kudos
Thanks @Prabakar Ammeappin​ , we're looking at this. Strangely, the last commit removed the rapids libraries from the base cuda-images. We're adding them back in.
- 2 kudos
- 6305 Views
- 3 replies
- 0 kudos
How can I extract/get the time, along with the status (Failed or Succeeded) into a table for every time my Databricks job finishes running?
I want to get a mail notification at the end of each day for when my Databricks job has finished running and for that I need to extract the time of it's completion and it's status. How can I achieve that?
- 6305 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Yatharth Kaushik​ you can use the JobsRunList API to get all the information of the job run. You can write a code to extract the information that you need for the table.The are multiple API's in the same doc that you can use to get information a...
- 0 kudos
- 9031 Views
- 4 replies
- 0 kudos
Resolved! SSLCertVerificationError how to disable SSL Certification
Hi, How is that possible to disable SSL Certification.With databricks API I got this error :SSLCertVerificationError SSLCertVerificationError: ("hostname 'https' doesn't match either of '*.numericable.fr', 'numericable.fr'",) MaxRetryError: HTTPS...
- 9031 Views
- 4 replies
- 0 kudos
- 0 kudos
@Bertrand BURCKER​ - Thanks for letting us know your issue is resolved. If @Prabakar Ammeappin​'s answer solved the problem, would you be happy to mark his answer as best so others can more easily find an answer for this?
- 0 kudos
- 19811 Views
- 1 replies
- 1 kudos
How to handle <IPython.core.display.HTML object>
Some libraries have intermediate IPython HTML-objects returned to the notebook cell output.Since this happens during training a machine learning model the statements are typically buried within in the library so I cannot easily interfere. (e.g. in or...
- 19811 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Kaniz Fatma​ ,thanks for showing me the link. This helps if you are in control of the generated html-object. If the html-content comes from a library, that is where the problems start, because I cannot wrap displayHTML().(I can of course look for...
- 1 kudos
- 4756 Views
- 3 replies
- 1 kudos
Train deep learning model with numpy arrays.
Hey guys,I'm trying to train deep learning model at ML databricks with numpy arrays as input.For now i organized all the data inside DF- df contains 4 columns : col1,col2,col3,col4col1 and col2 have arrays with shape (1,3,3,3,3), col 3 have array wit...
- 4756 Views
- 3 replies
- 1 kudos
- 1 kudos
Maybe you could save some your code. It will be easier to answer and also we could learn deep learning in databricks from your code.
- 1 kudos
- 712 Views
- 0 replies
- 0 kudos
Sell Handbags from Home. Free up your wardrobe space by selling a few preloved handbags at the best prices. Visit Sell Your Bag now!
Sell Handbags from Home. Free up your wardrobe space by selling a few preloved handbags at the best prices. Visit Sell Your Bag now!
- 712 Views
- 0 replies
- 0 kudos
- 20443 Views
- 6 replies
- 8 kudos
Resolved! Exporting csv files from Databricks
I'm trying to export a csv file from my Databricks workspace to my laptop.I have followed the below steps. 1.Installed databricks CLI2. Generated Token in Azure Databricks3. databricks configure --token5. Token:xxxxxxxxxxxxxxxxxxxxxxxxxx6. databrick...
- 20443 Views
- 6 replies
- 8 kudos
- 8 kudos
Hi @Sarvagna Mahakali​ There is an easier hack: a) You can save results locally on the disk and create a hyper link for downloading CSV . You can copy the file to this location: dbfs:/FileStore/table1_good_2020_12_18_07_07_19.csvb) Then download with...
- 8 kudos
- 10064 Views
- 8 replies
- 4 kudos
Resolved! Databricks SQL not displaying all the databases that i have on my cluster.
I have a cluster running on 7.3 LTS and it has about 35+ databases. When i tried to setup an endpoint on Databricks SQL, i do not see any database listed.
- 10064 Views
- 8 replies
- 4 kudos
- 4 kudos
hi @Arif Ali​ You may have to check the data access config to add the params for external metastore: spark.hadoop.javax.jdo.option.ConnectionDriverName org.mariadb.jdbc.Driverspark.hadoop.javax.jdo.option.ConnectionUserName <mysql-username>spark.had...
- 4 kudos
- 4548 Views
- 5 replies
- 8 kudos
Catch rejected Data ( Rows ) while reading with Apache-Spark.
I work with Spark-Scala and I receive Data in different formats ( .csv/.xlxs/.txt etc ), when I try to read/write this data from different sources to a any database, many records got rejected due to various issues like (special characters, data type ...
- 4548 Views
- 5 replies
- 8 kudos
- 8 kudos
or maybe schema evolution on delta lake is enough, in combination with Hubert's answer
- 8 kudos
- 8671 Views
- 5 replies
- 2 kudos
Resolved! Formatting on Databricks Alerts
Hi Guys. I have looked at the formatting options and I'm still struggling to work out how to best format the email body of a databricks alert. I want to be able to selectively choose columns from the query and dispaly them in a table. Or even if i ca...
- 8671 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Nick Hughes​ , unfortunately, this is not available for now. We have a feature request for the same. DB-I-4105 - SQL Alerts: Formatting message body when creating Custom TemplateThis feature has been considered by our product team and it will be...
- 2 kudos
| User | Count |
|---|---|
| 1614 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
