- 4833 Views
- 3 replies
- 2 kudos
Resolved! Databricks runtime from docker hub image
Hello,are databricks runtimes from docker hub ( https://hub.docker.com/r/databricksruntime/standard ) same as actual runtimes inside Databricks? I mean when we made our own docker image from databricksruntime/standard will be there same dependencies...
- 4833 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @michael henzl​ ,Just checking if you still need help with this or not any more? please let us know
- 2 kudos
- 2632 Views
- 2 replies
- 5 kudos
Resolved! Different configurations for same Databricks Runtime version
Hi all,On my DBR installations, s3a scheme is mapped to shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem. On my customer's DBR installations it is mapped to com.databricks.s3a.S3AFileSystem.We both use the same DBR runtime, and none of us has...
- 2632 Views
- 2 replies
- 5 kudos
- 5 kudos
@Yoni Au​ , If both of you are using the same DBR version, then you should not find any difference. As @Hubert Dudek​ mentioned, there might be some spark configuration change made on one of the clusters. Also, it's worth checking for any cluster sco...
- 5 kudos
- 2930 Views
- 3 replies
- 3 kudos
Resolved! Trigger.AvailableNow does not support maxOffsetsPerTrigger in Databricks runtime 10.3
Hello,I ran a spark stream job to ingest data from kafka to test Trigger.AvailableNow.What's environment the job run ?1: Databricks runtime 10.32: Azure cloud3: 1 Driver node + 3 work nodes( 14GB, 4core)val maxOffsetsPerTrigger = "500"spark.conf.set...
- 2930 Views
- 3 replies
- 3 kudos
- 3 kudos
You'd be better off with 1 node with 12 cores than 3 nodes with 4 each. You're shuffles are going to be much better one 1 machine.
- 3 kudos
- 974 Views
- 1 replies
- 19 kudos
Runtime 10.4 is available and is LTS. From today it is not beta anymore and it is LTS! mean Long Time Support. So for sure it will be with us for next...
Runtime 10.4 is available and is LTS.From today it is not beta anymore and it is LTS! mean Long Time Support. So for sure it will be with us for next 2 years.10.4 includes some awesome features like:Auto Compaction rollbacks are now enabled by defaul...
- 974 Views
- 1 replies
- 19 kudos
- 19 kudos
I have the same favorite.I am curious how it works under the hood. zipWithIndex?
- 19 kudos
- 7403 Views
- 4 replies
- 2 kudos
Resolved! Implementation of a stable Spark Structured Streaming Application
Hi folks,I have an issue. It's not critical but's annoying.We have implemented a Spark Structured Streaming Application.This application will be triggered wire Azure Data Factory (every 8 minutes). Ok, this setup sounds a little bit weird and it's no...
- 7403 Views
- 4 replies
- 2 kudos
- 2 kudos
@Markus Freischlad​ Looks like the spark driver was stuck. It will be good to capture the thread dump of the Spark driver to understand what operation is stuck
- 2 kudos
- 2542 Views
- 4 replies
- 1 kudos
Resolved! Next LTS version after 9.1
What is the next DBR LTS version after 9.1 and when it is planned to be released in Azure?
- 2542 Views
- 4 replies
- 1 kudos
- 10252 Views
- 3 replies
- 3 kudos
Resolved! Need help with this python import error.
I am using databricks runtime 9.1 LTS ML and I got this error when I tried to import Scikit Learn package. I got the following error message:TypeError Traceback (most recent call last) <command-181041> in <module> ...
- 10252 Views
- 3 replies
- 3 kudos
- 3 kudos
@Atanu Sarkar​ I am using databricks runtime 9.1ML LTS and python version is 3.8.10I am only just running import statementfrom sklearn.metrics import * from sklearn.preprocessing import LabelEncoder
- 3 kudos
- 5663 Views
- 4 replies
- 3 kudos
Resolved! Update Databricks Runtime
Hy guys,I need to upgrade my databricks runtime (current 8.0 ) What the precautions should I take ?Thank you very much
- 5663 Views
- 4 replies
- 3 kudos
- 3 kudos
If you want to know the version of Databricks runtime in Azure after creation: Go to Azure Data bricks portal => Clusters => Interactive Clusters => here you can find the run time version. For more details, refer "Azure Databricks Runtime versions".R...
- 3 kudos
- 2576 Views
- 3 replies
- 2 kudos
Do Databricks support XLA compilation for TensorFlow models?
I am defining a sequential Keras model using tensorflow.kerasRuntime: Databricks ML 8.3Cluster: Standard NC24 with 4 GPUs per node.To enable XLA compilation, I set the following flag:tf.config.optimizer.set_jit(True)Here is the output when I try to t...
- 2576 Views
- 3 replies
- 2 kudos
- 2 kudos
I don't think this is specific to Databricks, but rather Tensorflow. See https://stackoverflow.com/questions/68614547/tensorflow-libdevice-not-found-why-is-it-not-found-in-the-searched-path for a possibly relevant solution.I don't see evidence that t...
- 2 kudos
- 3823 Views
- 3 replies
- 3 kudos
Resolved! Installing new libraries on Azure Databricks Clusters
Hi Everyone,I was trying to install the newest python version on the Databricks Clusters and it has the runtime version 7.3 LTS, but no matter how many times I try it keeps installing the 3.7.5 version of python.I know that Runtime version 7.3 LTS co...
- 3823 Views
- 3 replies
- 3 kudos
- 3 kudos
I've done this before using a custom docker image, but even then the runtime itself continues to use the version of python 3 which is installed as part of the OS. The easiest way to get to a newer version is to use a newer runtime. If you're sticking...
- 3 kudos
- 2043 Views
- 3 replies
- 0 kudos
Resolved! SQL Issues with 10.0 runtime
I was testing my sqls with new 10.0 runtime and found some interesting/weird thing. The same sql with explode function fails for some scenarios in 10.0! Could not figure out yet the reason
- 2043 Views
- 3 replies
- 0 kudos
- 0 kudos
@Yadhuram MC​ if the issue persists, please email me at bilal dot aslam at databricks dot com. I would like to get to the root of this issue. It
- 0 kudos
- 7207 Views
- 10 replies
- 6 kudos
Older Spark Version loaded into the spark notebook
I have databricks runtime for a job set to latest 10.0 Beta (includes Apache Spark 3.2.0, Scala 2.12) .In the notebook when I check for the spark version, I see version 3.1.0 instead of version 3.2.0I need the Spark version 3.2 to process workloads a...
- 7207 Views
- 10 replies
- 6 kudos
- 6 kudos
hi @Dhaivat Upadhyay​ ,Good news, DBR 10 was release yesterday October 20th. You can find more details in the release notes website
- 6 kudos
- 3185 Views
- 4 replies
- 10 kudos
Resolved! How to make sparklyr extension work with Databricks runtime?
Hello. I'm the current maintainer of sparklyr (a R interface for Apache Spark) and a few sparklyr extensions such as sparklyr.flint.Sparklyr was fortunate to receive some contribution from Databricks folks, which enabled R users to run `spark_connect...
- 3185 Views
- 4 replies
- 10 kudos
- 10 kudos
Yes, as Sebastian said. Also, it would be good to know what the error is here. One possible explanation is that the JARs are not copied to the executor nodes. This would be solved by Sebasitian's suggestion.
- 10 kudos
- 1968 Views
- 2 replies
- 2 kudos
- 1968 Views
- 2 replies
- 2 kudos
- 2 kudos
@Werner Stinckens​ you can think about the Databricks Runtime as a contract. It does and will change over time. However, we offer Long Term Support versions of the runtime which offer multi-year support. If you have production jobs, I would definitel...
- 2 kudos
- 1883 Views
- 3 replies
- 0 kudos
Resolved! Libraries in Databricks Runtimes
Is it possible to easily determine what libraries and which version are included in a specific DBR Version?
- 1883 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello. My name is Piper and I'm one of the community moderators. One of the team members sent this information to me.This should be the correct path to check libraries installed with DBRs.https://docs.databricks.com/release-notes/runtime/8.3ml.html?_...
- 0 kudos