- 3382 Views
- 1 replies
- 0 kudos
What happened to the ephemeral notebook links????? and the job ids????
Hey Databricks, Why did you remove the ephemeral notebook links and job Ids from the parallel runs? This has created a huge gap for us. We can no longer view the ephemeral notebooks, and also the Jobids are missing from the output. Waccha doing?...
- 3382 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Kaniz, It's funny you mention these things - we are doing some of those - the problem now is that the JobId is obscured from the output meaning we can't tell which ephemeral notebook goes with which JobId. It looks like the ephemeral notebook ...
- 0 kudos
- 5459 Views
- 0 replies
- 0 kudos
Updating Databricks SQL Warehouse using Terraform
We can Update SQL Warehouse manually in Databricks.Click SQL Warehouses in the sidebarIn Advanced optionsWe can find Unity Catalog toggle button there! While Updating Existing SQL Warehouse in Azure to enable unity catalog using terraform, I couldn'...
- 5459 Views
- 0 replies
- 0 kudos
- 5851 Views
- 5 replies
- 3 kudos
OSError: [Errno 78] Remote address changed
Hello:)as part of deploying an app that previously ran directly on emr to databricks, we are running experiments using LTS 9.1, and getting the following error: PythonException: An exception was thrown from a UDF: 'pyspark.serializers.SerializationEr...
- 5851 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @liormayn , I can understand. I see the fix went on 20 March 2024, you would have to restart the clusters. Thanks!
- 3 kudos
- 2858 Views
- 1 replies
- 0 kudos
Py4JError: An error occurred while calling o992.resourceProfileManager
Hello I am trying to run the SparkXGBoostRegressor and I am getting the following error:SpoilerPy4JError: An error occurred while calling o992.resourceProfileManager. Trace: py4j.security.Py4JSecurityException: Method public org.apache.spark.resource...
- 2858 Views
- 1 replies
- 0 kudos
- 3254 Views
- 0 replies
- 0 kudos
Stream to stream join NullPointerException
I have a DLT pipeline running in continous mode. I have a stream to stream join which runs for the first 5hrs but then fails with a Null Pointer Exception. I need assistance to know what I need to do to handle this. my code is structured as below:@dl...
- 3254 Views
- 0 replies
- 0 kudos
- 6043 Views
- 4 replies
- 2 kudos
Resolved! How to choose a compute, and how to find alternatives for the current compute being used?
We are using a compute for an Interactive Cluster in Production which incurs X amount of cost. We want to know what are the options available to use with near about the same processing power as the current compute but incur a cost of Y, which is less...
- 6043 Views
- 4 replies
- 2 kudos
- 2 kudos
Hello @Ikanip , You can utilize the Databricks Pricing Calculator to estimate costs. For detailed information on compute capacity, please refer to your cloud provider's documentation regarding Virtual Machine instance types.
- 2 kudos
- 1643 Views
- 0 replies
- 0 kudos
Databricks Running Jobs and Terraform
What happens to a currently running job when a workspace is deployed again using Terraform? Are the jobs paused/resumed, or are they left unaffected without any down time? Searching for this specific scenario doesn't seem to come up with anything and...
- 1643 Views
- 0 replies
- 0 kudos
- 12306 Views
- 5 replies
- 1 kudos
Job parameters to get date and time
I'm trying to set up a workflow in databricks and I need my job parameter to get the date and time. I see in the documentation there's some options for dynamic values.I'm trying to use this one: {{job.start_time.[argument]}}For the "argument" there, ...
- 12306 Views
- 5 replies
- 1 kudos
- 1 kudos
Then please change the code to:```iso_datetime = dbutils.widgets.get("LoadID")```
- 1 kudos
- 985 Views
- 0 replies
- 0 kudos
Archive file support in Jar Type application
In my spark application, I am using set of python libraries. I am submitting spark application as Jar Task. But I am not able to find any option provide Archive Files.So, in order to handle python dependencies, I am using approach:Create archive file...
- 985 Views
- 0 replies
- 0 kudos
- 1485 Views
- 0 replies
- 0 kudos
Getting databricks-connect com.fasterxml.jackson.databind.exc.MismatchedInputException parse warning
Hi community, I am getting below warning when I try using pyspark code for some of my use-cases using databricks-connect. Is this a critical warning, and any idea what does it mean?Logs: WARN DatabricksConnectConf: Could not parse /root/.databricks-c...
- 1485 Views
- 0 replies
- 0 kudos
- 12895 Views
- 1 replies
- 0 kudos
Getting python version errors when using pyspark rdd using databricks connect
Hi community, When I use pyspark rdd related functions in my environment using databricks connect, I get below error: Databricks cluster version: 12.2. `RuntimeError: Python in worker has different version 3.9 than that in driver 3.10, PySpark cannot...
- 12895 Views
- 1 replies
- 0 kudos
- 0 kudos
Got it. As a side note, I tried above methods, but the error persisted, hence upon reading docs again, there was this statement: You must install Python 3 on your development machine, and the minor version of your client Python installation must be t...
- 0 kudos
- 2692 Views
- 0 replies
- 1 kudos
Native Slack Integration
Hi,Are there any plans to build native slack integration? I'm envisioning a one-time connector to Slack that would automatically populate all channels and users to select to use for example when configuring an alert notification. It is does not seem ...
- 2692 Views
- 0 replies
- 1 kudos
- 2768 Views
- 0 replies
- 1 kudos
connection from databricks to snowflake using OKTA
Hi team,This is how I connect to Snowflake from Jupyter Notebook:import snowflake.connector snowflake_connection = snowflake.connector.connect( authenticator='externalbrowser', user='U1', account='company1.us-east-1', database='db1',...
- 2768 Views
- 0 replies
- 1 kudos
- 5398 Views
- 2 replies
- 1 kudos
Workflows: Running dependent task despite earlier task fail
I have a scheduled task running in workflow.Task 1 computes some parameters then these are picked up by a dependent reporting task: Task 2.I want Task 2 to report "Failure" if Task 1 fails. Yet creating a dependency in workflows means that Task 2 wil...
- 5398 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @sharpbetty , Any suggestions how I can keep the parameter sharing and dependency from Task 1 to Task 2, yet also allow Task 2 to fire even on failure of Task 1?Setup:Task 2 dependent on Task1 Challenge: To Fire Task 2 even on Task 1 FailureSoluti...
- 1 kudos
- 2765 Views
- 1 replies
- 0 kudos
Issue with Python Package Management in Spark application
In a pyspark application, I am using set of python libraries. In order to handle python dependencies while running pyspark application, I am using the approach provided by spark : Create archive file of Python virtual environment using required set o...
- 2765 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, I have not tried it but based on the doc you have to go by this approach. ./environment/bin/pythonmust be replaced with the correct path. import os from pyspark.sql import SparkSession os.environ['PYSPARK_PYTHON'] = "./environment/bin/python" sp...
- 0 kudos
