cancel
Showing results for 
Search instead for 
Did you mean: 
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Benedetta
by New Contributor III
  • 845 Views
  • 2 replies
  • 1 kudos

What happened to the ephemeral notebook links????? and the job ids????

Hey Databricks,      Why did you remove the ephemeral notebook links and job Ids from the parallel runs? This has created a huge gap for us. We can no longer view the ephemeral notebooks, and also the Jobids are missing from the output. Waccha doing?...

  • 845 Views
  • 2 replies
  • 1 kudos
Latest Reply
Benedetta
New Contributor III
  • 1 kudos

Hi Kaniz,    It's funny you mention these things - we are doing some of those - the problem now is that the JobId is obscured from the output meaning we can't tell which ephemeral notebook goes with which JobId.  It looks like the ephemeral notebook ...

  • 1 kudos
1 More Replies
FelipeRegis
by New Contributor
  • 824 Views
  • 1 replies
  • 0 kudos

Not able to access data registered in Unity Catalog using Simba ODBC driver

Hi folks, I'm working on a project with Databricks using Unity Catalog and a connection to SSIS (SQL Server Integration Services).My team is trying to access data registered in Unity Catalog using Simba ODBC driver version 2.8.0.1002. They mentioned ...

odbc_driver_unity_catalog_issue.png
  • 824 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @FelipeRegis, It seems you’re encountering issues with accessing data registered in Unity Catalog using the Simba ODBC driver. Let’s explore some possible solutions: Delta Lake Native Connector: Consider using Delta Lake’s native Delta JDBC/OD...

  • 0 kudos
Benedetta
by New Contributor III
  • 709 Views
  • 1 replies
  • 0 kudos

What happened to the JobIds in the parallel runs (again)????

Hey Databricks,      Why did you take away the jobids from the parallel runs? We use those to identify which output goes with which run. Please put them back.Benedetta 

  • 709 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Benedetta,  Thank you for reaching out. I understand your concern regarding the jobids in parallel runs. I will look into this matter and get back to you with more information as soon as possible.

  • 0 kudos
dm7
by New Contributor
  • 486 Views
  • 1 replies
  • 0 kudos

DLT CDC/SCD - Taking the latest ID per day

Hi I'm creating a DLT pipeline which uses DLT CDC to implement SCD Type 1 to take the latest record using a datetime column which works with no issues:@dlt.view def users(): return spark.readStream.table("source_table") dlt.create_streaming_table(...

  • 486 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @dm7, Thank you for providing the details of your DLT pipeline and the desired outcome! It looks like you’re trying to implement a Slowly Changing Dimension (SCD) Type 2 behaviour where you want to capture historical changes over time. Let’s br...

  • 0 kudos
BenCCC
by New Contributor
  • 387 Views
  • 1 replies
  • 0 kudos

Installing R packages for a customer docker container for compute

Hi,I'm trying to create a customer docker image with some R packages re-installed. However, when I try to use it in a notebook, it can't seem to find the installed packages. The build runs fine.FROM databricksruntime/rbase:14.3-LTS## update system li...

  • 387 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @BenCCC, Here are a few things you can check: Package Installation in Dockerfile: In your Dockerfile, you’re using the RUN R -e 'install.packages(...)' command to install R packages. While this approach works, there are alternative methods th...

  • 0 kudos
groch_adam
by New Contributor
  • 256 Views
  • 1 replies
  • 0 kudos

Usage of SparkMetric_CL, SparkListenerEvent_CL and SparkLoggingEvent_CL

I am wondering If can retrieve any information from Azure Log Analytics custom tables (already set) for Azure Databricks. Would like to retrieve information about query and data performance for SQL Warehouse Cluster. I am not sure If I can get it fro...

  • 256 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @groch_adam, Retrieving information from Azure Log Analytics custom tables for Azure Databricks is possible.   Let me guide you through the process. Azure Databricks Monitoring Library: To send application logs and metrics from Azure Databric...

  • 0 kudos
liormayn
by New Contributor III
  • 396 Views
  • 1 replies
  • 3 kudos

Error while encoding: java.lang.RuntimeException: org.apache.spark.sql.catalyst.util.GenericArrayDa

Hello:)we are trying to run an existing working flow that works currently on EMR, on databricks.we use LTS 10.4, and when loading the data we get the following error:at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:...

  • 396 Views
  • 1 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @liormayn, It seems you’re encountering an issue related to the schema of your data when running your existing workflow on Databricks. Let’s explore some potential solutions: Parquet Decimal Columns Issue: The error message you’re seeing might...

  • 3 kudos
afdadfadsfadsf
by New Contributor
  • 411 Views
  • 1 replies
  • 0 kudos

Create Databricks model serving endpoint in Azure DevOps yaml

Hello,I need to create and destroy a model endpoint as part of CI/CD. I tried with mlflow deployments create-endpoint, giving databricks as --target however it errors saying that --endpoint is not a known argument when clearly --endpoint is required....

  • 411 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @afdadfadsfadsf, Creating and managing model endpoints as part of your CI/CD pipeline is essential for deploying machine learning models. I can provide some guidance on how to set up a CI/CD pipeline using YAML in Azure DevOps. You can adapt th...

  • 0 kudos
scottbisaillon
by New Contributor
  • 578 Views
  • 1 replies
  • 0 kudos

Databricks Running Jobs and Terraform

What happens to a currently running job when a workspace is deployed again using Terraform? Are the jobs paused/resumed, or are they left unaffected without any down time? Searching for this specific scenario doesn't seem to come up with anything and...

  • 578 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @scottbisaillon, When deploying a workspace again using Terraform, the behaviour regarding currently running jobs depends on the specific Terraform version and the platform you are using.   Let’s explore the details: Terraform Cloud (form...

  • 0 kudos
TinasheChinyati
by New Contributor
  • 771 Views
  • 1 replies
  • 0 kudos

Stream to stream join NullPointerException

I have a DLT pipeline running in continous mode. I have a stream to stream join which runs for the first 5hrs but then fails with a Null Pointer Exception. I need assistance to know what I need to do to handle this. my code is structured as below:@dl...

  • 771 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @TinasheChinyati, It looks like you’re encountering a Null Pointer Exception in your DLT pipeline when performing a stream-to-stream join. Let’s break down the issue and explore potential solutions: The error message indicates that the query te...

  • 0 kudos
liormayn
by New Contributor III
  • 1255 Views
  • 5 replies
  • 3 kudos

OSError: [Errno 78] Remote address changed

Hello:)as part of deploying an app that previously ran directly on emr to databricks, we are running experiments using LTS 9.1, and getting the following error: PythonException: An exception was thrown from a UDF: 'pyspark.serializers.SerializationEr...

  • 1255 Views
  • 5 replies
  • 3 kudos
Latest Reply
NandiniN
Honored Contributor
  • 3 kudos

Hi @liormayn , I can understand. I see the fix went on 20 March 2024, you would have to restart the clusters. Thanks!

  • 3 kudos
4 More Replies
Ikanip
by New Contributor II
  • 1426 Views
  • 4 replies
  • 2 kudos

Resolved! How to choose a compute, and how to find alternatives for the current compute being used?

We are using a compute for an Interactive Cluster in Production which incurs X amount of cost. We want to know what are the options available to use with near about the same processing power as the current compute but incur a cost of Y, which is less...

  • 1426 Views
  • 4 replies
  • 2 kudos
Latest Reply
raphaelblg
Contributor III
  • 2 kudos

Hello @Ikanip , You can utilize the Databricks Pricing Calculator to estimate costs. For detailed information on compute capacity, please refer to your cloud provider's documentation regarding Virtual Machine instance types.

  • 2 kudos
3 More Replies
ChristopherS5
by New Contributor
  • 474 Views
  • 1 replies
  • 0 kudos

Step-by-step guide to creating a Unity Catalog in Azure Databricks.

Hello everyone,There isn't an official document outlining the step-by-step procedure for enabling Unity Catalog in Azure Databricks.If anyone has created documentation or knows the process, please share it here.Thank you in advance.

  • 474 Views
  • 1 replies
  • 0 kudos
Latest Reply
PL_db
New Contributor III
  • 0 kudos

Setup Unity Catalog on Azure Unity Catalog best practices Which guidance/procedure are you missing?

  • 0 kudos
Hubcap7700
by New Contributor
  • 260 Views
  • 1 replies
  • 0 kudos

Native Slack Integration

Hi,Are there any plans to build native slack integration? I'm envisioning a one-time connector to Slack that would automatically populate all channels and users to select to use for example when configuring an alert notification. It is does not seem ...

  • 260 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Hubcap7700, If you have any further details or specific requirements, feel free to share, and I’ll be happy to assist! 

  • 0 kudos
sujan1
by New Contributor
  • 316 Views
  • 1 replies
  • 0 kudos

requirements.txt with cluster libraries

Cluster libraries are supported from version 15.0 - Databricks Runtime 15.0 | Databricks on AWS.How can I specify requirements.txt file path in the libraries in a job cluster in my workflow? Can I use relative path? Is it relative from the root of th...

  • 316 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

To specify the requirements.txt file path for libraries in a job cluster workflow in Databricks, you have a few options. Let’s break it down: Upload the requirements.txt File: First, upload your requirements.txt file to your Databricks workspace....

  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!