cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

samarth_solanki
by New Contributor II
  • 3512 Views
  • 4 replies
  • 0 kudos

Creating a python package that uses dbutils.secrets

Hello Databricks,I wanted to create python package which has a python script which has a class , this class is where we give scope and key and we het the secret. This package will be used inside a databricks notebook I want to use dbutils.secret for ...

  • 3512 Views
  • 4 replies
  • 0 kudos
Latest Reply
krishnab
New Contributor II
  • 0 kudos

from pyspark.dbutils import DBUtils from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() dbutils = DBUtils(spark)Adding this to the module file solves the problem.  

  • 0 kudos
3 More Replies
NIK251
by New Contributor III
  • 543 Views
  • 4 replies
  • 0 kudos

Delta Live Table Pipeline to EventHub

i want to read and load the data to eventhub. And there is an error message:org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 138.0 failed 4 times, most recent failure: Lost task 0.3 in stage 138.0 (TID 177) (10.139.6...

metrics-infor.png metrics-infor1.png
  • 543 Views
  • 4 replies
  • 0 kudos
Latest Reply
Sidhant07
Databricks Employee
  • 0 kudos

Hi, The error originates from the eventhub connector. Kindly check this with the EventHub Spark connector team. Please use the latest connector. https://github.com/Azure/azure-event-hubs-spark There are known issues with the Event hub connector like ...

  • 0 kudos
3 More Replies
naatyraja
by New Contributor
  • 172 Views
  • 1 replies
  • 0 kudos

%run working different in different taks of same job

I am using %run "/Workspace/Shared/SCPO_POC/Tredence-Dev-Phase3/Publix Weekly Sales POC - Main Folder/Weekly_Workflow_Parametrized/Get_Widgets_Values" before all my notebooks of a single job which has bunch of child jobs and child tasks... The % run ...

  • 172 Views
  • 1 replies
  • 0 kudos
Latest Reply
Sidhant07
Databricks Employee
  • 0 kudos

Hi, Can you please share the complete error?Can you try restarting the cluster, that might helo to mitigate the issue.

  • 0 kudos
NIK251
by New Contributor III
  • 935 Views
  • 2 replies
  • 1 kudos

Resolved! Delta Live Table Pipeline

I have the error message when try to create a delta live table pipeline.My error is: com.databricks.pipelines.common.errors.deployment.DeploymentException: Failed to launch pipeline cluster 1207-112912-8e84v9h5: Encountered Quota Exhaustion issue in ...

  • 935 Views
  • 2 replies
  • 1 kudos
Latest Reply
NIK251
New Contributor III
  • 1 kudos

Thanks sir, I solved it.

  • 1 kudos
1 More Replies
chethankumar
by New Contributor III
  • 214 Views
  • 1 replies
  • 0 kudos

How do I set up Databricks observability using AWS cloudwatch

How do I set up Databricks observability, including metrics and logging? I am using AWS-based Databricks and want to monitor it. I plan to use CloudWatch as the observability tool but couldn't find proper documentation to configure it.

  • 214 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @chethankumar, Setting up CloudWatch might required customer setup, by setting clusters using init script. You might want to consider contacting your account team to guide you through the process. Additional please find the post which could be use...

  • 0 kudos
HeckGunderson
by New Contributor
  • 218 Views
  • 1 replies
  • 0 kudos

Where can I quickly test custom metric queries?

I'm working on adapting some custom metrics to add to the tables' dashboard. Right now when I throw in a test query, I need to refresh metrics and it takes 5-10 minutes to let me know I've probably forgotten a parenthesis somewhere.Where can I test t...

  • 218 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @HeckGunderson, Have you tried running the code using notebook with your Databricks workspace? https://docs.databricks.com/en/notebooks/index.html Also you can do it via CLI using SQL connector. https://docs.databricks.com/ja/dev-tools/python-sql-...

  • 0 kudos
ismaelhenzel
by Contributor
  • 394 Views
  • 2 replies
  • 2 kudos

Databricks Materialized View - DLT Serverless Incremental

I'm currently working with Delta Live Tables, utilizing materialized views and serverless computing. While testing the incremental updates of materialized views, I've observed that deleting a record from the source table triggers a complete refresh o...

  • 394 Views
  • 2 replies
  • 2 kudos
Latest Reply
ismaelhenzel
Contributor
  • 2 kudos

EDIT: My Delta Lake table contains 136 columns. I initially tested with fewer columns, and both the updates and deletes were applied incrementally without issues. Specifically, I tested with 34 columns, and everything worked fine. However, when I inc...

  • 2 kudos
1 More Replies
pankajmehta1242
by New Contributor
  • 432 Views
  • 2 replies
  • 1 kudos

Can't download resources from Databricks Academy courses

I seem to be unable to download hands-on lab resources from the Databricks Academy courses. No link for such files (like .dbc, zip, slides, etc.) exist. It seems the problem is with the new UI. It was fine before. Please help. 

pankajmehta1242_0-1733316769310.png
  • 432 Views
  • 2 replies
  • 1 kudos
Latest Reply
BigRoux
Databricks Employee
  • 1 kudos

There was a recent change, students are no longer able to download dbc files.  The slides are still available in Academy but you can only view them from within Academy, you cannot download them. Hope this helps, Louis.

  • 1 kudos
1 More Replies
drag7ter
by Contributor
  • 449 Views
  • 3 replies
  • 0 kudos

Parameters in dashboards data section passing via asset bundles

A new functionality allows deploy dashboards with a asset bundles. Here is an example :# This is the contents of the resulting baby_gender_by_county.dashboard.yml file. resources: dashboards: baby_gender_by_county: display_name: "Baby gen...

  • 449 Views
  • 3 replies
  • 0 kudos
Latest Reply
drag7ter
Contributor
  • 0 kudos

variables:  catalog:    description: "Catalog name for the dataset"    default: "dev" parameters:        catalog: ${var.catalog}doesn't replace parameter values prod -> dev in json when it is being deployed"datasets": [    {      "displayName": "my_t...

  • 0 kudos
2 More Replies
amit_jbs
by New Contributor II
  • 1814 Views
  • 3 replies
  • 2 kudos

In databricks deployment .py files getting converted to notebooks

A critical issue has arisen that is impacting our deployment planning for our client. We have encountered a challenge with our Azure CI/CD pipeline integration, specifically concerning the deployment of Python files (.py). Despite our best efforts, w...

  • 1814 Views
  • 3 replies
  • 2 kudos
Latest Reply
amarantevitor94
New Contributor II
  • 2 kudos

Did you manage to find the solution? If so, could you share it? I have the same problem, and if I find the solution, I'll share it here.

  • 2 kudos
2 More Replies
martkev
by New Contributor
  • 643 Views
  • 1 replies
  • 0 kudos

how to enable unity catalog on azure databricks?

Hi guys We have an azure databricks workspace upgrade from standard to premium and we want to enable unity catalog such that we can federate with snowflake. the workspace is more than two years old. How to enable unity catalog? I am an admin but unde...

  • 643 Views
  • 1 replies
  • 0 kudos
Latest Reply
Rjdudley
Valued Contributor II
  • 0 kudos

Unity Catalog is configured at the account level, and then you add workspaces into it.  In your workspace selector, you should see "Manage account" at the bottom of the list (or go to https://accounts.azuredatabricks.net/).  If you can sign in to tha...

  • 0 kudos
musicman
by New Contributor II
  • 272 Views
  • 2 replies
  • 0 kudos

Getting scale to zero status of endpoint

Hello, I need to get the current scale to zero status of endpoint. How can I do that via python code? 

  • 272 Views
  • 2 replies
  • 0 kudos
Latest Reply
musicman
New Contributor II
  • 0 kudos

Hey, that worked. Thanks for you quick response.Cheers

  • 0 kudos
1 More Replies
VishalB
by New Contributor
  • 314 Views
  • 1 replies
  • 0 kudos

Not able to create or reset the password for Community edition.

I am trying to connect Databricks Community Edition with Power BI Desktop. When Power BI prompts for a username and password, I cannot provide the password because Databricks Community Edition uses a login system based on verification codes rather th...

  • 314 Views
  • 1 replies
  • 0 kudos
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @VishalB!You can find helpful resources for Databricks Community Edition here. If the available resource doesn’t resolve your concern, feel free to submit a ticket with Databricks Support team for further assistance.Thank you.

  • 0 kudos
Shawn_Eary
by Contributor
  • 462 Views
  • 3 replies
  • 1 kudos

Limited Results When trying to Access Delta Shared Tables from C#

This C# code only seems to return about 8 rows when it should return 100:// Per Databricks AI, you must run the following command if you don't // want to get an error about delta.enableDeltionVectors from C#: // ALTER TABLE [myCatalog].setest.billboa...

  • 462 Views
  • 3 replies
  • 1 kudos
Latest Reply
gchandra
Databricks Employee
  • 1 kudos

Can you try with the ODBC driver? https://docs.databricks.com/en/integrations/odbc/index.html  

  • 1 kudos
2 More Replies
ram0021
by New Contributor II
  • 1389 Views
  • 2 replies
  • 1 kudos

You haven't configured the CLI yet

I have coded pyfunc model code in Databricks notebook and deployed and served thr model thr endpoint. I tried to query the endpoint thr databricks notebook itself through below code but getting CLI error. Not sure why i am getting this error since I ...

  • 1389 Views
  • 2 replies
  • 1 kudos
Latest Reply
john_smith
New Contributor II
  • 1 kudos

Try this article which seems to have worked for most such use cases:https://medium.com/featurepreneur/solving-the-cli-configuration-error-in-databricks-d0462a96449f

  • 1 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors