- 1726 Views
- 4 replies
- 0 kudos
Delta Live Table Pipeline to EventHub
i want to read and load the data to eventhub. And there is an error message:org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 138.0 failed 4 times, most recent failure: Lost task 0.3 in stage 138.0 (TID 177) (10.139.6...
- 1726 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi, The error originates from the eventhub connector. Kindly check this with the EventHub Spark connector team. Please use the latest connector. https://github.com/Azure/azure-event-hubs-spark There are known issues with the Event hub connector like ...
- 0 kudos
- 531 Views
- 1 replies
- 0 kudos
%run working different in different taks of same job
I am using %run "/Workspace/Shared/SCPO_POC/Tredence-Dev-Phase3/Publix Weekly Sales POC - Main Folder/Weekly_Workflow_Parametrized/Get_Widgets_Values" before all my notebooks of a single job which has bunch of child jobs and child tasks... The % run ...
- 531 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Can you please share the complete error?Can you try restarting the cluster, that might helo to mitigate the issue.
- 0 kudos
- 1574 Views
- 1 replies
- 0 kudos
How do I set up Databricks observability using AWS cloudwatch
How do I set up Databricks observability, including metrics and logging? I am using AWS-based Databricks and want to monitor it. I plan to use CloudWatch as the observability tool but couldn't find proper documentation to configure it.
- 1574 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @chethankumar, Setting up CloudWatch might required customer setup, by setting clusters using init script. You might want to consider contacting your account team to guide you through the process. Additional please find the post which could be use...
- 0 kudos
- 669 Views
- 1 replies
- 0 kudos
Where can I quickly test custom metric queries?
I'm working on adapting some custom metrics to add to the tables' dashboard. Right now when I throw in a test query, I need to refresh metrics and it takes 5-10 minutes to let me know I've probably forgotten a parenthesis somewhere.Where can I test t...
- 669 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @HeckGunderson, Have you tried running the code using notebook with your Databricks workspace? https://docs.databricks.com/en/notebooks/index.html Also you can do it via CLI using SQL connector. https://docs.databricks.com/ja/dev-tools/python-sql-...
- 0 kudos
- 3663 Views
- 0 replies
- 0 kudos
Problem when use synapse ml lightgbm with feature store
I'm running a model using LGBM with Spark, within Pipeline, but when I log it to MlFlow using the log_model feature store function, I can't replicate the pipeline in prediction. When I execute:predict = fs.score_batch(logged_model, df_pred) display(...
- 3663 Views
- 0 replies
- 0 kudos
- 4193 Views
- 2 replies
- 2 kudos
Databricks Materialized View - DLT Serverless Incremental
I'm currently working with Delta Live Tables, utilizing materialized views and serverless computing. While testing the incremental updates of materialized views, I've observed that deleting a record from the source table triggers a complete refresh o...
- 4193 Views
- 2 replies
- 2 kudos
- 2 kudos
EDIT: My Delta Lake table contains 136 columns. I initially tested with fewer columns, and both the updates and deletes were applied incrementally without issues. Specifically, I tested with 34 columns, and everything worked fine. However, when I inc...
- 2 kudos
- 1303 Views
- 2 replies
- 3 kudos
Can't download resources from Databricks Academy courses
I seem to be unable to download hands-on lab resources from the Databricks Academy courses. No link for such files (like .dbc, zip, slides, etc.) exist. It seems the problem is with the new UI. It was fine before. Please help.
- 1303 Views
- 2 replies
- 3 kudos
- 3 kudos
There was a recent change, students are no longer able to download dbc files. The slides are still available in Academy but you can only view them from within Academy, you cannot download them. Hope this helps, Louis.
- 3 kudos
- 14932 Views
- 5 replies
- 3 kudos
Resolved! Databricks Monitoring
Hi Everyone,can someone suggest me to select best native job monitoring tool available in Databricks for fulfill my need;we need to monitor the following:Number of failed jobs and its name : for last 24 hoursTable that are not getting dataLatest inge...
- 14932 Views
- 5 replies
- 3 kudos
- 3 kudos
You can use the databricks API to collect all required information.. https://docs.databricks.com/api/workspace/jobs/listLoad the output to a delta table. Use the Databricks dashboards in displaying this data.. schedule the job for loading the databri...
- 3 kudos
- 2043 Views
- 1 replies
- 0 kudos
how to enable unity catalog on azure databricks?
Hi guys We have an azure databricks workspace upgrade from standard to premium and we want to enable unity catalog such that we can federate with snowflake. the workspace is more than two years old. How to enable unity catalog? I am an admin but unde...
- 2043 Views
- 1 replies
- 0 kudos
- 0 kudos
Unity Catalog is configured at the account level, and then you add workspaces into it. In your workspace selector, you should see "Manage account" at the bottom of the list (or go to https://accounts.azuredatabricks.net/). If you can sign in to tha...
- 0 kudos
- 939 Views
- 2 replies
- 0 kudos
Getting scale to zero status of endpoint
Hello, I need to get the current scale to zero status of endpoint. How can I do that via python code?
- 939 Views
- 2 replies
- 0 kudos
- 1410 Views
- 1 replies
- 0 kudos
Not able to create or reset the password for Community edition.
I am trying to connect Databricks Community Edition with Power BI Desktop. When Power BI prompts for a username and password, I cannot provide the password because Databricks Community Edition uses a login system based on verification codes rather th...
- 1410 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @VishalB!You can find helpful resources for Databricks Community Edition here. If the available resource doesn’t resolve your concern, feel free to submit a ticket with Databricks Support team for further assistance.Thank you.
- 0 kudos
- 1514 Views
- 3 replies
- 1 kudos
Limited Results When trying to Access Delta Shared Tables from C#
This C# code only seems to return about 8 rows when it should return 100:// Per Databricks AI, you must run the following command if you don't // want to get an error about delta.enableDeltionVectors from C#: // ALTER TABLE [myCatalog].setest.billboa...
- 1514 Views
- 3 replies
- 1 kudos
- 1 kudos
Can you try with the ODBC driver? https://docs.databricks.com/en/integrations/odbc/index.html
- 1 kudos
- 2405 Views
- 2 replies
- 1 kudos
You haven't configured the CLI yet
I have coded pyfunc model code in Databricks notebook and deployed and served thr model thr endpoint. I tried to query the endpoint thr databricks notebook itself through below code but getting CLI error. Not sure why i am getting this error since I ...
- 2405 Views
- 2 replies
- 1 kudos
- 1 kudos
Try this article which seems to have worked for most such use cases:https://medium.com/featurepreneur/solving-the-cli-configuration-error-in-databricks-d0462a96449f
- 1 kudos
- 1414 Views
- 2 replies
- 0 kudos
Is there a way to clone an AIBI Dashboard?
We moved some of our processes from notebooks to dashboards, but it looks like I cannot clone a dashboard like I can clone a notebook.
- 1414 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, Please follow these steps to clone AIBI dashboard. 1. Open a dashboard and select the draft mode. 2. Click the kebab menu (three vertical dots) and select the clone option.
- 0 kudos
- 10402 Views
- 6 replies
- 2 kudos
Alter table to add/update multiple column comments
I was wondering if there's a way to alter table and add/update comments for multiple columns at once using SQL or API calls. For instance - ALTER TABLE <table_name> CHANGE COLUMN <col1> COMMENT '<comment1>',CHANGE COLUMN <col2> COMMENT 'comment2' ; ...
- 10402 Views
- 6 replies
- 2 kudos
- 2 kudos
@filipniziol Is batch update maybe a preview feature or something that was only added in a recent version of the databricks runtime? I'm seeing the same error as @bharathjs in DBR 14.3, 16.0 ML and serverless V1
- 2 kudos
