- 1622 Views
- 1 replies
- 0 kudos
How to Efficiently Sync MLflow Traces and Asynchronous User Feedback with a Delta Table
I’m building a custom UI table (using Next.js and FastAPI) to display MLflow trace data from a Retrieval-Augmented Generation (RAG) application running on Databricks Managed MLflow 3.0. The table needs to show answer generation speed (from CHAT_MODEL...
- 1622 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello! Here are the answers to your questions: - Yes! See databricks managed mlflow tracing - enable production monitor or endpoint config to collect traces in a delta table - We have example code for implementing async feedback collection - Definit...
- 0 kudos
- 5479 Views
- 3 replies
- 2 kudos
SQL While do loops
Hello. Could you please suggest a workaround for a while do loop in Databricks SQL?WHILE LSTART>0 DO SET LSTRING=CONCAT(LSTRING, VSTRING2)Thank you in advance
- 5479 Views
- 3 replies
- 2 kudos
- 2 kudos
@elgeo Here are two alternatives.1. Use a recursive CTEWITH RECURSIVE loop_cte (lstart, lstring) AS ( SELECT 5 AS lstart, '' AS lstring UNION ALL SELECT lstart - 1, CONCAT(lstring, 'VSTRING2') FROM loop_cte WHERE lstart > 1 ) SELECT * FROM ...
- 2 kudos
- 11 Views
- 0 replies
- 0 kudos
Dabs Databricks asset bundles
Hi!I am relatively new to Dabs, but getting on quite well.I have managed to deploy both a job that uses a notebook defined in the bundle itself and a job that points to a notebook living in an azure devops git repo. While these are two viable solutio...
- 11 Views
- 0 replies
- 0 kudos
- 211 Views
- 3 replies
- 0 kudos
DAB + DLT destroy fails due to ownership/permissions mismatch
Hi all,We are running into an issue with Databricks Asset Bundles (DAB) when trying to destroy a DLT pipeline. Setup is as follows:Two separate service principals:Deployment SP: used by Azure DevOps for deploying bundles.Run_as SP: used for running t...
- 211 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @szymon_dybczak Thank you for you response. Coming back to this issue, how did Databricks design DAB's to be used with DLT's? If I have two service principals as stated in the original message, one for deployment and one for running the bundle res...
- 0 kudos
- 18 Views
- 1 replies
- 0 kudos
How to invoke Databricks AI Assistant from a notebook cell?
Hello Community,I am exploring the Databricks AI Assistant and wondering if there is a way to invoke or interact with it directly from a notebook cell instead of using the workspace sidebar UI.Is there any built-in command (like %assistant) to open o...
- 18 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @shashankB ,Here's a good documentation of all possible interaction with an assistant:What is Databricks Assistant? | Databricks on AWSBasically, is supports various modes:Chat mode: Get answers to your questions by chatting with the Assistant. It...
- 0 kudos
- 27 Views
- 1 replies
- 0 kudos
SQL Stored Procedure in Databricks
Hello, Is there a sql server equivalent stored procedure in Databricks which supports while loop along with delay as below. or are there any other alternative to achieve the same.while (select count(*) from schema.mart_daily with (nolock)) = 0 begin ...
- 27 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ashishasr ,Yes, support for stored procedures is in public preview:CREATE PROCEDURE | Databricks on AWSIn the definition of stored procedure you can use compound statement ( SQL compound statement (BEGIN ... END) with the definition of the SQL Pr...
- 0 kudos
- 63 Views
- 9 replies
- 6 kudos
Streaming- Results not getting updated on arrival of new files
Hi!I'm trying to stream some files using read_files.format("cloudFiles"). However, when new files arrive, the subsequent SQL query and monitoring graphs are not getting updated. Please suggest.
- 63 Views
- 9 replies
- 6 kudos
- 6 kudos
Hi @AanchalSoni If you are using .readStream, make sure you have set a trigger interval (e.g., .trigger(processingTime='1 minute'))
- 6 kudos
- 45 Views
- 3 replies
- 3 kudos
Resolved! Best Practices for Managing ACLs on Jobs and Job Clusters in Databricks
Hi all,I’m setting up access control for Databricks jobs and have two questions:Ephemeral Job Clusters: Since job clusters are created at runtime, is it best practice to set ACLs on the job itself? The /api/2.0/permissions/clusters/{cluster_id} endp...
- 45 Views
- 3 replies
- 3 kudos
- 31 Views
- 0 replies
- 0 kudos
Ingestion Framework
I would to like to update my ingestion framework that is orchestrated by ADF, running couples Databricks notebook and copying the data to DB afterwards. I want to rely everything on Databricks i though this could be the design:Step 1. Expose target t...
- 31 Views
- 0 replies
- 0 kudos
- 108 Views
- 9 replies
- 0 kudos
Proposal: Switch to Zstd Compression for Parquet to Reduce S3 Costs
We are thinking to change the Spark configuration for Parquet files to use zstd compression.Configuration: spark.sql.parquet.compression.codec = zstdThis will only affect new data written by our Spark jobs. All existing data will remain compressed wi...
- 108 Views
- 9 replies
- 0 kudos
- 0 kudos
Yes my believe is it should support 7.3 LTS as well, we will prove it with thorough testingThanks for the discussion. Cheers
- 0 kudos
- 32 Views
- 3 replies
- 3 kudos
Resolved! from pyspark.ml.stat import KolmogorovSmirnovTest is not working on Serverless compute.
Hi everyone,I am trying to run a Kolmogorov–Smirnov (KS) test on a Spark DataFrame column in Databricks using the built-in pyspark.ml.stat.KolmogorovSmirnovTest. The goal is to apply the KS test directly on Spark DataFrame data without converting it...
- 32 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @parthesh24 ,It looks more like KolmogorovSmirnovTest module under the hood is trying to access SparkContext which is not supported in serverless. You can check it yourself by trying to use sparkContext in serverless
- 3 kudos
- 11 Views
- 1 replies
- 0 kudos
Dashboard tagging
How can i tag dashbaord? i do not see any place to add tagging to it?
- 11 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @pinikrisher ,Unforutnately, you can't. Tagging is currently supported on catalogs, schemas, tables, table columns, volumes, views, registered models, and model version
- 0 kudos
- 105 Views
- 6 replies
- 2 kudos
databricks dashboard deployment error
Hi, i am trying to deploy a dashboard using a bundle and github action. but i am getting an error on CI even though the dashboard is deployed. im using the latest version of CLI from https://raw.githubusercontent.com/databricks/setup-cli/main/install...
- 105 Views
- 6 replies
- 2 kudos
- 2 kudos
@szymon_dybczak ,@nayan_wylde , I checked the permissions and the SP has 'can manage' access on the folder.
- 2 kudos
- 62 Views
- 2 replies
- 2 kudos
Resolved! Lakehouse monitoring dashboard shows no data
Hello, I am replicating the demo for Lakehouse monitoring found here: https://notebooks.databricks.com/demos/lakehouse-monitoring/index.htmlFor some reason, my dashboards show empty, i.e., they say 'no data' - like nothing fits the criteria from the ...
- 62 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @heli123 Can you share the image again? looks like didnt upload well.
- 2 kudos
- 59 Views
- 3 replies
- 3 kudos
Databricks model serving endpoint returns 403 Unauthorized access to workspace when using service
I deployed a simple Iris model in Databricks Model Serving and exposed it as an endpoint. I’m trying to query the endpoint using a service principal. I can successfully fetch the access token with the following databricks_token() function:def databri...
- 59 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @szymon_dybczak, Thanks for your comment.One of the admins in this workspace tried using the token generated via client id and secret, and were able to successfully get a response from the serving endpoint using this same above mentioned code.Coul...
- 3 kudos
| User | Count |
|---|---|
| 1616 | |
| 787 | |
| 394 | |
| 349 | |
| 287 |
