- 11698 Views
- 5 replies
- 2 kudos
Trying out Dolly - how to load pytorch_model.bin so it's not downloading it every time the cluster is restarted
Hi, I am new to LLM and am curious to try it out. I did the following code to test from the databricks website:import torch from transformers import pipeline instruct_pipeline = pipeline(model="databricks/dolly-v2-12b", torch_dtype=torch.bfloat16, tr...
- 11698 Views
- 5 replies
- 2 kudos
- 2 kudos
Just set the HF cache dir to a persistent path on /dbfs:import os os.environ['TRANSFORMERS_CACHE'] = "/dbfs/..."
- 2 kudos
- 911 Views
- 0 replies
- 0 kudos
I found these videos to be beneficial CI/CD with Azure Dev OpsCI/CD with GitHub Actions
I found these videos to be beneficialCI/CD with Azure Dev OpsCI/CD with GitHub Actions
- 911 Views
- 0 replies
- 0 kudos
- 3301 Views
- 4 replies
- 3 kudos
Azure Databricks Capabilities: My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.
hi,We have Real time streaming usecase where we have to build pipeline using Azure Databricks.My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.Can you please provide c...
- 3301 Views
- 4 replies
- 3 kudos
- 3 kudos
I found these youtube videos to be beneficial. CI/CD with Azure Dev Ops Terraform Enablement - Part 1 of 2
- 3 kudos
- 5755 Views
- 6 replies
- 1 kudos
FFmpeg frame extraction explodes memory, how to mitigate?
For a computer vision project, my raw data consists of encrypted videos (60fps) stored in Azure Blob Storage. In order to have the data usable for model training, I need to do some preprocessing and for that I need the video split into individual fra...
- 5755 Views
- 6 replies
- 1 kudos
- 1 kudos
In the end, I decided to change around the workflow so it is as efficient as I could imagine it:Extract frames of video files in a containerized application somewhere running ffmpeg and storing the resulting frames in a parquet file in blob storage (...
- 1 kudos
- 2124 Views
- 2 replies
- 0 kudos
Can an HMS-managed table be upgraded to Unity Catalog?
As the question states, I am not getting the option to upgrade managed tables on UC. Is that possible, I can't find anything on the documentation?
- 2124 Views
- 2 replies
- 0 kudos
- 7182 Views
- 3 replies
- 2 kudos
Resolved! Issues loading .txt files from DBFS into Langchain TextLoader()
Hello,I am working on building a Langchain QA application in Databricks. I currently have 13 .txt files loaded into the DBFS and am trying to read them in iteratively with TextLoader(), load them into the RecursiveCharacterTextSplitter() from Langcha...
- 7182 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @David Kersey​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 2 kudos
- 5822 Views
- 0 replies
- 1 kudos
Failure in Databricks Serving endpoint <build log says Pip failed due to conflicting dependency.>
Hello All,We are trying to deploy some models using Databricks Serving endpoint, But while deploying the artifact created during experiment run the serving endpoint build log says Pip failed due to conflicting dependency.The model is logged in experi...
- 5822 Views
- 0 replies
- 1 kudos
- 7563 Views
- 1 replies
- 2 kudos
Resolved! error in running a LLM model in pyfunc.spark_udf
error in running a LLM model in pyfunc.spark_udf
- 7563 Views
- 1 replies
- 2 kudos
- 2 kudos
Solution: Please find below the example. Creating a secret and scope is a one time activity once we create a scope and secret we can access the token using any notebook or cluster in the workspace as shown below. After creating a secret if we want ...
- 2 kudos
- 11141 Views
- 5 replies
- 2 kudos
Resolved! mlflow down in workspace?
Mlflow started failing all of a sudden for no reason when logged in databricks community edition:Any idea why this is happening or is there a way to restart the mlflow server?
- 11141 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Zheng Han​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...
- 2 kudos
- 2546 Views
- 2 replies
- 1 kudos
Mlflow :loading script failed !!
I am using mlflow to track experimentation with databricks but todaty i tried to access my experimetations in dtabricks and i face the error .
- 2546 Views
- 2 replies
- 1 kudos
- 1 kudos
I didn't manage to solve the error . I guess it is related to databricks community cloud because I tested with another account and it all the same.
- 1 kudos
- 8778 Views
- 2 replies
- 2 kudos
Resolved! AutoML with Stratified Sampling
Is it possible to use a stratified sampling strategy for the train/test/validate splits that the automl library does? We are working in a context where we need to segregate certain groups from the training and test sets to see how our models general...
- 8778 Views
- 2 replies
- 2 kudos
- 2 kudos
HI @Jared Webb​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 2 kudos
- 6883 Views
- 2 replies
- 1 kudos
Isolation Forest prediction failing DLT pipeline, the same model works fine when prediction is done outside DLT pipeline.
Hey community membersI am new to Databricks and was building a simple DLT pipleine that loads data from S3 and runs an Isolation forest prediction to detect anomalies. The model has been stored in Model Registry. Here's the code for the pipeline:@dlt...
- 6883 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Mukul Degweker​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 1 kudos
- 1533 Views
- 1 replies
- 2 kudos
Sample Archietcture for Databricks MLOps
Do anyone have sample architectures for Mlops using Databricks and other possible variations of architecture ?
- 1533 Views
- 1 replies
- 2 kudos
- 2 kudos
@Saurabh Singh​ This is well documented here:https://www.databricks.com/blog/2022/06/22/architecting-mlops-on-the-lakehouse.htmlPlease see: Reference architecture for MLOpsFurther refrences: Refer to The Big Book of MLOps for more discussion of the a...
- 2 kudos
- 900 Views
- 0 replies
- 1 kudos
Hello Everyone, I am thrilled to announce that we have our 5th winner for the raffle contest - @Emilia​. Please join me in congratulating her on this ...
Hello Everyone,I am thrilled to announce that we have our 5th winner for the raffle contest - @Emilia​. Please join me in congratulating her on this remarkable achievement!Your dedication and hard work have paid off, and we are delighted to have you ...
- 900 Views
- 0 replies
- 1 kudos
- 1678 Views
- 1 replies
- 1 kudos
When should you use the directory listing vs file notification
We are using Delta Live Tables for running ingestion pipelines and have come across the two options for the autoloader "file notification" vs "directory listing" this is reflected in the option cloudFiles.useIncrementalListing. We are wondering what ...
- 1678 Views
- 1 replies
- 1 kudos
- 1 kudos
@Bennett Lambert​ :The choice between using "file notification" vs "directory listing" for the autoloader in Delta Live Tables depends on your specific use case and requirements. Here are some general guidelines:Use file notification if you need real...
- 1 kudos
