- 7499 Views
- 6 replies
- 3 kudos
How to apply Primary Key constraint in Delta Live Table?
In this blog I can see for dimension and fact tables, the primary key constraint has been applied. Following is the example:-- Store dimensionCREATE OR REPLACE TABLE dim_store( store_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, business_key ...
- 7499 Views
- 6 replies
- 3 kudos
- 3 kudos
@SRK Please see a copy of this answer on stackoverflow here. You can use DLT Expectations to have this check (see my previous answer if you're using SQL and not Python):@dlt.table(name="table1",)def create_df():schema = T.StructType([T.StructField("i...
- 3 kudos
- 1197 Views
- 0 replies
- 0 kudos
Classroom Setup Error in LLM Course
Hi All, I encountered this error when running the classroom setup for the LLM Course and would love to know if there's a subscription that Databricks offers to learn and practice these pieces of training.Course name: LLM : Foundation Models from the ...
- 1197 Views
- 0 replies
- 0 kudos
- 2972 Views
- 2 replies
- 0 kudos
Cannot set experiment in a non-ML runtime
Hello,If we:%pip install mlflow import mlflow mlflow.set_experiment(experiment_name = '/Shared/xx')we get:InvalidConfigurationError: You haven't configured the CLI yet! Please configure by entering `/databricks/python_shell/scripts/db_ipykernel_launc...
- 2972 Views
- 2 replies
- 0 kudos
- 1890 Views
- 1 replies
- 0 kudos
NVIDIA driver update
I want to update the cuda driver for the NVIDIA tesla T4 GPU on the cluster. using the following command%shsudo apt-get --purge remove "*nvidia*"sudo /usr/bin/nvidia-uninstallwget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x8...
- 1890 Views
- 1 replies
- 0 kudos
- 0 kudos
I want to update the cuda driver for the NVIDIA tesla T4 GPU on the cluster. using the following command%shsudo apt-get --purge remove "*nvidia*"sudo /usr/bin/nvidia-uninstallwget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x8...
- 0 kudos
- 17126 Views
- 11 replies
- 12 kudos
Resolved! INFORMATION_SCHEMA IS NOT POPULATED WITH TABLE INFORMATION
I have created a metastore and within that metastore i have created multiple schemas and tables underlying it but none of table details is visible from information schema. All the tables are empty.Could you please let me know if I am missing here. Be...
- 17126 Views
- 11 replies
- 12 kudos
- 12 kudos
I noticed this issue is currently caused when you rename a catalog. The contents of <catalog>.information_schema are all views like this:SELECT * FROM system.information_schema.columns WHERE table_catalog = '<catalog>'If you rename the catalog...
- 12 kudos
- 2669 Views
- 1 replies
- 1 kudos
AutoML Trials Failing
Sometimes an AutoML experiment will have all trials fail and I cannot figure out what is causing it. Each individual run reports a validation f1 value but the source notebook is not available so I cannot track down the error. This seems to happen at ...
- 2669 Views
- 1 replies
- 1 kudos
- 1 kudos
Sometimes an AutoML experiment will have all trials fail and I cannot figure out what is causing it. Each individual run reports a validation f1 value but the source notebook is not available so I cannot track down the error. This seems to happen at ...
- 1 kudos
- 4748 Views
- 2 replies
- 2 kudos
MLFlow model loading taking long time and "model serving" failing during init
I am trying to load a simple Minmaxscaler model that was logged as a run through spark's ML Pipeline api for reuse. On average it takes 40+ seconds just to load the model with the following example: This is fine and the model transforms my data corre...
- 4748 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello,Any solutions found for this issue?I'm serving up a large number of models at a time, but since we converted to PySpark (due to our data demands), the mlflow.spark.load_model() is taking hours.Part of the reason to switch to spark was to help w...
- 2 kudos
- 1184 Views
- 1 replies
- 0 kudos
- 1184 Views
- 1 replies
- 0 kudos
- 0 kudos
Large Language Models (LLMs) revolutionize the insurance sector, automating support and enhancing accuracy across claims and underwriting. They're crucial for market analysis processing extensive textual data, and we welcome your insights on deployin...
- 0 kudos
- 3366 Views
- 0 replies
- 0 kudos
MlflowException: Unable to download model artifacts in Databricks while registering model with MLflo
I am attempting to log, register, and deploy a finetuned GPT2 model in Databricks. While I have been able to get my logging code to run, when I try to run my registration code, I get an MlflowException error.Here is my model logging code.mlflow.set_r...
- 3366 Views
- 0 replies
- 0 kudos
- 4109 Views
- 1 replies
- 1 kudos
Help installing R package "sf" azure Databricks cluster
I need help installing the R package "sf" on the azure Databricks cluster. Any help will be greatly appreciated.I'm getting the error below when I try installing the "sf" package on a cluster."Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Me...
- 4109 Views
- 1 replies
- 1 kudos
- 1 kudos
I need help installing the R package "sf" on the azure Databricks cluster. Any help will be greatly appreciated.I'm getting the error below when I try installing the "sf" package on a cluster."Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Me...
- 1 kudos
- 1320 Views
- 0 replies
- 0 kudos
Terraform - Creating Jobs
Hello, Does anyone know how to create a job via Terraform, that automatically overwrites/updates an existing job with the same name?Tried a few different methods but there doesn't seem to be a clean approach. Wondering if anyone has worked this out?I...
- 1320 Views
- 0 replies
- 0 kudos
- 1458 Views
- 0 replies
- 0 kudos
databricks.mrm import ModelRiskApi
Hi, I am trying to import/access this module by DBX. However I have issue importing it and I don't find any blog posts/more information about this module beside the public github repo
- 1458 Views
- 0 replies
- 0 kudos
- 10073 Views
- 7 replies
- 2 kudos
Resolved! Served model creation failed
I have a model registered in unity catalog which works fine and I can load / run and get results returned. I wanted to create a serving endpoint but when I try I get this error.Served model creation failed for served model 'model_name', config versi...
- 10073 Views
- 7 replies
- 2 kudos
- 2 kudos
@kashy Looks like the model is not correctly referenced while loading. You should reference the path of the model till ‘model-best’, which is the top-level directory. loaded_model = mlflow.spacy.load_model("</path/to/your/model/>/model-best")
- 2 kudos
- 2972 Views
- 3 replies
- 1 kudos
Cluster termination issue (It was working fine with same code previously)
Hi, I am running arima code on databricks commmunity edition. Last time, it worked but now the cluster gets terminated at this command and says its completed. but i cant further plot the predictions since the cluster is terminated. # Fit the AutoARI...
- 2972 Views
- 3 replies
- 1 kudos
- 1 kudos
@Humi1245 You can view the event log of the cluster to understand the cause of the cluster termination. If it is due to gc issues, then we have to understand the reason behind high driver memory utilization which caused the GC. If it's due to inact...
- 1 kudos
- 4762 Views
- 2 replies
- 1 kudos
Resolved! Model serving Ran out of memory
Hi fellows, I encountered memory(?) error when sending POST requests to my real-time endpoint, and I'm unable to find hardware setting to increase memory, as suggested by the Service Logs (below). Steps to Repro:(1) I registered a custom MLFlow model...
- 4762 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you for your responses, @Annapurna_Hiriy and @Retired_modIndeed, it appeared that my original model (~800MB) was too big for the current server. Based on your suggestion, I made a simpler/smaller model for this project, and then I was able to d...
- 1 kudos
