- 7864 Views
- 2 replies
- 4 kudos
Dynamic variable and multi-instance tasks.
1. How to pass dynamic variable values like "sysdate" to a job parameters, so that it will automatically take the updated values on the fly.2. How to run multiple instance of set of tasksin a job (for different parameters). For e.g the same pipeline ...
- 7864 Views
- 2 replies
- 4 kudos
- 4 kudos
Hey Maverick1,Did you find a solution for your second question?I have also same approach. In databricks, it has workflows, job clusters, tasks etc.I'm trying to do creating one job cluster with one configuration or specification which has a workflow ...
- 4 kudos
- 7578 Views
- 7 replies
- 2 kudos
ModuleNotFoundError: No module named 'databricks.feature_store.mlflow_model'
I'm using runtime DBR 12.2 LTS MLGetting this error when running. import mlflow logged_model = 'runs:/.../model' #my run id # Load model as a PyFuncModel. loaded_model = mlflow.pyfunc.load_model(logged_model) # Predict on a Pandas DataFrame. impor...
- 7578 Views
- 7 replies
- 2 kudos
- 2 kudos
Hi @SOlivero and @hollybaker354,Can you check by setting the maxDepth to the default value and see how it works?
- 2 kudos
- 15507 Views
- 3 replies
- 3 kudos
Resolved! Issues importing mediapipe TypeError: 'numpy._DTypeMeta' object is not subscriptable
TypeError: 'numpy._DTypeMeta' object is not subscriptableI have tried upgrading pip and installing newer and older versions of NumPy.
- 15507 Views
- 3 replies
- 3 kudos
- 3 kudos
I ran into the same issue. Before installing OpenCV I was on Numpy 1.21.5 in my notebook. After upgrading to 1.23, the error went away for me.
- 3 kudos
- 5301 Views
- 2 replies
- 3 kudos
Can't select cluster when create AutoML experiment
I'm trying to create my experiment using AutoML. I have a running cluster using 12.2 LTS. However, it isn't available as an option for me. How could I solve it? Thank you
- 5301 Views
- 2 replies
- 3 kudos
- 3 kudos
@HAKO411 AutoML needs Databricks Runtime 9.1 ML or above. For time series forecasting, you will need Databricks Runtime 10.0 ML or above. Looks like you are using 12.2 LTS, a non-ML version. Using 12.2 ML LTS should resolve your issue.
- 3 kudos
- 13958 Views
- 4 replies
- 1 kudos
Spark connector to mongodb - mongo-spark-connector_2.12:10.1.1
Hello, I´ve added a library to the cluster and it appears in SPARK UI as Added By Userspark://10.139.64.4:43001/jars/addedFile307892533757162075org_mongodb_spark_mongo_spark_connector_2_12_10_1_1-98946.jarAdded By UserI'm trying to connect using the ...
- 13958 Views
- 4 replies
- 1 kudos
- 1 kudos
@DmytroSokhach I think it works if you change mongo to mongodb in the options. and use spark.mongodb.read.connection.uri instead of spark.mongodb.input.uri as @silvadev suggested.
- 1 kudos
- 4034 Views
- 1 replies
- 0 kudos
Use on-premise MinIO as an artifact store in experiment
Hi. I'm trying to use managed MLflow with our own MinIO as an artifact storage. I can see that there is a description about storage options at landing page and there is an input for artifact store URI when creating empty experiment in databicks works...
- 4034 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks. I will post there if the feature I asked is doesn't exist.Anyway, the feature I asked about is clearly described on the landing page. I'm looking for documentation for that feature.
- 0 kudos
- 17025 Views
- 6 replies
- 3 kudos
How to apply Primary Key constraint in Delta Live Table?
In this blog I can see for dimension and fact tables, the primary key constraint has been applied. Following is the example:-- Store dimensionCREATE OR REPLACE TABLE dim_store( store_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, business_key ...
- 17025 Views
- 6 replies
- 3 kudos
- 3 kudos
@SRK Please see a copy of this answer on stackoverflow here. You can use DLT Expectations to have this check (see my previous answer if you're using SQL and not Python):@dlt.table(name="table1",)def create_df():schema = T.StructType([T.StructField("i...
- 3 kudos
- 1671 Views
- 0 replies
- 0 kudos
Classroom Setup Error in LLM Course
Hi All, I encountered this error when running the classroom setup for the LLM Course and would love to know if there's a subscription that Databricks offers to learn and practice these pieces of training.Course name: LLM : Foundation Models from the ...
- 1671 Views
- 0 replies
- 0 kudos
- 4654 Views
- 2 replies
- 0 kudos
Cannot set experiment in a non-ML runtime
Hello,If we:%pip install mlflow import mlflow mlflow.set_experiment(experiment_name = '/Shared/xx')we get:InvalidConfigurationError: You haven't configured the CLI yet! Please configure by entering `/databricks/python_shell/scripts/db_ipykernel_launc...
- 4654 Views
- 2 replies
- 0 kudos
- 3299 Views
- 1 replies
- 0 kudos
NVIDIA driver update
I want to update the cuda driver for the NVIDIA tesla T4 GPU on the cluster. using the following command%shsudo apt-get --purge remove "*nvidia*"sudo /usr/bin/nvidia-uninstallwget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x8...
- 3299 Views
- 1 replies
- 0 kudos
- 30792 Views
- 11 replies
- 12 kudos
Resolved! INFORMATION_SCHEMA IS NOT POPULATED WITH TABLE INFORMATION
I have created a metastore and within that metastore i have created multiple schemas and tables underlying it but none of table details is visible from information schema. All the tables are empty.Could you please let me know if I am missing here. Be...
- 30792 Views
- 11 replies
- 12 kudos
- 12 kudos
I noticed this issue is currently caused when you rename a catalog. The contents of <catalog>.information_schema are all views like this:SELECT * FROM system.information_schema.columns WHERE table_catalog = '<catalog>'If you rename the catalog...
- 12 kudos
- 4500 Views
- 1 replies
- 1 kudos
AutoML Trials Failing
Sometimes an AutoML experiment will have all trials fail and I cannot figure out what is causing it. Each individual run reports a validation f1 value but the source notebook is not available so I cannot track down the error. This seems to happen at ...
- 4500 Views
- 1 replies
- 1 kudos
- 10354 Views
- 2 replies
- 2 kudos
MLFlow model loading taking long time and "model serving" failing during init
I am trying to load a simple Minmaxscaler model that was logged as a run through spark's ML Pipeline api for reuse. On average it takes 40+ seconds just to load the model with the following example: This is fine and the model transforms my data corre...
- 10354 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello,Any solutions found for this issue?I'm serving up a large number of models at a time, but since we converted to PySpark (due to our data demands), the mlflow.spark.load_model() is taking hours.Part of the reason to switch to spark was to help w...
- 2 kudos
- 2137 Views
- 1 replies
- 0 kudos
- 2137 Views
- 1 replies
- 0 kudos
- 0 kudos
Large Language Models (LLMs) revolutionize the insurance sector, automating support and enhancing accuracy across claims and underwriting. They're crucial for market analysis processing extensive textual data, and we welcome your insights on deployin...
- 0 kudos
- 5578 Views
- 1 replies
- 1 kudos
Help installing R package "sf" azure Databricks cluster
I need help installing the R package "sf" on the azure Databricks cluster. Any help will be greatly appreciated.I'm getting the error below when I try installing the "sf" package on a cluster."Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Me...
- 5578 Views
- 1 replies
- 1 kudos
