Hi Community,I am currently evaluating various gradient boosting options on Databricks using production-level data, including the CatBoost Spark integration (ai.catboost:catboost-spark).I would love to hear from others who have successfully used this...
Hello, I need to develop some ML use case. I would like to understand if the serverless functionality unlocks any additional features or if it is mandatory for certain capabilities.Thank you!
Serverless functionality in Databricks is not mandatory for utilizing machine learning (ML) capabilities. However, it does unlock specific benefits and features that can enhance certain workflows. Here’s how serverless compute can add value, based on...
Hi everyone, I am encountering a problem when using ipywidgets with plotly on Databricks. I am trying to pass interactive arguments to a function and then plot with plotly. When I do the followingdef f(m, b) : plt.figure(2) x = np.linspace(-10,...
Hi, I've encountered a problem of serving a langchain model I just created successfully on Databricks.I was using the following code to set up a model in unity catalog:from mlflow.models import infer_signatureimport mlflowimport langchainmlflow.set_r...
I suspected the issue is coming from this small error I got: Got error: Must specify a chain Type in config. I used the chain_type="stuff" when building the langchain but I'm not sure how to fix it.
Hi,I want to read an Excel "xlsx" file. The excel file has several sheets and multi-row header. The original file format was "xlsm" and I changed the extension to "xlsx". I try the following code:filepath_xlsx = "dbfs:/FileStore/Sample_Excel/data.xl...
copying the data onto a newer file solved my issue. Likely issue related to files metadata!
I'm deploying a custom model using the `mlflow.pyfunc.PythonModel` class as described here. My model uses the `params` argument in the `predict` method to allow the user to choose some aspects of the model at inference time. For example:class CustomM...
That's great to hear @umesh_gattem ! Are you able to provide some example code for your predict function?I have tried specifying the params as part of the data_json request, but they are not being recognised by the model when it is called this way.Fo...
I am still lost on the Spark and Deep Learning model.If I have a (2D) time series that I want to use for e.g. an LSTM model. Then I first convert it to a 3D array and then pass it to the model. This is normally done in memory with numpy. But what hap...
Same problem as @imgaboy here, is the solution was to save into table our inputs after formating them ready to feed the lstm and just turn 2d to 3d via datagenerator??
How do I get MLflow child runs to appear as children of their parent run in the MLflow GUI, if I'm choosing my own experiment location instead of letting everything be written to the default experiment location?If I run the standard tutorial (https:/...
OK, here's more info about what's wrong, and a solution.I used additional parameter logging to determine that no matter how I adjust the parameters of the inner call to ```mlflow.start_run()```the `experiment_id` parameter of the child runs differs f...
I am trying to train an forecasting model along with Hyperparameters tuning with Hyperopt.I have multiple time series for "KEY" each of which I want to train a separate model. To do this I am using Spark's applyInPandas to tune and train model for ea...
Hi,I have a use case of developing an entire ML solution within Databricks starting from ingestion to inference and monitoring, but the problem is that we have unstructured data containing Images and Video for training the model using frameworks such...
Context: Running the train_model_py.py file stored in Databricks through GitHub Actions. The notebook reads the Unity Catalog tables for pre-processing and works fine when run through the Databricks UI. However, it gives an error when run through Git...
Hi @sagarb, It sounds like a permission issue or setup issue... what is the error you are hitting?
I have an issue with databricks-vectorsearch package. Version 0.51 suddenly stopped working this week because:It now expected me to provide azure_tenant_id in addition to service principal's client ID and secret.After supplying tenant ID, it showed s...
I am using MLFlow to register custom model (python model) in Unity Catalog, and Databricks Asset Bundle to create a serving endpoint for that custom model. I was able to create the serving endpoint using DABs, but I want to deploy the model by using ...
Hi @MLOperator Since model_serving_endpoints only accepts a version number of a served entity, I think that is not possible. However, the get-by-alias version API can be used to retrieve a version number from a model alias name. Then the model name...
Hello everyone ,Here are the sequence of steps i have followed:1. I have used petastorm to convert the spark dataframe to tf.datasetimport numpy as np# Read the Petastorm dataset and convert it to TensorFlow Datasetwith converter.make_tf_dataset() as...
The error occurs because make_tf_dataset() returns an inferred_schema_view object, which is a Petastorm wrapper representing the dataset schema. This object does not have a .numpy() attribute, so calling batch.numpy() will throw the AttributeError. ...
I am trying to configure an interactive EDA task as part of a job workflow. I'd like to be able to trigger a workflow, perform some basic analysis then proceed to a subsequent task. I haven't had any success freezing execution. Also, the job workflow...
Hello @cmd0160, Freezing job execution to perform interactive tasks directly within a job workflow is not natively supported in Databricks. The job workflow UI and the notebook UI serve different purposes, and the interactive capabilities you find in...
