cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Science & Machine Learning

Forum Posts

User16752245767
by Contributor
  • 682 Views
  • 0 replies
  • 5 kudos

youtu.be

I'm Avi, a Solutions Architect at Databricks working at the intersection of Data Engineering and Machine Learning.Streaming data processing has moved from niche to mainstream, and deploying machine learning models in such data streams opens up a mult...

  • 682 Views
  • 0 replies
  • 5 kudos
Kristof
by New Contributor III
  • 5454 Views
  • 3 replies
  • 3 kudos

Resolved! Spark Error/Exception Handling

I am creating new application and looking for ideas how to handle exceptions in Spark, for example ThreadPoolExecution. Are there any good practice in terms of error handling and dealing with specific exceptions ?

  • 5454 Views
  • 3 replies
  • 3 kudos
Latest Reply
Shalabh007
Honored Contributor
  • 3 kudos

@Krzysztof Nojman​ Can you please click on the "Select As Best" button if you find the information provided helps resolve your question.

  • 3 kudos
2 More Replies
matte
by New Contributor III
  • 8381 Views
  • 7 replies
  • 16 kudos

Resolved! Way of using pymc.model_to_graphviz into a Databricks notebook

Hi everybody,I created a simple bayesian model using the pymc library in Python. I would like to graphically represent my model using the pymc.model_to_graphviz(model=model) method.However, it seems it does not work within a databrcks notebook, even ...

  • 8381 Views
  • 7 replies
  • 16 kudos
Latest Reply
Own
Contributor
  • 16 kudos

%sh apt install -y graphviz

  • 16 kudos
6 More Replies
elgeo
by Valued Contributor II
  • 3431 Views
  • 1 replies
  • 4 kudos

Resolved! Insert into delta table fails

Hello experts. We are trying to execute an insert command with less columns than the target table:Insert into table_name( col1, col2, col10)Select col1, col2, col10from table_name2However the above fails with:Error in SQL statement: DeltaAnalysisExce...

  • 3431 Views
  • 1 replies
  • 4 kudos
Latest Reply
UmaMahesh1
Honored Contributor III
  • 4 kudos

Hi @ELENI GEORGOUSI​ Yes. When you are doing an insert, your provided schema should match with the target schema else it would throw an error.But you can still insert the data using another approach. Create a dataframe with your data having less colu...

  • 4 kudos
jonathan-dufaul
by Valued Contributor
  • 1484 Views
  • 3 replies
  • 1 kudos

How does mlflow determine if a pyfunc model uses SparkContext?

I've been getting this error pretty regularly while working with mlflow:"It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that ...

  • 1484 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

I checked the page and it looks like there is no integration with Datarobot and Datarobot doesn't contribute to mlflow. https://mlflow.org/ has all the integrations listed

  • 1 kudos
2 More Replies
ajeet1080
by New Contributor III
  • 1485 Views
  • 1 replies
  • 2 kudos

Resolved! Unable to create feature table using databricks API .FeatureStoreClient()

I am following example steps from databricks documentation https://docs.databricks.com/_static/notebooks/machine-learning/feature-store-taxi-example.htmlI am using Feature Store client v0.3.6 and above.However on trying to create feature table with f...

Screenshot 2022-11-29 at 2.10.43 PM pickup_features dataframe screenshot dropoff_features dataframe screenshot
  • 1485 Views
  • 1 replies
  • 2 kudos
Latest Reply
ajeet1080
New Contributor III
  • 2 kudos

After much digging, observed i was using standard runtime. Once i switched to ML runtime of databricks, issue was resolved. To use Feature Store capability, ensure that you select a Databricks Runtime ML version from the Databricks Runtime Version dr...

  • 2 kudos
MA
by New Contributor II
  • 912 Views
  • 1 replies
  • 4 kudos

Stream data from Delta tables replicated with Fivetran into DLT

I'm attempting to stream into a DLT pipeline with data replicated from Fivetran directly into Delta tables in another database than the one that the DLT pipeline uses. This is not an aggregate, and I don't want to recompute the entire data model eac...

  • 912 Views
  • 1 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @M A​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon. Thanks

  • 4 kudos
Siebert_Looije
by Contributor
  • 837 Views
  • 1 replies
  • 1 kudos

What is the best way to deal with pymc3 in MLFLOW models in databricks?

Last week, we started with using mlflow within databricks. The bayesian models that we are using right now are the pymc3 models (https://docs.pymc.io/en/v3/index.html).We could use the experiment feature of databricks/mlflow to save the models as an ...

  • 837 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Siebert Looije​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon. Thanks.

  • 1 kudos
jonathan-dufaul
by Valued Contributor
  • 757 Views
  • 0 replies
  • 1 kudos

is it possible to change the boilerplate code on a logged/saved pyfunc mlflow model?

When I log a pyfunc mlflow model, it generates a page that has this helpful code for using the model in production. Make Predictions Predict on a Spark DataFrame: import mlflow from pyspark.sql.functions import struct, col logged_model = 'runs:/1d......

  • 757 Views
  • 0 replies
  • 1 kudos
Slalom_Tobias
by New Contributor III
  • 2205 Views
  • 3 replies
  • 3 kudos

Resolved! ML Practioner | ML 11 - XGBoost notebook | cannot import keras.applications.resnet50

the following code...from sparkdl.xgboost import XgboostRegressorfrom pyspark.ml import Pipelineparams = {"n_estimators": 100, "learning_rate": 0.1, "max_depth": 4, "random_state": 42, "missing": 0}xgboost = XgboostRegressor(**params)pipeline = Pipel...

  • 2205 Views
  • 3 replies
  • 3 kudos
Latest Reply
Prabakar
Esteemed Contributor III
  • 3 kudos

You need to choose the runtime for ML instead of the standard.

  • 3 kudos
2 More Replies
elgeo
by Valued Contributor II
  • 1188 Views
  • 1 replies
  • 1 kudos

Pass parameter from python to SQL - Null result

Hello. Could someone please explain at the below example, why having the prefix "da" at the parameter name allows us to select the parameter value but not having it returns to a null value?Correct valueNull value Thank you in advance

UDF_OK UDF_NOT_OK
  • 1188 Views
  • 1 replies
  • 1 kudos
Latest Reply
elgeo
Valued Contributor II
  • 1 kudos

Any insight on this? Thank you!

  • 1 kudos
anthonylavado
by New Contributor III
  • 1559 Views
  • 3 replies
  • 7 kudos

Can't Add Cluster-scoped Init Script to Model Serving Cluster

Similar to this other question: https://community.databricks.com/s/question/0D58Y00008hahwuSAA/cant-edit-the-cluster-created-by-mlflow-model-servingWe're using Azure Databricks, and have a model that requires a WHL to be downloaded from a private add...

  • 1559 Views
  • 3 replies
  • 7 kudos
Latest Reply
939772
New Contributor III
  • 7 kudos

Has anyone had success with this? Trying to solve a resolve issue.

  • 7 kudos
2 More Replies
ianchenmu
by New Contributor II
  • 2950 Views
  • 5 replies
  • 7 kudos

Parallelization in training machine learning models using MLFlow

I'm training a ML model (e.g., XGboost) and I have a large combination of 5 hyperparameters, say each parameter has 5 candidates, it will be 5^5 = 3,125 combos.Now I want to do parallelization for the grid search on all the hyperparameter combos for ...

  • 2950 Views
  • 5 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hi @Chen Mu​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 7 kudos
4 More Replies
Mado
by Valued Contributor II
  • 9084 Views
  • 5 replies
  • 17 kudos

Resolved! Error when reading Excel file: "java.lang.NoClassDefFoundError: shadeio/poi/schemas/vmldrawing/XmlDocument"

Hi, I want to read an Excel file by:filepath_xlsx = "dbfs:/FileStore/data.xlsx"       sampleDF = (spark.read.format("com.crealytics.spark.excel")   .option("Header", "true")   .option("inferSchema", "false")   .option("treatEmptyValuesAsNulls", ...

  • 9084 Views
  • 5 replies
  • 17 kudos
Latest Reply
Mado
Valued Contributor II
  • 17 kudos

For this dataset, I also tried binary file reading as below: xldf_xlsx = ( spark.read.format("binaryFile") .option("pathGlobFilter", "*.xls*") .load(filepath_xlsx) )   excel_content = xldf_xlsx.head(1)[0].content file_like_obj = io.BytesIO(excel...

  • 17 kudos
4 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels