cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Avi711
by New Contributor III
  • 3413 Views
  • 3 replies
  • 5 kudos

Resolved! Access denied error to S3 bucket while running Kinesis spark streaming.

I get this below error while trying to simulate kinesis streams as mentioned in Databricks documentation at https://docs.databricks.com/getting-started/streaming.htmlError:java.nio.file.AccessDeniedException:Amazon S3; Status Code: 403; Error Code: A...

  • 3413 Views
  • 3 replies
  • 5 kudos
Latest Reply
jsuarezg
New Contributor II
  • 5 kudos

If you do spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3a.access.key", AWS_ACCESS_KEY_ID) + secret with any other secret that has less access than your default one this sometimes happens, so running those commands but with your normal secre...

  • 5 kudos
2 More Replies
ptawil
by New Contributor III
  • 1973 Views
  • 2 replies
  • 4 kudos

Runtime error using MLFlow and Spark on databricks

Here is some model I created:class SomeModel(mlflow.pyfunc.PythonModel): def predict(self, context, input): # do fancy ML stuff # log results pandas_df = pd.DataFrame(...insert predictions here...) spark_df = spark...

  • 1973 Views
  • 2 replies
  • 4 kudos
Latest Reply
Nikhil3107
New Contributor III
  • 4 kudos

Any updates on this? I am running into the same issue@Patrick Tawil​ were you able to solve this problem? If so, do you mind sharing?

  • 4 kudos
1 More Replies
vittal
by New Contributor
  • 736 Views
  • 1 replies
  • 0 kudos

Getting errors in DLT Pipeline while using ML Model

I am getting the following error when I try to run ML Models in Delta live Table Pipeline File "/local_disk0/.ephemeral_nfs/repl_tmp_data/ReplId-55c61-9b898-2c4b6-d/mlflow/envs/virtualenv_envs/mlflow-888f8c9b966409e6bddca3894244b4df9d1f94c1/lib/pyth...

  • 736 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@Vittal Pai​  - In general, please follow the below steps for the mlflow CLI error,Step 1: set up API token and create secrets as mentioned in the below documenthttps://docs.databricks.com/machine-learning/manage-model-lifecycle/multiple-workspaces.h...

  • 0 kudos
notsure
by New Contributor
  • 1731 Views
  • 3 replies
  • 2 kudos

Error with calling a machine learning serving endpoint

Hi!I have registered a spark model and generated a serving endpoint based on that.I am calling the endpoint with the relevant dataframe, somehow I got below errors. Could anyone show me how to tackle it, please? "Exception: Request failed with status...

  • 1731 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @mavis chen​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...

  • 2 kudos
2 More Replies
DebK
by New Contributor III
  • 2787 Views
  • 6 replies
  • 6 kudos

Resolved! MLFlow is throwing error for the shape of input

I am running the code for prediction which will take the model from mlflow deployment. Code I have copied from the example given by mlflow experiment tab.import mlflow logged_model = 'runs:/id/model'   # Load model as a PyFuncModel. loaded_model = ml...

  • 2787 Views
  • 6 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Koushik Deb​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...

  • 6 kudos
5 More Replies
NSRBX
by Contributor
  • 2780 Views
  • 6 replies
  • 6 kudos

Resolved! Error loading model from mlflow: java.io.StreamCorruptedException: invalid type code: 00

Hello,I'm using, in my IDE, Databricks Connect version 9.1LTS ML to connect to a databricks cluster with spark version 3.1 and download a spark model that's been trained and saved using mlflow.So it seems like it's able to find a copy the model, but ...

  • 2780 Views
  • 6 replies
  • 6 kudos
Latest Reply
NSRBX
Contributor
  • 6 kudos

Hi @Kaniz Fatma​ and @Shanmugavel Chandrakasu​,It works after putting hadoop.dll into C:\Windows\System32 folder.I have hadoop version 3.3.1.I already had winutils.exe in the Hadoop bin folder.RegardsNath

  • 6 kudos
5 More Replies
ryojikn
by New Contributor III
  • 1013 Views
  • 1 replies
  • 1 kudos

Error on pandas udf usage in databricks, sc.broadcasting random forest loaded from Kedro MLFlow Logger DataSet, cannot pickle '_thread.RLock' object

I'm trying to broadcast a Random forest (sklearn 1.2.0) recently loaded from mlflow, and using Pandas UDF to predict a model.​However, the same code works perfectly on Spark 2.4 + our OnPrem cluster.​I thought it was due to Spark 2.4 to 3 changes, an...

  • 1013 Views
  • 1 replies
  • 1 kudos
Latest Reply
ryojikn
New Contributor III
  • 1 kudos

Anyone?

  • 1 kudos
PNegro
by New Contributor III
  • 2133 Views
  • 4 replies
  • 4 kudos

conda-env: error: unrecognized arguments: 'virtualenv': 'python_env.yaml'

I have registered an experiment as model in the model registry and when I start serving the model I get the following error:usage: conda-env [-h] {create,export,list,remove,update,config} ...conda-env: error: unrecognized arguments: 'virtualenv': 'py...

  • 2133 Views
  • 4 replies
  • 4 kudos
Latest Reply
PNegro
New Contributor III
  • 4 kudos

Hi Follks, Is there any new on this?.What should I do?ThanksBestPablo

  • 4 kudos
3 More Replies
jonathan-dufaul
by Valued Contributor
  • 1238 Views
  • 3 replies
  • 1 kudos

How does mlflow determine if a pyfunc model uses SparkContext?

I've been getting this error pretty regularly while working with mlflow:"It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that ...

  • 1238 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

I checked the page and it looks like there is no integration with Datarobot and Datarobot doesn't contribute to mlflow. https://mlflow.org/ has all the integrations listed

  • 1 kudos
2 More Replies
Mado
by Valued Contributor II
  • 8169 Views
  • 5 replies
  • 17 kudos

Resolved! Error when reading Excel file: "java.lang.NoClassDefFoundError: shadeio/poi/schemas/vmldrawing/XmlDocument"

Hi, I want to read an Excel file by:filepath_xlsx = "dbfs:/FileStore/data.xlsx"       sampleDF = (spark.read.format("com.crealytics.spark.excel")   .option("Header", "true")   .option("inferSchema", "false")   .option("treatEmptyValuesAsNulls", ...

  • 8169 Views
  • 5 replies
  • 17 kudos
Latest Reply
Mado
Valued Contributor II
  • 17 kudos

For this dataset, I also tried binary file reading as below: xldf_xlsx = ( spark.read.format("binaryFile") .option("pathGlobFilter", "*.xls*") .load(filepath_xlsx) )   excel_content = xldf_xlsx.head(1)[0].content file_like_obj = io.BytesIO(excel...

  • 17 kudos
4 More Replies
Mado
by Valued Contributor II
  • 2536 Views
  • 1 replies
  • 4 kudos

Resolved! Error when reading Excel file: "org.apache.poi.ooxml.POIXMLException: Strict OOXML isn't currently supported, please see bug #57699"

Hi,I want to read an Excel "xlsx" file. The excel file has several sheets and multi-row header. The original file format was "xlsm" and I changed the extension to "xlsx". I try the following code:filepath_xlsx = "dbfs:/FileStore/Sample_Excel/data.xl...

  • 2536 Views
  • 1 replies
  • 4 kudos
Latest Reply
Kaniz
Community Manager
  • 4 kudos

Hi @Mohammad Saber​, The error says, Don't save your spreadsheet in "strict OOXML" format.For example, in Excel use.Save As --> "Excel Workbook (.xlsx)" instead ofSave As --> "Strict Open XML Spreadsheet (.xlsx)"

  • 4 kudos
eshaanpathak
by New Contributor III
  • 2133 Views
  • 3 replies
  • 4 kudos

AttributeError: 'NoneType' object has no attribute 'enum_types_by_name'

I run into this error while using MLFlow: AttributeError: 'NoneType' object has no attribute 'enum_types_by_name'Here is the relevant stack trace:/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/mlflow/tracking/fluent....

  • 2133 Views
  • 3 replies
  • 4 kudos
Latest Reply
Kaniz
Community Manager
  • 4 kudos

Hi @Eshaan Pathak​ , We haven’t heard from you on the last response from @Debayan Mukherjee​​, and I was checking back to see if their suggestions helped you. Or else, If you have any solution, please do share that with the community as it can be hel...

  • 4 kudos
2 More Replies
brendanmckenna
by New Contributor III
  • 1716 Views
  • 4 replies
  • 4 kudos

Resolved! How to avoid an error when using the automl python api on a classification problem

I am working through a basic example to get familiar with databricks automl. When I run classify, I hit an mlflow error. How can I avoid this error? My code:summary = databricks.automl.classify(train_df, target_col='new_cases', data_dir='dbfs:/automl...

  • 1716 Views
  • 4 replies
  • 4 kudos
Latest Reply
Kaniz
Community Manager
  • 4 kudos

Hi @Brendan McKenna​ , We haven’t heard from you since the last response from @Debayan Mukherjee​. Or else, If you have any solution, please share it with the community, as it can be helpful to others. Also, Please don't forget to click on the "Selec...

  • 4 kudos
3 More Replies
sameer_gupta
by New Contributor
  • 1100 Views
  • 3 replies
  • 0 kudos

Error in importing feature_store

from databricks import feature_storeI am trying to import feature_store but it is showing this error.ImportError: cannot import name 'feature_store' from 'databricks' (/databricks/python/lib/python3.8/site-packages/databricks/__init__.py)

  • 1100 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Is this issue resolved completely? We are facing the same problem. this might help.

  • 0 kudos
2 More Replies
Benji
by New Contributor II
  • 2672 Views
  • 5 replies
  • 0 kudos

Error when running job in databricks

Hello, I am very new with databricks and MLflow. I faced with the problem about running job. When the job is run, it usually failed and retried itself, so it incasesed running time, i.e., from normally 6 hrs to 12-18 hrs. From the error log, it shows...

Fail image
  • 2672 Views
  • 5 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hey there @Tanawat Benchasirirot​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hea...

  • 0 kudos
4 More Replies
Labels