- 5299 Views
- 3 replies
- 5 kudos
Resolved! Access denied error to S3 bucket while running Kinesis spark streaming.
I get this below error while trying to simulate kinesis streams as mentioned in Databricks documentation at https://docs.databricks.com/getting-started/streaming.htmlError:java.nio.file.AccessDeniedException:Amazon S3; Status Code: 403; Error Code: A...
- 5299 Views
- 3 replies
- 5 kudos
- 5 kudos
If you do spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3a.access.key", AWS_ACCESS_KEY_ID) + secret with any other secret that has less access than your default one this sometimes happens, so running those commands but with your normal secre...
- 5 kudos
- 3234 Views
- 2 replies
- 4 kudos
Runtime error using MLFlow and Spark on databricks
Here is some model I created:class SomeModel(mlflow.pyfunc.PythonModel): def predict(self, context, input): # do fancy ML stuff # log results pandas_df = pd.DataFrame(...insert predictions here...) spark_df = spark...
- 3234 Views
- 2 replies
- 4 kudos
- 4 kudos
Any updates on this? I am running into the same issue@Patrick Tawil​ were you able to solve this problem? If so, do you mind sharing?
- 4 kudos
- 1349 Views
- 1 replies
- 0 kudos
Getting errors in DLT Pipeline while using ML Model
I am getting the following error when I try to run ML Models in Delta live Table Pipeline File "/local_disk0/.ephemeral_nfs/repl_tmp_data/ReplId-55c61-9b898-2c4b6-d/mlflow/envs/virtualenv_envs/mlflow-888f8c9b966409e6bddca3894244b4df9d1f94c1/lib/pyth...
- 1349 Views
- 1 replies
- 0 kudos
- 0 kudos
@Vittal Pai​ - In general, please follow the below steps for the mlflow CLI error,Step 1: set up API token and create secrets as mentioned in the below documenthttps://docs.databricks.com/machine-learning/manage-model-lifecycle/multiple-workspaces.h...
- 0 kudos
- 2967 Views
- 2 replies
- 1 kudos
Error with calling a machine learning serving endpoint
Hi!I have registered a spark model and generated a serving endpoint based on that.I am calling the endpoint with the relevant dataframe, somehow I got below errors. Could anyone show me how to tackle it, please? "Exception: Request failed with status...
- 2967 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @mavis chen​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 1 kudos
- 6264 Views
- 6 replies
- 6 kudos
Resolved! MLFlow is throwing error for the shape of input
I am running the code for prediction which will take the model from mlflow deployment. Code I have copied from the example given by mlflow experiment tab.import mlflow logged_model = 'runs:/id/model' # Load model as a PyFuncModel. loaded_model = ml...
- 6264 Views
- 6 replies
- 6 kudos
- 6 kudos
Hi @Koushik Deb​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 6 kudos
- 4635 Views
- 4 replies
- 5 kudos
Resolved! Error loading model from mlflow: java.io.StreamCorruptedException: invalid type code: 00
Hello,I'm using, in my IDE, Databricks Connect version 9.1LTS ML to connect to a databricks cluster with spark version 3.1 and download a spark model that's been trained and saved using mlflow.So it seems like it's able to find a copy the model, but ...
- 4635 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi @Kaniz Fatma​ and @Shanmugavel Chandrakasu​,It works after putting hadoop.dll into C:\Windows\System32 folder.I have hadoop version 3.3.1.I already had winutils.exe in the Hadoop bin folder.RegardsNath
- 5 kudos
- 1927 Views
- 1 replies
- 1 kudos
Error on pandas udf usage in databricks, sc.broadcasting random forest loaded from Kedro MLFlow Logger DataSet, cannot pickle '_thread.RLock' object
I'm trying to broadcast a Random forest (sklearn 1.2.0) recently loaded from mlflow, and using Pandas UDF to predict a model.​However, the same code works perfectly on Spark 2.4 + our OnPrem cluster.​I thought it was due to Spark 2.4 to 3 changes, an...
- 1927 Views
- 1 replies
- 1 kudos
- 1 kudos
Anyone?
- 1 kudos
- 3913 Views
- 4 replies
- 4 kudos
conda-env: error: unrecognized arguments: 'virtualenv': 'python_env.yaml'
I have registered an experiment as model in the model registry and when I start serving the model I get the following error:usage: conda-env [-h] {create,export,list,remove,update,config} ...conda-env: error: unrecognized arguments: 'virtualenv': 'py...
- 3913 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi Follks, Is there any new on this?.What should I do?ThanksBestPablo
- 4 kudos
- 2357 Views
- 3 replies
- 1 kudos
How does mlflow determine if a pyfunc model uses SparkContext?
I've been getting this error pretty regularly while working with mlflow:"It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that ...
- 2357 Views
- 3 replies
- 1 kudos
- 1 kudos
I checked the page and it looks like there is no integration with Datarobot and Datarobot doesn't contribute to mlflow. https://mlflow.org/ has all the integrations listed
- 1 kudos
- 4819 Views
- 0 replies
- 4 kudos
Error when reading Excel file: "org.apache.poi.ooxml.POIXMLException: Strict OOXML isn't currently supported, please see bug #57699"
Hi,I want to read an Excel "xlsx" file. The excel file has several sheets and multi-row header. The original file format was "xlsm" and I changed the extension to "xlsx". I try the following code:filepath_xlsx = "dbfs:/FileStore/Sample_Excel/data.xl...
- 4819 Views
- 0 replies
- 4 kudos
- 12084 Views
- 4 replies
- 17 kudos
Error when reading Excel file: "java.lang.NoClassDefFoundError: shadeio/poi/schemas/vmldrawing/XmlDocument"
Hi, I want to read an Excel file by:filepath_xlsx = "dbfs:/FileStore/data.xlsx" sampleDF = (spark.read.format("com.crealytics.spark.excel") .option("Header", "true") .option("inferSchema", "false") .option("treatEmptyValuesAsNulls", ...
- 12084 Views
- 4 replies
- 17 kudos
- 17 kudos
For this dataset, I also tried binary file reading as below: xldf_xlsx = ( spark.read.format("binaryFile") .option("pathGlobFilter", "*.xls*") .load(filepath_xlsx) ) excel_content = xldf_xlsx.head(1)[0].content file_like_obj = io.BytesIO(excel...
- 17 kudos
- 4360 Views
- 2 replies
- 4 kudos
AttributeError: 'NoneType' object has no attribute 'enum_types_by_name'
I run into this error while using MLFlow: AttributeError: 'NoneType' object has no attribute 'enum_types_by_name'Here is the relevant stack trace:/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/mlflow/tracking/fluent....
- 4360 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi, Could you please refer to this to check if this is an issue: https://github.com/protocolbuffers/protobuf/issues/10151, Also, could you please let us know the DBR version you are using?
- 4 kudos
- 3437 Views
- 3 replies
- 4 kudos
Resolved! How to avoid an error when using the automl python api on a classification problem
I am working through a basic example to get familiar with databricks automl. When I run classify, I hit an mlflow error. How can I avoid this error? My code:summary = databricks.automl.classify(train_df, target_col='new_cases', data_dir='dbfs:/automl...
- 3437 Views
- 3 replies
- 4 kudos
- 4 kudos
I had (accidentally) manually installed a very early version of databricks automl. Once I upgraded the error resolved.
- 4 kudos
- 2165 Views
- 3 replies
- 0 kudos
Error in importing feature_store
from databricks import feature_storeI am trying to import feature_store but it is showing this error.ImportError: cannot import name 'feature_store' from 'databricks' (/databricks/python/lib/python3.8/site-packages/databricks/__init__.py)
- 2165 Views
- 3 replies
- 0 kudos
- 0 kudos
Is this issue resolved completely? We are facing the same problem. this might help.
- 0 kudos
- 5751 Views
- 5 replies
- 0 kudos
Error when running job in databricks
Hello, I am very new with databricks and MLflow. I faced with the problem about running job. When the job is run, it usually failed and retried itself, so it incasesed running time, i.e., from normally 6 hrs to 12-18 hrs. From the error log, it shows...
- 5751 Views
- 5 replies
- 0 kudos
- 0 kudos
Hey there @Tanawat Benchasirirot​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hea...
- 0 kudos