Creating or using a custom defined model with SpaCy

[kashy]

I want to train and use a custom model with spaCy.   I don't know how to manage and create folders that the model would be need to save and load custom models and associated files (e.g. from DBFS)

It should be something like this but it doesn't accept a path from dbfs

nlp = spacy.load("/path/to/pipeline") # string path 

Any help appreciated

[Kaniz]

Hi @kashy , To train and use a custom model with spaCy, you would need to save and load your model. However, you're correct that spaCy does not directly accept a path from DBFS.

To work around this, you can save your trained model to DBFS and then load it from there.

Here's a general way to do it:

1. Save your trained model to DBFS:

python
nlp.to_disk('/dbfs/path/to/model')

2. Load your model from DBFS:

python
nlp = spacy.load('/dbfs/path/to/model')

The provided document discusses how to use custom libraries and private packages with Model Serving to create model deployments with enterprise-grade security.

It also explains how to upload dependency files to DBFS and log the model with a custom library.

However, it does not provide information on how to save and load a spaCy model specifically.

Please note that the path you provide should be a valid path in your DBFS where you have the appropriate read and write permissions.

[kashy]

Thank you @Kaniz did you mean to link another document regarding  how to use custom libraries and private packages with Model Serving to create model deployments with enterprise-grade security?