topic Re: How to register datasets for Detectron2 in Data Engineering

How to register datasets for Detectron2

SarahDorich — Mon, 16 Aug 2021 19:23:31 GMT

I'm trying to run a Detectron2 model in Databricks and cannot figure out how to register my train, val and test datasets. My datasets live in an Azure data lake. I have tried the following with no luck. Any help is appreciated.

1) Specifying full path to Azure:

path_to_data = "abfss://<>@<>.dfs.core.windows.net/recommender/house-detector-datasets"

from detectron2.data.datasets import register_coco_instances

register_coco_instances("house_train3", {}, f"{path_to_data}/train/instances_default.json", f"{path_to_data}/train")

2) Moving to temporary local storage first:

import os

os.mkdir("house-detector-datasets")

my_blob_folder = "abfss://<>@<>.dfs.core.windows.net/recommender/house-detector-datasets"

dbutils.fs.cp(my_blob_folder, "house-detector-datasets", recurse=True)

path_to_data = "house-detector-datasets"

register_coco_instances("house_train4", {}, f"{path_to_data}/train/instances_default.json", f"{path_to_data}/train")

3) Moving to dbfs first:

Same code as 2) except moving to dbfs:/tmp/.

In all of these cases, I get the error when I try and access my registered datasets (for example, the code below fails with the error "No such file or directory")...

my_dataset_train_metadata = MetadataCatalog.get("house_train3") dataset_dicts = DatasetCatalog.get("house_train3")

Re: How to register datasets for Detectron2

matthews163 — Tue, 17 Aug 2021 07:38:20 GMT

mygiftcardsite Wrote:

I think this might help you

<code>from detectron2.data.datasets import register_coco_instances
register_coco_instances("YourTrainDatasetName", {},"path to train.json", "path to train image folder")
register_coco_instances("YourTestDatasetName", {}, "path to test.json", "path to test image folder")

Let me know if it works for you.I have trained detectron2 using this.

Re: How to register datasets for Detectron2

SarahDorich — Tue, 17 Aug 2021 12:51:38 GMT

I have trained using this as well but not in Databricks (it's what I'm trying to get working). For some reason, the paths that I'm specifying the model cannot find. What did your paths look like to your datasets?

Re: How to register datasets for Detectron2

Thurman — Wed, 18 Aug 2021 04:58:35 GMT