Found an answer!

Basically pip was somehow installed the dependencies from the git repo first and was not following the given order so in order to solve this, I added the libraries for conda to install.

```
conda_env = {
    "channels": [
        "defaults",
        "pytorch"
    ],
    "dependencies": [
        "python=3.8",
        "numpy==1.24.3",
        "pytorch==2.2.2",
        "pip",
        {
            "pip": [
                "fvcore==0.1.5.post20221221",
                "git+https://github.com/wookayin/gpustat",
                "pycocotools==2.0.6",
                "torchvision==0.15.2",
                "git+https://github.com/facebookresearch/detectron2.git"
            ]
        }
    ]
}
```