'Unity Catalog Volumes is not enabled on this instance' error

alexiswl — Mon, 04 Dec 2023 06:50:54 GMT

Hi all,

tl;dr

I ran the following on a docker-backed personal compute instance (running 13.3-LTS)

```

%sql

USE CATALOG hail;

USE SCHEMA volumes_testing;

CREATE VOLUME 1kg

COMMENT 'Testing 1000 Genomes volume';

```

But this gives

```
ParseException: [UC_VOLUMES_NOT_ENABLED] Support for Unity Catalog Volumes is not enabled on this instance
```

Full version

I am trying to use Hail | Hail 0.2 with DataBricks.

Although not technically supported, I have built a docker container that seems to support Spark 3.4.1 with the latest branch of the hail version (with a patch or too), code is here for those interested.

docker-who/repositories/hail/0.2.126--spark-3.4.1-patch at main · umccr/docker-who (github.com)

Container is open access here - https://github.com/umccr/docker-who/pkgs/container/hail

This docker image has been backed onto DataBricks Runtime 13.3 LTS.

I have a personal compute cluster with the following configuration, and use the docker image above.

```json

{
"num_workers": 0,
"cluster_name": "hail",
"spark_version": "13.3.x-scala2.12",
"spark_conf": {
"spark.databricks.cluster.profile": "singleNode",
"spark.master": "local[*, 4]"
},
"aws_attributes": {
"first_on_demand": 1,
"availability": "ON_DEMAND",
"zone_id": "auto",
"spot_bid_price_percent": 100,
"ebs_volume_count": 0
},
"node_type_id": "i3.xlarge",
"driver_node_type_id": "i3.xlarge",
"ssh_public_keys": [],
"custom_tags": {
"ResourceClass": "SingleNode"
},
"spark_env_vars": {},
"autotermination_minutes": 20,
"enable_elastic_disk": true,
"init_scripts": [],
"docker_image": {
"url": "ghcr.io/umccr/hail:latest"
},
"single_user_name": "alexis.lucattini@umccr.org",
"policy_id": "D063D7FAC000009E",
"enable_local_disk_encryption": false,
"data_security_mode": "SINGLE_USER",
"runtime_engine": "STANDARD",
"cluster_id": "1203-223920-6d7l06ne"
}
```

I then ran the following code (derived from Hail | MatrixTable Tutorial)

```

# Imports
import hail as hl

from tempfile import TemporaryDirectory

from pathlib import Path

# Initialisation

hl.init(

sc,

idempotent=True,

quiet=True,

skip_logging_configuration=True

)

# TMP Information

TMP_DATA_OBJ = TemporaryDirectory()

TMP_DATA_PATH = Path(TMP_DATA_OBJ.name)

# TMP Data Path

hl.utils.get_1kg(str(TMP_DATA_PATH) + "/")

# Import VCF

hl.import_vcf(

str(TMP_DATA_PATH / '1kg.vcf.bgz')

).write(

str(TMP_DATA_PATH / '1kg.mt'),

overwrite=True

)

# Read Matrix Table

mt = hl.read_matrix_table(

str(TMP_DATA_PATH / '1kg.mt')

)

%sql

USE CATALOG hail;

USE SCHEMA volumes_testing;

CREATE VOLUME 1kg

COMMENT 'Testing 1000 Genomes volume';

```

But gives

```
ParseException: [UC_VOLUMES_NOT_ENABLED] Support for Unity Catalog Volumes is not enabled on this instance
```

I thought Volumes were enabled in Private preview for 13.2 and above?
Announcing Public Preview of Volumes in Databricks Unity Catalog | Databricks Blog

Is it because my instance is backed by a docker image?

Can someone point me to the documentation on this if I have missed it?

Alexis

Re: 'Unity Catalog Volumes is not enabled on this instance' error

alexiswl — Tue, 05 Dec 2023 05:47:54 GMT

Resolved with the setting "spark.databricks.unityCatalog.volumes.enabled" = "true"

topic Re: 'Unity Catalog Volumes is not enabled on this instance' error in Get Started Discussions

'Unity Catalog Volumes is not enabled on this instance' error

Re: 'Unity Catalog Volumes is not enabled on this instance' error