topic Re: DeltaTable.forPath(spark, path) doesnt recognize table in Data Governance

DeltaTable.forPath(spark, path) doesnt recognize table

amitca71 — Fri, 23 Sep 2022 21:16:15 GMT

Hi,

I'm working with unity catalog for the last week.

I'm refering to delta table by path, as follwing:

path='s3://<my_bucket_name>/silver/data/<table_name>

DeltaTable.forPath(spark, path)

I get an exception that "is not a Delta table"

using the table name using: DeltaTable.forName(spark, <table_name>)

everything works fine. both the attributes are exactly as apear on uc catalog (and data apears in S3.

dbutils.fs.ls(path) - also recognizes the content

till yesterday it was working fine, and the problem started 23/9/22 on the morning.

Anybody face the same issue?

Thanks,

Amit

Re: DeltaTable.forPath(spark, path) doesnt recognize table

shan_chandra — Fri, 23 Sep 2022 21:35:58 GMT

@Amit Cahanovich - could you please share the full error stack trace?

Re: DeltaTable.forPath(spark, path) doesnt recognize table

amitca71 — Fri, 23 Sep 2022 22:23:56 GMT

AnalysisException: `s3://......` is not a Delta table.

---------------------------------------------------------------------------

AnalysisException Traceback (most recent call last)

<command-1069785358704323> in <cell line: 2>()

1 silver_table_uri

----> 2 DeltaTable.forPath(spark,silver_table_uri)

/databricks/spark/python/delta/tables.py in forPath(cls, sparkSession, path, hadoopConf)

385 jsparkSession: "JavaObject" = sparkSession._jsparkSession # type: ignore[attr-defined]

386

--> 387 jdt = jvm.io.delta.tables.DeltaTable.forPath(jsparkSession, path, hadoopConf)

388 return DeltaTable(sparkSession, jdt)

389

/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py in __call__(self, *args)

1319

1320 answer = self.gateway_client.send_command(command)

-> 1321 return_value = get_return_value(

1322 answer, self.gateway_client, self.target_id, self.name)

1323

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)

200 # Hide where the exception came from that shows a non-Pythonic

201 # JVM exception message.

--> 202 raise converted from None

203 else:

204 raise

AnalysisException: `s3://.....` is not a Delta table.

Re: DeltaTable.forPath(spark, path) doesnt recognize table

amitca71 — Sat, 24 Sep 2022 06:04:00 GMT

its even more weired. on one next cells it doesnt... see below older version even by name doesnt work

Re: DeltaTable.forPath(spark, path) doesnt recognize table

amitca71 — Sat, 24 Sep 2022 07:27:38 GMT

on version 10.4