topic delta sharing presigned url was removed, what should I do? in Data Engineering

delta sharing presigned url was removed, what should I do?

echozhuoocl — Tue, 26 Aug 2025 05:05:41 GMT

Caused by: java.lang.IllegalStateException: table s3a://dmsa/tmp/the_credential_of_deltasharing/on_prem_deltasharing.share#on-prem-delta-sharing.dmsa_in_nrt.shp_rating_snapshot was removed at org.apache.spark.delta.sharing.CachedTableManager.getPreSignedUrl(PreSignedUrlCache.scala:136) at org.apache.spark.delta.sharing.PreSignedUrlCacheEndpoint$$anonfun$receiveAndReply$1.applyOrElse(PreSignedUrlCache.scala:276) at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:103) at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213) at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100) at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75) at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)

Re: delta sharing presigned url was removed, what should I do?

szymon_dybczak — Tue, 26 Aug 2025 05:22:09 GMT

Hi @echozhuoocl ,

Did you VACCUM your table? If you're not sure, run:

DESCRIBE HISTORY catalog.schema.table

Re: delta sharing presigned url was removed, what should I do?

echozhuoocl — Tue, 26 Aug 2025 07:51:46 GMT

hi, please be patient to see my issue, thank you very much:

25/08/26 13:46:21 WARN RemoteSnapshot: Getting table size from a full file scan for table: on-prem-delta-sharing.dmsa_in_nrt.shp_rating_snapshot

25/08/26 14:35:12 INFO CachedTableManager: Updating pre signed urls for s3a://dmsa/tmp/the_credential_of_deltasharing/on_prem_deltasharing.share#on-prem-delta-sharing.dmsa_in_nrt.shp_rating_snapshot (expiration time: Tue Aug 26 14:49:22 CST 2025)

25/08/26 14:49:25 INFO CachedTableManager: Removing table s3a://dmsa/tmp/the_credential_of_deltasharing/on_prem_deltasharing.share#on-prem-delta-sharing.dmsa_in_nrt.shp_rating_snapshot from the pre signed url cache as it was not accessed after 3600000 ms

25/08/26 15:04:19 ERROR Inbox: Ignoring error

java.lang.IllegalStateException: table s3a://dmsa/tmp/the_credential_of_deltasharing/on_prem_deltasharing.share#on-prem-delta-sharing.dmsa_in_nrt.shp_rating_snapshot was removed

at org.apache.spark.delta.sharing.CachedTableManager.getPreSignedUrl(PreSignedUrlCache.scala:136) ~[dmsa_shp-0.0.1.jar:?]