Why am I getting NameError name _all_timezones_unchecked' is not defined

[AgusBudianto]

I defined the following local time function get datetime:

def get_sysdate():

    jkt_tz = pytz.timezone('Asia/Jakarta')

    sysdate = datetime.now(jkt_tz).strftime('%Y-%m-%d %H:%M:%S')

    return sysdate

spark.udf.register("get_sysdate", get_sysdate)

But sometimes I get the error "_all_timezones_unchecked' is not defined," and the next run works fine.

[Khaja_Zaffer]

Hello @AgusBudianto 

can you this and check once? I got it from GPT

ensure the pytz library and its timezone data are fully initialized on every worker node before the UDF is called.

import pytz
from datetime import datetime

# Force pytz to load all timezones on the driver and workers
# This is a key step to prevent the race condition
pytz.timezone('UTC')

def get_sysdate():
jkt_tz = pytz.timezone('Asia/Jakarta')
sysdate = datetime.now(jkt_tz).strftime('%Y-%m-%d %H:%M:%S')
return sysdate

spark.udf.register("get_sysdate", get_sysdate)

I did run on my workspace and got these results. 

[AgusBudianto]

@Khaja_Zaffer, thank you for being willing to reply to my post I have initiated the function on the 2nd cell, and this problem only appears occasionally, and I am using General Compute

error

in function I have using library:

from datetime import datetime, timedelta

import pytz

Thank's

Anto

[Khaja_Zaffer]

Hello @AgusBudianto 

Thank you so much for your response. 

Can you try explicitly import inside the UDF, lets see if the error still persisting. 

[AgusBudianto]

Sorry I'm just replying, I think this might be due to the choice of compute specifications but I'm not sure about this

[AgusBudianto · Accepted solution]

Hi @Khaja_Zaffer 

I have connected with MS Support and explained: pytz is no thread safe package, I believe it will have some issue when executor init it parallelly. Second, this is a 3rd party lib, and suggest using the built-in library from ZoneInfo  # Python 3.9+

def get_sysdate():
jkt_tz = ZoneInfo("Asia/Jakarta")
return datetime.now(jkt_tz).strftime('%Y-%m-%d %H:%M:%S')

spark.udf.register("get_sysdate", get_sysdate)

and this worked, the error no longer appeared