topic Re: Why am I getting NameError name _all_timezones_unchecked' is not defined in Data Engineering

Why am I getting NameError name _all_timezones_unchecked' is not defined

AgusBudianto — Sun, 28 Sep 2025 08:35:47 GMT

I defined the following local time function get datetime:

def get_sysdate():

jkt_tz = pytz.timezone('Asia/Jakarta')

sysdate = datetime.now(jkt_tz).strftime('%Y-%m-%d %H:%M:%S')

return sysdate

spark.udf.register("get_sysdate", get_sysdate)

But sometimes I get the error "_all_timezones_unchecked' is not defined," and the next run works fine.

Re: Why am I getting NameError name _all_timezones_unchecked' is not defined

Khaja_Zaffer — Mon, 29 Sep 2025 00:03:30 GMT

Hello @AgusBudianto

can you this and check once? I got it from GPT

ensure the pytz library and its timezone data are fully initialized on every worker node before the UDF is called.

import pytz
from datetime import datetime

# Force pytz to load all timezones on the driver and workers
# This is a key step to prevent the race condition
pytz.timezone('UTC')

def get_sysdate():
jkt_tz = pytz.timezone('Asia/Jakarta')
sysdate = datetime.now(jkt_tz).strftime('%Y-%m-%d %H:%M:%S')
return sysdate

spark.udf.register("get_sysdate", get_sysdate)

I did run on my workspace and got these results.

Re: Why am I getting NameError name _all_timezones_unchecked' is not defined

AgusBudianto — Mon, 29 Sep 2025 01:00:59 GMT

@Khaja_Zaffer, thank you for being willing to reply to my post I have initiated the function on the 2nd cell, and this problem only appears occasionally, and I am using General Compute

error

in function I have using library:

from datetime import datetime, timedelta

import pytz

Thank's

Anto

Re: Why am I getting NameError name _all_timezones_unchecked' is not defined

Khaja_Zaffer — Mon, 29 Sep 2025 09:04:21 GMT

Hello @AgusBudianto

Thank you so much for your response.

Can you try explicitly import inside the UDF, lets see if the error still persisting.

Re: Why am I getting NameError name _all_timezones_unchecked' is not defined

AgusBudianto — Mon, 06 Oct 2025 08:35:19 GMT

Sorry I'm just replying, I think this might be due to the choice of compute specifications but I'm not sure about this

Re: Why am I getting NameError name _all_timezones_unchecked' is not defined

AgusBudianto — Fri, 24 Oct 2025 01:54:10 GMT

Hi @Khaja_Zaffer

I have connected with MS Support and explained: pytz is no thread safe package, I believe it will have some issue when executor init it parallelly. Second, this is a 3rd party lib, and suggest using the built-in library from ZoneInfo # Python 3.9+

def get_sysdate():
jkt_tz = ZoneInfo("Asia/Jakarta")
return datetime.now(jkt_tz).strftime('%Y-%m-%d %H:%M:%S')

spark.udf.register("get_sysdate", get_sysdate)

and this worked, the error no longer appeared