[CONFIG_NOT_AVAILABLE] Configuration spark.sql.legacy.timeParserPolicy is not available. SQLSTATE:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-04-2025 12:39 AM
Running a DBT into the development environment, QA and PROD. Same config is working in QA and PROD but in dev facing this issue [CONFIG_NOT_AVAILABLE] Configuration spark.sql.legacy.timeParserPolicy is not available. SQLSTATE: and the compute type is sql warehouse serverless for all environment. How come issue is happening is DEV and not in QA and PROD, please support.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-04-2025 01:01 AM
Hi, I think the most likely thing causing this is that the config of the SQL warehouse on dev is slightly different to the config on your QA and Prod workspaces, this can happen for a few reasons.
- Workspace or Compute Version Drift: It's possible that the DEV environment is running on a slightly different version of serverless SQL warehouse infrastructure, or has not received recent updates that made this configuration available. Databricks sometimes rolls out engine upgrades or configuration support gradually across workspaces/environments, which can cause temporary discrepancies.
- Warehouse-Level Settings: The configuration availability can also be impacted if the DEV serverless warehouse was created before a platform upgrade, and thus does not support certain config keys. QA and PROD may have been created or refreshed after the necessary support was added.
- Preview/Channel Differences: Databricks SQL Warehouses can be attached to different channels (Current or Preview), and there may be differences in config support between channels. If DEV is running on a Preview version, or vice versa, the config support matrix may differ .
- Underlying Workspace Policies or Permissions: Sometimes, workspace-level cluster policies or admin settings could restrict which Spark configs are exposed or modifiable, leading to a config being “not available” in one environment but functioning in another.
To try to troubleshoot try the following steps:
- Check Warehouse Version & Channel: Compare the "Channel" (Current or Preview) and version of the SQL warehouse in DEV versus QA/PROD.
- Recreate DEV Warehouse: If the warehouse is older or was created before certain features/configs were rolled out, try restarting or recreating the DEV serverless SQL warehouse. Newer warehouses generally get the latest config support.
- Review Workspace Settings: Ensure no workspace policies or admin controls are restricting configs in DEV.
- Compare Settings: Use
SHOW PARAMETERSor check the warehouse details in all environments to review which configs are available and supported.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-04-2025 01:43 AM
I tried to recreate warehouse and also tried both current and preview,still it is same issue.Can we please connect in a teams call and check if it is possible
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-04-2025 02:34 AM
SET legacy_time_parser_policy = legacy;
https://docs.databricks.com/aws/en/sql/language-manual/parameters/legacy_time_parser_policy
https://docs.getdbt.com/docs/core/connect-data-platform/databricks-setup
your_profile_name:
target: dev
outputs:
dev:
type: databricks
host: YOURORG.databrickshost.com
http_path: /sql/1.0/warehouses/XXXX
schema: YOUR_SCHEMA
token: "{{ env_var('DATABRICKS_TOKEN') }}"
session_properties:
legacy_time_parser_policy: legacy
After that remove any other spark.sql.legacy.timeParserPolicy from your code
My blog: https://databrickster.medium.com/