- 3709 Views
- 1 replies
- 0 kudos
Issue with Disabled "Repair DAG", "Repair All DAGs" Buttons in Airflow UI, functionality is working.
We are encountering an issue in the Airflow UI where the 'Repair DAG' and 'Repair All DAGs' options are disabled when a specific task fails. While the repair functionality itself is working properly (i.e., the DAGs can still be repaired through execu...
- 3709 Views
- 1 replies
- 0 kudos
- 0 kudos
The issue with the 'Repair DAG' and 'Repair All DAGs' options being disabled in the Airflow UI when using the Databricks Workflow Operator is a known UI-specific problem that does not affect backend execution or the actual repair functionality. While...
- 0 kudos
- 3701 Views
- 1 replies
- 1 kudos
How to Fetch Azure OpenAI api_version and engine Dynamically After Resource Creation via Python?
Hello,I am using Python to automate the creation of Azure OpenAI resources via the Azure Management API. I am successfully able to create the resource, but I need to dynamically fetch the following details after the resource is created:API Version (a...
- 3701 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Sudheer, It's been a while since you posted, but are you still facing this issue? Here are a few things you could check if needed: API version In Azure OpenAI, api-version is a query parameter on the data-plane (inference) requests, not a proper...
- 1 kudos
- 1309 Views
- 5 replies
- 2 kudos
Resolved! #data bricks snowflake dialect
Hello,I’m encountering an issue while converting SQL code to the Lake bridge Snowflake dialect. It seems that DML and DDL statements may not be supported in the Snowflake dialect within Lake bridge.Could you please confirm whether DML and DDL stateme...
- 1309 Views
- 5 replies
- 2 kudos
- 2 kudos
@jadhav_vikas , I did some digging through internal docs and I have some hints/suggestions. Short answer Databricks Lakehouse Federation (often referred to as “Lakehouse Bridge”) provides read‑only access to Snowflake; DML and DDL are not supported ...
- 2 kudos
- 653 Views
- 3 replies
- 0 kudos
Resolved! API call fails to initiate create Service Principal secret
Hi,I've constructed an AWS lambda function which is used to auto rotate my Service Principal Secret in the Databricks account. Authentication is setup with OAuth2, the api call for the token generation is successful but when executing the api call to...
- 653 Views
- 3 replies
- 0 kudos
- 0 kudos
Your error message, "Invalid service principal id," typically indicates a mismatch or formatting problem with the service principal's unique identifier in your API request. Although you checked the client_id, this value is not always the one needed f...
- 0 kudos
- 1071 Views
- 1 replies
- 2 kudos
Databricks Taiwan User Community
Would be great have a group for databricks Taiwan community.
- 1071 Views
- 1 replies
- 2 kudos
- 2 kudos
Hey @kennyhsieh , hope it's not too late to reply to the post! A "Databricks User Group Taiwan" has recently been formed on LinkedIn.
- 2 kudos
- 4025 Views
- 2 replies
- 0 kudos
Can AWS workspaces share subnets?
The docs state:"You can choose to share one subnet across multiple workspaces or both subnets across workspaces."as well as:"You can reuse existing security groups rather than create new ones."and on this page:"If you plan to share a VPC and subnets ...
- 4025 Views
- 2 replies
- 0 kudos
- 0 kudos
AWS WorkSpaces can be configured with subnets that can be shared within an AWS account or across AWS accounts using resource sharing mechanisms, but this depends on the specific AWS service and context. For Databricks workspaces on AWS, documentation...
- 0 kudos
- 3567 Views
- 1 replies
- 0 kudos
Joblib with optuna and SB3
Hi everyone,I am training some reinforcement learning models and I am trying to automate the hyperparameter search using optuna. I saw in the documentation that you can use joblib with spark as a backend to train in paralel. I got that working with t...
- 3567 Views
- 1 replies
- 0 kudos
- 0 kudos
Stable Baselines 3 (SB3) models can be optimized with Optuna for hyperparameter search, but parallelizing these searches using Joblib with Spark as the backend—like the classic scikit-learn example—commonly encounters issues. The root problem is that...
- 0 kudos
- 3512 Views
- 1 replies
- 0 kudos
NiFi on EKS Fails to Connect to Databricks via JDBC – "Connection reset" Error
I'm using Apache Nifi (running on AWS EKS) to connect to Databricks (with compute on EC2) via JDBC. My JDBC URL is as follows: jdbc:databricks://server_hostname:443/default;transportMode=http;ssl=1;httpPath=my_httppath;AuthMech=3;UID=token;PWD=my_tok...
- 3512 Views
- 1 replies
- 0 kudos
- 0 kudos
A "Connection reset" error in NiFi when connecting to Databricks via JDBC, despite successful telnet and working connectivity from DBeaver, usually points to subtle protocol or compatibility issues rather than network-level blocks. Common Causes JD...
- 0 kudos
- 4170 Views
- 1 replies
- 0 kudos
.py script execution failed but succeeded when run in Python notebook
Background:My code executing without problem if run in a python notebook. However, the same code fails when execute from a .py script in the workspace. Seems like the 2 execution methods don't have identical version of the packagesError message: Attr...
- 4170 Views
- 1 replies
- 0 kudos
- 0 kudos
The error "AttributeError: 'DeltaMergeBuilder' object has no attribute 'withSchemaEvolution'" when running the same code from a .py script but not in a Python notebook is likely caused by a mismatch in the Delta Lake or Databricks Runtime versions or...
- 0 kudos
- 4770 Views
- 2 replies
- 0 kudos
Getting error while installing applicationinsights
Library installation attempted on the driver node of cluster 0210-115502-3lo6gkwd and failed. Pip could not find a version that satisfies the requirement for the library. Please check your library version and dependencies. Error code: ERROR_NO_MATCHI...
- 4770 Views
- 2 replies
- 0 kudos
- 0 kudos
The error indicates that Databricks could not install the applicationinsights or azure-identity libraries because pip could not find a matching distribution, and network connection attempts to the PyPI repository were repeatedly reset. This is common...
- 0 kudos
- 1371 Views
- 1 replies
- 0 kudos
Databricks Roadmap 13/11/2025
Can you provide me with a link to the recording that happened today:Databricks Product Roadmap Webinar, Thursday, November 13, 2025, 9:00 AM–10:00 AM GMT
- 1371 Views
- 1 replies
- 0 kudos
- 0 kudos
@Superstar_Singh , Is this the one you are asking about? https://www.databricks.com/resources/webinar/productroadmapwebinar
- 0 kudos
- 3912 Views
- 1 replies
- 0 kudos
Is there a way to prevent databricks-connect from installing a global IPython Spark startup script?
I'm currently using databricks-connect through VS Code on MacOS. However, this seems to install (and re-install upon deletion) an IPython startup script which initializes a SparkSession. This is fine as far as it goes, except that this script is *glo...
- 3912 Views
- 1 replies
- 0 kudos
- 0 kudos
Databricks Connect on MacOS (and some other platforms) adds a file to the global IPython startup folder, which causes every new IPython session—including those outside the Databricks environment—to attempt loading this SparkSession initialization. Th...
- 0 kudos
- 6905 Views
- 1 replies
- 0 kudos
PYTEST: Module not found error
Hi,Apologies, as I am trying to use Pytest first time. I know this question has been raised but I went through previous answers but the issue still exists.I am following DAtabricks and other articles using pytest. My structure is simple as -tests--co...
- 6905 Views
- 1 replies
- 0 kudos
- 0 kudos
Your issue with ModuleNotFoundError: No module named 'test_tran' when running pytest from a notebook is likely caused by how Python sets the module import paths and the current working directory inside Databricks notebooks (or similar environments). ...
- 0 kudos
- 4231 Views
- 1 replies
- 0 kudos
CloudFormation Stack Failure: Custom::CreateWorkspace in CREATE_FAILED State
I am trying to create a workspace using AWS CloudFormation, but the stack fails with the following error:"The resource CreateWorkspace is in a CREATE_FAILED state. This Custom::CreateWorkspace resource is in a CREATE_FAILED state. Received response s...
- 4231 Views
- 1 replies
- 0 kudos
- 0 kudos
When a CloudFormation stack fails with “The resource CreateWorkspace is in a CREATE_FAILED state” for a Custom::CreateWorkspace resource, it typically means the Lambda or service backing the custom resource returned a FAILED signal to CloudFormation ...
- 0 kudos
- 4004 Views
- 1 replies
- 0 kudos
How to Define Constants at Bundle Level in Databricks Asset Bundles for Use in Notebooks?
I'm working with Databricks Asset Bundles and need to define constants at the bundle level based on the target environment. These constants will be used inside Databricks notebooks.For example, I want a constant gold_catalog to take different values ...
- 4004 Views
- 1 replies
- 0 kudos
- 0 kudos
There is currently no explicit, built-in mechanism in Databricks Asset Bundles (as of 2024) for directly defining global, environment-targeted constants at the bundle level that can be seamlessly accessed inside notebooks without using job or task pa...
- 0 kudos
