- 4498 Views
- 4 replies
- 6 kudos
Resolved! How to use variable-overrides.json for environment-specific configuration in Asset Bundles?
Hi all,Could someone clarify the intended usage of the variable-overrides.json file in Databricks Asset Bundles?Let me give some context. Let's say my repository layout looks like this:databricks/ ├── notebooks/ │ └── notebook.ipynb ├── resources/ ...
- 4498 Views
- 4 replies
- 6 kudos
- 6 kudos
It does. Thanks for the reponse. I also continued playing around with it and found a way using the variable-overrides.json file. I'll leave it here just in case anyone is interested:Repository layout:databricks/ ├── notebooks/ │ └── notebook.ipynb ...
- 6 kudos
- 2980 Views
- 4 replies
- 1 kudos
Best practices for optimizing Spark jobs
What are some best practices for optimizing Spark jobs in Databricks, especially when dealing large datasets? Any tips or resources would be greatly appreciated! I’m trying to analyze data on restaurant menu prices so that insights would be especiall...
- 2980 Views
- 4 replies
- 1 kudos
- 1 kudos
In addition to above cool comments, try to use clusters with VMs enabled for disk caching as well. This caches data at parquet files level in VM local storage, acting as a great complement to spark caching.
- 1 kudos
- 370 Views
- 0 replies
- 1 kudos
Agent Bricks Webinar
Our Databricks x Unifeye Meetup community just hit 150 members! A huge milestone, especially considering we’ve consistently had 50+ people joining every webinar. The momentum is real, and the audience keeps growing! This week, we’re taking it one s...
- 370 Views
- 0 replies
- 1 kudos
- 677 Views
- 2 replies
- 1 kudos
Resolved! SQL cell v spark.sql in notebooks
I am fairly new to Databricks, and indeed Python, so apologies if this has been answered elsewhere but I've been unable to find it.I have been mainly working in notebooks as opposed to the SQL editor, but coding in SQL where possible using SQL cells ...
- 677 Views
- 2 replies
- 1 kudos
- 7446 Views
- 5 replies
- 3 kudos
Connect to Onelake using Service Principal, Unity Catalog and Databricks Access Connector
We are trying to connect Databricks to OneLake, to read data from a Fabric workspace into Databricks, using a notebook. We also use Unity Catalog. We are able to read data from the workspace with a Service Principal like this:from pyspark.sql.types i...
- 7446 Views
- 5 replies
- 3 kudos
- 3 kudos
As commented you need to assign "Storage Blob Data Contributor or Storage Account Contributor to the service principal you're using in the "connection" provided to the "external location". Another more advanced and even better option would be to use ...
- 3 kudos
- 4473 Views
- 1 replies
- 0 kudos
Updating a Delta Table in Delta Live Tables (DLT) from Two Event Hubs
I am working with Databricks Delta Live Tables (DLT) and need to ingest data from two different Event Hubs. My goal is to:Ingest initial data from the first Event Hub (Predictor) and store it in a Delta Table (data_predictions).Later, update this tab...
- 4473 Views
- 1 replies
- 0 kudos
- 0 kudos
To achieve robust, persistent CDC (Change Data Capture)–style updates in Databricks DLT with your scenario—while keeping data_predictions as a Delta Table (not a Materialized View)—you need to carefully avoid streaming joins and side effects across s...
- 0 kudos
- 3835 Views
- 1 replies
- 1 kudos
Resolved! Databricks UMF Best Practice
Hi there, I would like to get some feedback on what are the ideal/suggested ways to get UMF data from our Azure cloud into Databricks. For context, UMF can mean either:User Managed FileUser Maintained FileBasically, a UMF could be something like a si...
- 3835 Views
- 1 replies
- 1 kudos
- 1 kudos
Several effective patterns exist for ingesting User Managed Files (UMF) such as CSVs from Azure into Databricks, each with different trade-offs depending on governance, user interface preferences, and integration with Microsoft 365 services. Common A...
- 1 kudos
- 4273 Views
- 1 replies
- 0 kudos
DLT detecting changes but not applying them
We have three source tables used for a streaming dimension table in silver. Around 50K records are changed in one of the source tables, and the DLT pipeline shows that it has updated those 50K records, but they remain unchanged. The only way to pick ...
- 4273 Views
- 1 replies
- 0 kudos
- 0 kudos
The most likely reason your DLT pipeline shows 50K updates but the records remain unchanged is related to how Delta Live Tables (DLT) handle streaming tables, update logic, and schema constraints. When the target table uses an auto-increment ID (espe...
- 0 kudos
- 3642 Views
- 1 replies
- 0 kudos
Issue with Disabled "Repair DAG", "Repair All DAGs" Buttons in Airflow UI, functionality is working.
We are encountering an issue in the Airflow UI where the 'Repair DAG' and 'Repair All DAGs' options are disabled when a specific task fails. While the repair functionality itself is working properly (i.e., the DAGs can still be repaired through execu...
- 3642 Views
- 1 replies
- 0 kudos
- 0 kudos
The issue with the 'Repair DAG' and 'Repair All DAGs' options being disabled in the Airflow UI when using the Databricks Workflow Operator is a known UI-specific problem that does not affect backend execution or the actual repair functionality. While...
- 0 kudos
- 2957 Views
- 1 replies
- 1 kudos
How to Fetch Azure OpenAI api_version and engine Dynamically After Resource Creation via Python?
Hello,I am using Python to automate the creation of Azure OpenAI resources via the Azure Management API. I am successfully able to create the resource, but I need to dynamically fetch the following details after the resource is created:API Version (a...
- 2957 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Sudheer, It's been a while since you posted, but are you still facing this issue? Here are a few things you could check if needed: API version In Azure OpenAI, api-version is a query parameter on the data-plane (inference) requests, not a proper...
- 1 kudos
- 1120 Views
- 5 replies
- 2 kudos
Resolved! #data bricks snowflake dialect
Hello,I’m encountering an issue while converting SQL code to the Lake bridge Snowflake dialect. It seems that DML and DDL statements may not be supported in the Snowflake dialect within Lake bridge.Could you please confirm whether DML and DDL stateme...
- 1120 Views
- 5 replies
- 2 kudos
- 2 kudos
@jadhav_vikas , I did some digging through internal docs and I have some hints/suggestions. Short answer Databricks Lakehouse Federation (often referred to as “Lakehouse Bridge”) provides read‑only access to Snowflake; DML and DDL are not supported ...
- 2 kudos
- 547 Views
- 3 replies
- 0 kudos
Resolved! API call fails to initiate create Service Principal secret
Hi,I've constructed an AWS lambda function which is used to auto rotate my Service Principal Secret in the Databricks account. Authentication is setup with OAuth2, the api call for the token generation is successful but when executing the api call to...
- 547 Views
- 3 replies
- 0 kudos
- 0 kudos
Your error message, "Invalid service principal id," typically indicates a mismatch or formatting problem with the service principal's unique identifier in your API request. Although you checked the client_id, this value is not always the one needed f...
- 0 kudos
- 1030 Views
- 1 replies
- 2 kudos
Databricks Taiwan User Community
Would be great have a group for databricks Taiwan community.
- 1030 Views
- 1 replies
- 2 kudos
- 2 kudos
Hey @kennyhsieh , hope it's not too late to reply to the post! A "Databricks User Group Taiwan" has recently been formed on LinkedIn.
- 2 kudos
- 3951 Views
- 2 replies
- 0 kudos
Can AWS workspaces share subnets?
The docs state:"You can choose to share one subnet across multiple workspaces or both subnets across workspaces."as well as:"You can reuse existing security groups rather than create new ones."and on this page:"If you plan to share a VPC and subnets ...
- 3951 Views
- 2 replies
- 0 kudos
- 0 kudos
AWS WorkSpaces can be configured with subnets that can be shared within an AWS account or across AWS accounts using resource sharing mechanisms, but this depends on the specific AWS service and context. For Databricks workspaces on AWS, documentation...
- 0 kudos
- 3427 Views
- 1 replies
- 0 kudos
Joblib with optuna and SB3
Hi everyone,I am training some reinforcement learning models and I am trying to automate the hyperparameter search using optuna. I saw in the documentation that you can use joblib with spark as a backend to train in paralel. I got that working with t...
- 3427 Views
- 1 replies
- 0 kudos
- 0 kudos
Stable Baselines 3 (SB3) models can be optimized with Optuna for hyperparameter search, but parallelizing these searches using Joblib with Spark as the backend—like the classic scikit-learn example—commonly encounters issues. The root problem is that...
- 0 kudos
