Certifications
Access industry-recognized credentials to validate your expertise in data engineering, analytics, an...
0 Posts
Activity in Learning & Certification
- 105 Views
- 0 replies
- 1 kudos
Databricks on GCP: Opening new ports for Databricks
To improve scalability and enable new features, the Databricks classic compute plane will start communicating to the Databricks control plane over 10 new ports: 3306 and 8443-8451. You may need to update your network configurations before we start us...
- 105 Views
- 0 replies
- 1 kudos
- 100 Views
- 0 replies
- 0 kudos
Databricks on AWS: Opening new ports for Databricks
To improve scalability and enable new features, Databricks classic compute plane will start communicating to the Databricks control plane over 10 new ports: 3306 and 8443-8451. You may need to update your network configurations before we start using ...
- 100 Views
- 0 replies
- 0 kudos
- 451 Views
- 0 replies
- 0 kudos
Network Configuration for Accessing Azure Key Vault from Enhanced Python UDFs in Unity Catalog
Network configuration doesn’t have to be hard! In this article, I will walk you through different scenarios of network configuration for accessing Azure Key Vault(AKV) from Databricks. AKV backed Databricks Secrets Scopes Network configuration is str...
- 451 Views
- 0 replies
- 0 kudos
- 259 Views
- 0 replies
- 1 kudos
Strengthening Data Privacy with Enhanced Python UDFs in Unity Catalog
This article walks you through a step-by-step guide of how to strengthen Data Privacy with enhanced Python UDFs in Databricks Unity Catalog Step 1, Strengthen security of encryption key or hash salt key used for data protection with Databricks Unit...
- 259 Views
- 0 replies
- 1 kudos
- 8722 Views
- 2 replies
- 1 kudos
Getting started with Databricks - Deploy an LLM chatbot using Databricks Vector Search and Foundation Model API
Getting started with Databricks - Deploy an LLM chatbot using Databricks Vector Search and Foundation Model API This guide demonstrates how to build a simple Retrieval-Augmented Generation (RAG) application using Databricks Vector Search, a vect...
- 8722 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
If you want to try UC and Vector Search, you will need a paid Databricks workspace with UC enabled and proper compute access. See docs here https://docs.databricks.com/aws/en/data-governance/unity-catalog/get-started.
- 1 kudos
- 6674 Views
- 6 replies
- 4 kudos
[Azure] Outbound public IP address ranges for the control plane will be updated May 20, 2025
To improve the security and zone support availability of the Azure Databricks control plane, we'll be updating the outbound public IP address range and associated Databricks service tags on May 20, 2025. Required Action: If you use a resource firewal...
- 6674 Views
- 6 replies
- 4 kudos
- 1448 Views
- 0 replies
- 0 kudos
[Azure] Azure Databricks Upgrading to Standard SKU Public IP Addresses
Due to the upcoming retirement of Static IPv4's Basic SKU, Azure Databricks will gradually start adopting the Standard SKU for new public IPs on 11 April 2025. As part of this change, the cost of new public Static IPv4 SKUs will increase from $0.004 ...
- 1448 Views
- 0 replies
- 0 kudos
- 34911 Views
- 4 replies
- 9 kudos
Getting Started with Databricks - Building a Forecasting Model on Databricks
Building a Forecasting Model on Databricks: A Step-by-Step Guide This guide offers a detailed, step-by-step approach for building a forecasting model on Databricks. By leveraging the power of Databricks, you will unlock new potentials in your data w...
- 34911 Views
- 4 replies
- 9 kudos
- 23606 Views
- 6 replies
- 3 kudos
We are opening additional ports for Azure Databricks
IMPORTANT NOTE: we have delayed opening additional ports for new workspaces until January 31, 2025. For existing workspaces, Network Security Groups (NSGs) have been updated. If you received a communication that updates for your existing workspaces f...
- 23606 Views
- 6 replies
- 3 kudos
Latest Reply
- 3 kudos
Hello @shiv_DB25 According to me, your NSG config is fine. The rule 65001 on your outbound should allow your cluster to access internet.Not sure it will help, but there is this line in your log:'Error code: ERROR_NO_MATCHING_DISTRIBUTION'I don't know...
- 3 kudos
- 2353 Views
- 0 replies
- 1 kudos
[Action required] [GCP] Databricks is updating the public IP range for our control plane services
This blog is to notify you that on January 20, 2025, Databricks will begin using new control plane service components on Google Cloud Platform (GCP). This will affect the egress and ingress IP addresses Databricks’ control plane uses. If you use a fi...
- 2353 Views
- 0 replies
- 1 kudos
- 19556 Views
- 2 replies
- 13 kudos
Getting Started with Databricks - Build a simple Lakehouse analytics pipeline
Getting started with Databricks - Build a simple Lakehouse analytics pipeline The demo example in this guide illustrates a lakehouse analytics pipeline using the well-known NYC taxi trip dataset. This public dataset is also available at Kaggle. This...
- 19556 Views
- 2 replies
- 13 kudos
- 52135 Views
- 21 replies
- 6 kudos
[Action required] [Azure] update outbound connectivity for classic workspaces
IMPORTANT NOTE: We have indefinitely delayed the automatic enforcement described below for workspaces that had enabled workspace IP access lists prior to July 29, 2024. We still recommend manually enforcing IP access lists on compute plane requests i...
- 52135 Views
- 21 replies
- 6 kudos
Latest Reply
- 6 kudos
@Abadoom you must have configured backend private link as well, including setting requiredNsgRules=NoAzureDatabricksRules CC @rugger-bricks @kyrrewk if your traffic goes through Azure Firewall, then you can add the IPs of the Azure Firewall instance...
- 6 kudos
- 4707 Views
- 0 replies
- 0 kudos
Get started with an exploratory analysis on Databricks using Sample Data
Getting started with Databricks - Exploratory analysis This guide walks you through using a Databricks notebook to query sample data stored in Unity Catalog using Python and then visualize the query results in the notebook.This is a beginner’s tutori...
- 4707 Views
- 0 replies
- 0 kudos
- 12541 Views
- 0 replies
- 4 kudos
Update your UC AWS IAM Roles to include self-assume capabilities by September 20th, 2024
Background In order for Unity Catalog to moderate fine-grained access control in S3 cloud storage, Unity Catalog performs STS:AssumeRole operations on the IAM roles that underlie storage credentials. This article details the upcoming changes to the r...
- 12541 Views
- 0 replies
- 4 kudos
- 15999 Views
- 6 replies
- 0 kudos
[Action required] [AWS] update outbound connectivity for classic workspaces
IMPORTANT NOTE: We have indefinitely delayed the automatic enforcement described below for workspaces that had enabled workspace IP access lists prior to July 29, 2024. We still recommend manually enforcing IP access lists on compute plane requests i...
- 15999 Views
- 6 replies
- 0 kudos
Latest Reply
- 0 kudos
@SathwickKollipa if I understood your set-up correctly, all compute plane traffic will egress through proxy before hitting the control plane. In this case, allow-listing the IPs of the proxy in the workspace IP ACL should be sufficient. @krikotti ca...
- 0 kudos
