Get started with Databricks Asset Bundles (DABs) today! Simplify project management, versioning, testing, and deployment for your data and AI projects on the Databricks Platform. Embrace software engineering best practices with DABs—source control, c...
- 208 Views
- 0 replies
- 1 kudos
yesterday
For a limited time, we're offering 50% off training and certification at Data + AI Summit with the following code: TRAIN50FOTY. This offer expires on May 3, 2024. Register today and add training. Read more here!
- 164 Views
- 0 replies
- 0 kudos
Tuesday
The Big Book of Generative AI brings together best practices and know-how for building production-quality GenAI applications. You’ll find technical content and code samples that will help you do everything from deploying your first application to bui...
- 421 Views
- 1 replies
- 2 kudos
Monday
Get your first look at DBRX April 25, 2024 | 8 AM PT If you’re using off-the-shelf LLMs to build GenAI applications, you’re probably struggling with quality, privacy and governance issues. What you need is a way to cost-effectively build a custom LLM...
- 3257 Views
- 3 replies
- 3 kudos
3 weeks ago
Community Activity
- 7 Views
- 0 replies
- 0 kudos
Migrate a notebook that reside in workspace using Databricks Asset Bundle
Hello Community Folks -Did anyone implemented migration of notebooks that is in workspace to production databricks workspace using Databricks Asset Bundle? If so can you please help me with any documentation which I can refer? Thanks!!RegardsNiruban ...
- 7 Views
- 0 replies
- 0 kudos
- 167 Views
- 1 replies
- 0 kudos
Create Databricks managed service principal programatically ?
For the current Databricks service principal API or the Databricks SDK, an ID is required. However, when dealing with Databricks-managed service principals, you typically only have the name. For registering with cloud providers, like Microsoft Entra ...
- 167 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@SunilPoluri, could you please provide some more context by providing screenshots?
- 0 kudos
- 59 Views
- 2 replies
- 0 kudos
Databricks SQL connectivity in Python with Service Principals
Tried to use M2M OAuth connectivity on Databricks SQL Warehouse in Python:from databricks.sdk.core import Config, oauth_service_principal from databricks import sql .... config = Config(host=f"https://{host}", client_...
- 59 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
@harripy , could you please brief us about your use case? So that we can understand the issue better.
- 0 kudos
- 56 Views
- 1 replies
- 0 kudos
SparkContext lost when running %sh script.py
I need to execute a .py file in Databricks from a notebook (with arguments which for simplicity i exclude here). For this i am using:%sh script.pyscript.py:from pyspark import SparkContext def main(): sc = SparkContext.getOrCreate() print(sc...
- 56 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@madrhr I think this occurs because one session is initiated within the Python script (.py file), while in the Databricks notebook, we have a pre-configured Spark session. It is important to note that we cannot use more than one Spark session per not...
- 0 kudos
- 127 Views
- 1 replies
- 0 kudos
UC Enablement in Databricks workspace for metastores in different region - Azure Cloud
SpoilerWe have a UC enabled on Databricks where we have two Metastores,Metastore in West Europe regionMetastore in North America regionAnd Databricks workspace.Workspace in West Europe regionAnd we want to have a single Databricks workspace for both ...
- 127 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@Nivethan A metastore is the top-level container for data in Unity Catalog. Unity Catalog metastores register metadata about securable objects (such as tables, volumes, external locations, and shares) and the permissions that govern access to them. E...
- 0 kudos
- 29 Views
- 0 replies
- 0 kudos
Menophix: Is This a Real, Risk-Free Formula? Verified Customer Reviews! (UK)
Sale is Live Now: https://www.biowellgrade.com/menophix-detox-reviews/A called MenoPhix aids women in controlling the symptoms of menopause. Its organic makeup encourages the equilibrium of hormones in the body. This nutritional supplement is safe fo...
- 29 Views
- 0 replies
- 0 kudos
- 5209 Views
- 6 replies
- 7 kudos
Resolved! Installing libraries on job clusters
Simple question : what is the way to go to install libraries on job clusters ? There does not seem to be a "Libraries" tab on the UI as opposed to regular clusters. Does it mean that the only option is to use init scripts ?
- 5209 Views
- 6 replies
- 7 kudos
- 208 Views
- 0 replies
- 1 kudos
Announcing the General Availability of Databricks Asset Bundles
Get started with Databricks Asset Bundles (DABs) today! Simplify project management, versioning, testing, and deployment for your data and AI projects on the Databricks Platform. Embrace software engineering best practices with DABs—source control, c...
- 208 Views
- 0 replies
- 1 kudos
- 131 Views
- 0 replies
- 0 kudos
Hiring Databricks Data Architect roles
Hi,I am a recruiter and I am looking for places to post some data bricks I have coming out. I have several fully remote, high-level data databricks, architect roles. Of course I will post to LinkedIn, but I was just curious if there are any other pla...
- 131 Views
- 0 replies
- 0 kudos
- 121 Views
- 1 replies
- 0 kudos
Cached Views in MERGE INTO operation
Hi everyone!I want to use in-memory cached views in a merge into operation, but I am not entirely sure if the exactly saved in-memory view is used in this operation or not.So, suppose I have a table named table_1 and a cached view named cached_view_1...
- 121 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@deng_dev - Are you using external metastore by any chance. From the physical plan, we could see the catalog`.`db`.`table_1` is not cached. If it is glue catalog, then caching can be enabled based on the below configs in the article below https://do...
- 0 kudos
- 316 Views
- 5 replies
- 3 kudos
Azure Databricks with standard private link cluster event log error: "Metastore down"...
We have Azure Databricks with standard private link (back-end and front-end private link).We are able to successfully attach a Databricks workspace to the Databricks metastore (ADLS Gen2 storage).However, when trying to create tables in a catalog in ...
- 316 Views
- 5 replies
- 3 kudos
Latest Reply
- 3 kudos
can confirm that the approach will solve your error. Ran into a similar issue a while back.
- 3 kudos
- 33 Views
- 0 replies
- 0 kudos
Summit Watch Party
Hey hey Stockholm,Anyone keen on getting together to watch keynotes of the summit live? (for those of us who can't be there in person of course?/Karol
- 33 Views
- 0 replies
- 0 kudos
- 37 Views
- 0 replies
- 0 kudos
Unity Catalog Enabled Clusters using PrivateNIC
Hello,When reviewing the VM settings for Databricks worker VMs, we can see that there are two(2) NICs.A primary ( PublicNIC (primary)) and a secondary (PrivateNIC (primary)).The workers VM is always assigned the PublicNIC and this is reachable from w...
- 37 Views
- 0 replies
- 0 kudos
- 46 Views
- 0 replies
- 1 kudos
How many times can we reschedule exam?
I would like to know how many time we can reschedule or postpone the certification exam.
- 46 Views
- 0 replies
- 1 kudos
- 4832 Views
- 15 replies
- 8 kudos
Resolved! What are some best practices for CICD?
A number of people have questions on using Databricks in a productionalized environment. What are the best practices to enable CICD automation?
- 4832 Views
- 15 replies
- 8 kudos
Latest Reply
- 8 kudos
Any leads/posts for Databricks CI/CD integration with Bitbucket pipeline. I am facing the below error while I creation my CICD pipeline pipelines:branches:master:- step:name: Deploy Databricks Changesimage: docker:19.03.12services:- dockerscript:# U...
- 8 kudos
Announcements
Latest from our Blog
Prepare Your Journey to Migrate from AWS Glue Data Catalog to Databricks Unity Catalog
In this blog, we will look at the migration from AWS Glue Data Catalog to Unity Catalog. We cover how to plan this migration as a step-by-step approach and emphasize meticulous planning, phased migrat...
Mastering the Spark UI
Almost anyone who uses Spark or Databricks is aware of the Spark UI, and we all know that it’s a super powerful tool in the right hands. It can reveal what’s going wrong and any inefficiencies in you...
Optimize costs for your Data and AI workloads with Azure and AWS Spot Instances
Introduction Cost optimisation remains a pivotal challenge for customers dealing with processing large volumes of data and machine learning model training at scale in the cloud. Spot instances have re...
MLOps Gym - Beginners Guide to Cluster Configuration for MLOps
When setting up compute, there are many options and knobs to tweak and tune, and it can get quite overwhelming very quickly. To help you with optimally configuring your clusters, we have broken dow...
