The Big Book of Generative AI brings together best practices and know-how for building production-quality GenAI applications. You’ll find technical content and code samples that will help you do everything from deploying your first application to bui...
- 281 Views
- 1 replies
- 2 kudos
Monday
Join tens of thousands of data leaders, engineers, scientists and architects from around the world at Moscone Center in San Francisco, June 10–13. Explore the latest advances in Apache Spark™, Delta Lake, MLflow, LangChain, PyTorch, dbt, Prest...
- 8316 Views
- 1 replies
- 5 kudos
02-12-2024
Get your first look at DBRX April 25, 2024 | 8 AM PT If you’re using off-the-shelf LLMs to build GenAI applications, you’re probably struggling with quality, privacy and governance issues. What you need is a way to cost-effectively build a custom LLM...
- 3123 Views
- 3 replies
- 3 kudos
3 weeks ago
Get live training from world-class experts on generative AI, machine learning, and data engineering. Save 50% for a limited time with the code : TRAIN50FOTY. Read more here.
- 138 Views
- 0 replies
- 0 kudos
yesterday
Community Activity
- 5098 Views
- 5 replies
- 7 kudos
Resolved! Installing libraries on job clusters
Simple question : what is the way to go to install libraries on job clusters ? There does not seem to be a "Libraries" tab on the UI as opposed to regular clusters. Does it mean that the only option is to use init scripts ?
- 5098 Views
- 5 replies
- 7 kudos
- 29 Views
- 0 replies
- 0 kudos
Hiring Databricks Data Architect roles
Hi,I am a recruiter and I am looking for places to post some data bricks I have coming out. I have several fully remote, high-level data databricks, architect roles. Of course I will post to LinkedIn, but I was just curious if there are any other pla...
- 29 Views
- 0 replies
- 0 kudos
- 50 Views
- 1 replies
- 0 kudos
Cached Views in MERGE INTO operation
Hi everyone!I want to use in-memory cached views in a merge into operation, but I am not entirely sure if the exactly saved in-memory view is used in this operation or not.So, suppose I have a table named table_1 and a cached view named cached_view_1...
- 50 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@deng_dev - Are you using external metastore by any chance. From the physical plan, we could see the catalog`.`db`.`table_1` is not cached. If it is glue catalog, then caching can be enabled based on the below configs in the article below https://do...
- 0 kudos
- 220 Views
- 5 replies
- 3 kudos
Azure Databricks with standard private link cluster event log error: "Metastore down"...
We have Azure Databricks with standard private link (back-end and front-end private link).We are able to successfully attach a Databricks workspace to the Databricks metastore (ADLS Gen2 storage).However, when trying to create tables in a catalog in ...
- 220 Views
- 5 replies
- 3 kudos
Latest Reply
- 3 kudos
can confirm that the approach will solve your error. Ran into a similar issue a while back.
- 3 kudos
- 21 Views
- 0 replies
- 0 kudos
Summit Watch Party
Hey hey Stockholm,Anyone keen on getting together to watch keynotes of the summit live? (for those of us who can't be there in person of course?/Karol
- 21 Views
- 0 replies
- 0 kudos
- 24 Views
- 0 replies
- 0 kudos
UC Enablement in Databricks workspace for metastores in different region - Azure Cloud
SpoilerWe have a UC enabled on Databricks where we have two Metastores,Metastore in West Europe regionMetastore in North America regionAnd Databricks workspace.Workspace in West Europe regionAnd we want to have a single Databricks workspace for both ...
- 24 Views
- 0 replies
- 0 kudos
- 24 Views
- 0 replies
- 0 kudos
Unity Catalog Enabled Clusters using PrivateNIC
Hello,When reviewing the VM settings for Databricks worker VMs, we can see that there are two(2) NICs.A primary ( PublicNIC (primary)) and a secondary (PrivateNIC (primary)).The workers VM is always assigned the PublicNIC and this is reachable from w...
- 24 Views
- 0 replies
- 0 kudos
- 32 Views
- 0 replies
- 1 kudos
How many times can we reschedule exam?
I would like to know how many time we can reschedule or postpone the certification exam.
- 32 Views
- 0 replies
- 1 kudos
- 4789 Views
- 15 replies
- 8 kudos
Resolved! What are some best practices for CICD?
A number of people have questions on using Databricks in a productionalized environment. What are the best practices to enable CICD automation?
- 4789 Views
- 15 replies
- 8 kudos
Latest Reply
- 8 kudos
Any leads/posts for Databricks CI/CD integration with Bitbucket pipeline. I am facing the below error while I creation my CICD pipeline pipelines:branches:master:- step:name: Deploy Databricks Changesimage: docker:19.03.12services:- dockerscript:# U...
- 8 kudos
- 32 Views
- 0 replies
- 0 kudos
Create Databricks model serving endpoint in Azure DevOps yaml
Hello,I need to create and destroy a model endpoint as part of CI/CD. I tried with mlflow deployments create-endpoint, giving databricks as --target however it errors saying that --endpoint is not a known argument when clearly --endpoint is required....
- 32 Views
- 0 replies
- 0 kudos
- 21 Views
- 0 replies
- 0 kudos
Error during deserializing protobuf data
I am receiving protobuf data in a json attribute and along with it I receive a descriptor file.I am using from_protobuf to deserialize the data as below,It works most of the time but giving error when there are some recursive fields within the protob...
- 21 Views
- 0 replies
- 0 kudos
- 227 Views
- 2 replies
- 0 kudos
Resolved! Not able to set run_as service_principal_name
I'm trying to run: databricks bundle deploy -t prod --profile PROD_Service_Principal My bundle looks: bundle: name: myproject include: - resources/jobs/bundles/*.yml targets: # The 'dev' target, for development purposes. This target is the de...
- 227 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
In my case I replaced alias PROD_Service_Principal with id c250831b-5a2a-4461-a855-83b9102f797e and it works. Not intuitive, probably this is a bug in CLI ot bundles service_principal_name: c250831b-5a2a-4461-a855-83b9102f797e
- 0 kudos
- 48 Views
- 0 replies
- 0 kudos
column "storage_sub_directory" is now always NULL in system.information_schema.tables
Hello,I am running a job that depends on the information provided in column storage_sub_directory in system.information_schema.tables .... and it worked until 1-2 weeks ago.Now I discovered in the doc that this column is deprecated and always null , ...
- 48 Views
- 0 replies
- 0 kudos
- 48 Views
- 0 replies
- 0 kudos
SparkContext lost when running %sh script.py
I need to execute a .py file in Databricks from a notebook (with arguments which for simplicity i exclude here). For this i am using:%sh script.pyscript.py:from pyspark import SparkContext def main(): sc = SparkContext.getOrCreate() print(sc...
- 48 Views
- 0 replies
- 0 kudos
- 80 Views
- 1 replies
- 1 kudos
Databricks Certification Exam Suspended. Kindly help to reschedule
@Cert-Team Hello team,Today, I had my Databricks Associate exam, but my exam was suspended without any reason. I tried logging in many times but was unable to log in and after some time it starts showing launch time is passed. I hope you can understa...
- 80 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Helloabhi_161999 Thank you for raising a ticket with Databricks Training & Certification regarding your issue, We are working on your request. Please allow us 24 hours to revert back to you.
- 1 kudos
Latest from our Blog
Prepare Your Journey to Migrate from AWS Glue Data Catalog to Databricks Unity Catalog
In this blog, we will look at the migration from AWS Glue Data Catalog to Unity Catalog. We cover how to plan this migration as a step-by-step approach and emphasize meticulous planning, phased migrat...
Mastering the Spark UI
Almost anyone who uses Spark or Databricks is aware of the Spark UI, and we all know that it’s a super powerful tool in the right hands. It can reveal what’s going wrong and any inefficiencies in you...
Optimize costs for your Data and AI workloads with Azure and AWS Spot Instances
Introduction Cost optimisation remains a pivotal challenge for customers dealing with processing large volumes of data and machine learning model training at scale in the cloud. Spot instances have re...
MLOps Gym - Beginners Guide to Cluster Configuration for MLOps
When setting up compute, there are many options and knobs to tweak and tune, and it can get quite overwhelming very quickly. To help you with optimally configuring your clusters, we have broken dow...
