cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 
Databricks Learning Festival (Virtual): 15 January - 31 January 2025

Join us for the return of the Databricks Learning Festival (Virtual)! Mark your calendars from 15 January - 31 January 2025! Upskill today across data engineering, data analysis, machine learning, and generative AI. Join the thousands who have el...

  • 1924 Views
  • 4 replies
  • 2 kudos
2 weeks ago
The Future 50: The companies most likely to adapt, thrive, and grow

The global economy in 2024 is a tale of two forces: optimism sparked by falling interest rates, and the uncertainty caused by geopolitical unrest. As companies focus on sustainable growth, such a volatile environment inevitably tests their resilience...

  • 136 Views
  • 0 replies
  • 0 kudos
Thursday
Databricks Community Champion - December 2024 - Sujesh Menon

Meet Sujesh Menon, a valued member of our community! Sujesh is a Senior Data Engineer at Glanbia, Plc. He brings a wealth of knowledge and expertise to the group, and we are thrilled to have him here.  We presented him with a range of questions, and...

  • 281 Views
  • 3 replies
  • 4 kudos
Thursday
Databricks Brings AI to the Enterprise using NVIDIA AI and Accelerated Computing

The world of artificial intelligence (AI) and data analytics is about to get a significant boost, thanks to Databricks’ collaboration with NVIDIA. This work brings together the cutting-edge capabilities of Databricks’ Mosaic AI platform and NVIDIA AI...

  • 156 Views
  • 0 replies
  • 0 kudos
Thursday

Community Activity

DebIT2011
by > Visitor
  • 2 Views
  • 0 replies
  • 0 kudos

Choosing between Azure Data Factory (ADF) and Databricks PySpark notebooks

I’m working on a project where I need to pull large datasets from Cosmos DB into Databricks for further processing, and I’m trying to decide whether to use Azure Data Factory (ADF) or Databricks PySpark notebooks for the extraction and processing tas...

  • 2 Views
  • 0 replies
  • 0 kudos
anshi_t_k
by > New Contributor III
  • 500 Views
  • 3 replies
  • 0 kudos

Practice question for data engineer exam

A data engineer, User A, has promoted a pipeline to production by using the REST API to programmatically create several jobs. A DevOps engineer, User B, has configured an external orchestration tool to trigger job runs through the REST API. Both user...

  • 500 Views
  • 3 replies
  • 0 kudos
Latest Reply
brickster
New Contributor II
  • 0 kudos

Option C will be the right answer in my perspective. Though a job ownership is transferred to another user, creater_user field will never change. In fact, "Creater User" field in job details panel is non-editable field.If I am wrong, clarification wi...

  • 0 kudos
2 More Replies
cosmicwhoop
by > Visitor
  • 14 Views
  • 0 replies
  • 0 kudos

Delta Live Tables UI - missing EVENTS

I am new to Databricks and my setup is using Microsoft Azure (Premium Tier) + DatabricksI am trying to build Delta Live Tables and dont see events, without it i am finding it hard to understand the reason for job failure. Attached are 2 screenshot1) ...

  • 14 Views
  • 0 replies
  • 0 kudos
Pranav_jha
by > New Contributor III
  • 1502 Views
  • 10 replies
  • 3 kudos

Unable to Login - Account Verification Loop

I'm having trouble logging in to my Databricks account at databricks.com. Here's what happens:I enter my email address and password.I receive an account verification code via email.I enter the verification code on the login page.Instead of logging me...

  • 1502 Views
  • 10 replies
  • 3 kudos
Latest Reply
yss539
Visitor
  • 3 kudos

I was also facing the same problem and clicking on the Sign in to Community Edition  option at the bottom of the page helped me.

  • 3 kudos
9 More Replies
chethankumar
by > New Contributor III
  • 37 Views
  • 1 replies
  • 0 kudos

How do I set up Databricks observability using AWS cloudwatch

How do I set up Databricks observability, including metrics and logging? I am using AWS-based Databricks and want to monitor it. I plan to use CloudWatch as the observability tool but couldn't find proper documentation to configure it.

  • 37 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @chethankumar, Setting up CloudWatch might required customer setup, by setting clusters using init script. You might want to consider contacting your account team to guide you through the process. Additional please find the post which could be use...

  • 0 kudos
HeckGunderson
by > New Contributor
  • 38 Views
  • 1 replies
  • 0 kudos

Where can I quickly test custom metric queries?

I'm working on adapting some custom metrics to add to the tables' dashboard. Right now when I throw in a test query, I need to refresh metrics and it takes 5-10 minutes to let me know I've probably forgotten a parenthesis somewhere.Where can I test t...

  • 38 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @HeckGunderson, Have you tried running the code using notebook with your Databricks workspace? https://docs.databricks.com/en/notebooks/index.html Also you can do it via CLI using SQL connector. https://docs.databricks.com/ja/dev-tools/python-sql-...

  • 0 kudos
NIK251
by > Visitor
  • 36 Views
  • 1 replies
  • 0 kudos

Delta Live Table Pipeline

I have the error message when try to create a delta live table pipeline.My error is: com.databricks.pipelines.common.errors.deployment.DeploymentException: Failed to launch pipeline cluster 1207-112912-8e84v9h5: Encountered Quota Exhaustion issue in ...

  • 36 Views
  • 1 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 0 kudos

Hi @NIK251 ,Try to run this pipeline once again. If the error appears, check what message is logged in azure activity log. 

  • 0 kudos
Mithos
by > Visitor
  • 13 Views
  • 0 replies
  • 0 kudos

ZCube Tags not present in Databricks Delta Tables

The design doc for Liquid Clustering for Delta refer to Z-Cube to enable  incremental clustering in batches. This is the link - https://docs.google.com/document/d/1FWR3odjOw4v4-hjFy_hVaNdxHVs4WuK1asfB6M6XEMw/edit?pli=1&tab=t.0.It is also mentioned th...

  • 13 Views
  • 0 replies
  • 0 kudos
bantarobugs
by > New Contributor
  • 273 Views
  • 1 replies
  • 0 kudos

Job Run failure - Azure Container does not exist

Hello,I have an ETL pipeline in Databricks that works perfectly when I execute it manually in the notebook using an all-purpose cluster. However, when I try to schedule it using a job cluster, it fails immediately with the error message: 'Azure conta...

Screenshot 2024-08-28 154926.png
  • 273 Views
  • 1 replies
  • 0 kudos
Latest Reply
PiotrMi
New Contributor
  • 0 kudos

Hey @bantarobugs There might be a problem with the permissions or roles assigned to the user or service principal trying to access the Azure container. Please check who/what is assigned and it role/permission: 

  • 0 kudos
DBUser2
by > New Contributor III
  • 256 Views
  • 1 replies
  • 0 kudos

Simba ODBC batch queries

I'm using Simba ODBC driver to Connect to databricks. Since this driver doesn't support transactions, I was trying to run a DELETE and then INSERT query from a within a single execute, but I get an error. Is there an alternate way to perform a batch ...

  • 256 Views
  • 1 replies
  • 0 kudos
Latest Reply
PiotrMi
New Contributor
  • 0 kudos

Hey @DBUser2 It looks that delete is not supported at all based on documentation: Write-backThe Simba Apache Spark ODBC Connector supports translation for the followingsyntax when connecting to a Spark Thrift Server instance that is running Spark 1.3...

  • 0 kudos
Eren_DE
by > New Contributor
  • 640 Views
  • 2 replies
  • 0 kudos

legacy Git integration has been removed from Notebook

How to integrate a notebook saved in workspace folder with git repos with feature branch? 

  • 640 Views
  • 2 replies
  • 0 kudos
Latest Reply
FierceSkirtsist
  • 0 kudos

Integrating a Databricks notebook with a Git repository using a feature branch sounds like a clean workflow for version control. The step-by-step process makes it straightforward to collaborate and track changes effectively. It's great that Databrick...

  • 0 kudos
1 More Replies
vivekn
by > New Contributor III
  • 809 Views
  • 8 replies
  • 3 kudos

Resolved! Machine Learning with Databricks

Hi,I am going through the Machine Learning with Databricks course under the learning festival. However, I do not see any link to the repository/notebooks/material where we can download and configure it in my environment. Could you help us sharing the...

  • 809 Views
  • 8 replies
  • 3 kudos
Latest Reply
Steven19977
New Contributor II
  • 3 kudos

Hey, I have prepared for my Machine Learning in just one week and passed my exam with the help of practice questions from certsleader

  • 3 kudos
7 More Replies
tgburrin-afs
by > New Contributor
  • 254 Views
  • 2 replies
  • 0 kudos

Limiting concurrent tasks in a job

I have a job with > 10 tasks in it that interacts with an external system outside of databricks.  At the moment that external system cannot handle more than 3 of the tasks executing concurrently.  How can I limit the number of tasks that concurrently...

  • 254 Views
  • 2 replies
  • 0 kudos
Latest Reply
filipniziol
Contributor III
  • 0 kudos

Hi @tgburrin-afs, @Mounika_Tarigop ,As I understand the question is about running concurrent tasks within a single job rather than running concurrent jobs. max_concurrent_runs controls how many times a whole job can run simultaneously, not the concur...

  • 0 kudos
1 More Replies
iemaiman
by > Visitor
  • 23 Views
  • 0 replies
  • 0 kudos

Wrong Email For Registering

Hi Team, at the time of creating/scheduling exam for Databricks Data Engineer Associate I accidentally put wrong Email address which is iamaimanansari@gmail.com whereas it should have been iamansariaiman@gmail.com.  My exam is in next one hour. Can s...

  • 23 Views
  • 0 replies
  • 0 kudos
tommyhmt
by > New Contributor
  • 331 Views
  • 1 replies
  • 0 kudos

Add CreatedDate to Delta Live Table

Hi all,I have a very simple DLT set up using the following code:@dlt.view( name="view_name", comment="comments" ) def vw_DLT(): return spark.readStream.format("cloudFiles").option("cloudFiles.format", "csv").load(file_location) dlt.create_stre...

tommyhmt_0-1723758970190.png tommyhmt_1-1723759255869.png
  • 331 Views
  • 1 replies
  • 0 kudos
Latest Reply
Mounika_Tarigop
Databricks Employee
  • 0 kudos

To add a CreatedDate column that captures the timestamp when a record is first inserted into the table, you can modify your Delta Live Tables (DLT) pipeline setup as follows: 1) Define the schema for your streaming table to include the CreatedDate co...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Featured Event

Join Us for an Exclusive Databricks Community Event in San Francisco!

Thursday, December 12, 2024

View Event
Top Kudoed Authors
Read Databricks Data Intelligence Platform reviews on G2

Latest from our Blog

Understanding Unity Catalog

Throughout the dozens of engagements I’ve had since joining Databricks, I’ve found that customers often struggle to understand the scope and concept of Unity Catalog. Questions like “Does it store my ...

1042Views 3kudos