cancel
Showing results for 
Search instead for 
Did you mean: 
Page Title

Welcome to the Databricks Community

Discover the latest insights, collaborate with peers, get help from experts and make meaningful connections

99589members
49879posts
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcing new training in Japanese and Brazilian Portuguese

Exciting news! Databricks is expanding available on-demand and instructor-led training offerings for Japan and Brazil. Two new introductory courses are now available: Databricks Fundamentals [Japanese | Brazilian Portuguese] A conceptual introduction...

  • 348 Views
  • 0 replies
  • 2 kudos
a week ago
Registration now open! Databricks Data + AI Summit 2024

Join tens of thousands of data leaders, engineers, scientists and architects from around the world at Moscone Center in San Francisco, June 10–13.  Explore the latest advances in Apache Spark™, Delta Lake, MLflow, LangChain, PyTorch, dbt, Prest...

  • 960 Views
  • 0 replies
  • 2 kudos
2 weeks ago
Calling all innovators and visionaries! The 2024 Data Team Awards are open for nominations

Each year, we celebrate the amazing customers that rely on Databricks to innovate and transform their organizations — and the world — with the power of data and AI. The nomination form is now open to submit nominations. Nominations will close on Marc...

  • 1329 Views
  • 0 replies
  • 0 kudos
2 weeks ago
Drive Value With Generative AI

Join us this March to see how generative AI can drive value. Build your own LLM trained on your data with Databricks’ new MPT model: Mosaic AI. Imagine if you could train and deploy your own LLM — customized with your own data that gives accurate res...

  • 1300 Views
  • 1 replies
  • 1 kudos
2 weeks ago

Community Activity

Peter_Jones
by New Contributor
  • 5 Views
  • 0 replies
  • 0 kudos

Syntax of UPDATE Command in DataBricks

Hi All,I am testing the sql generated by our ETL software to see if it can run on data bricks SQL which I believe is Delta Tables underneath. This is the statement we are testing. As far as I can tell from the manual the from clause is not supported ...

  • 5 Views
  • 0 replies
  • 0 kudos
isaac_gritz
by Valued Contributor II
  • 5054 Views
  • 3 replies
  • 4 kudos

Using Plotly Dash with Databricks

How to use Plotly Dash with DatabricksWe recommend checking out this article for the latest on building Dash Applications on top of the Databricks Lakehouse.Let us know in the comments if you use Plotly and if you're planning on adopting the latest i...

  • 5054 Views
  • 3 replies
  • 4 kudos
Latest Reply
dave-at-plotly
  • 4 kudos

Hey all.  Just wanted to make sure everyone had some up-to-date intel regarding leveraging Plotly Dash with Databricks.Most Dash app integrations w Databricks today leverage the Databricks Python SQL Connector.  More technical details are available v...

  • 4 kudos
2 More Replies
Brad
by New Contributor
  • 50 Views
  • 1 replies
  • 0 kudos

Dash in Databricks notebook directly

Hi team,Is there a way to embed plotly dash directly inside Databricks notebook?Thanks

  • 50 Views
  • 1 replies
  • 0 kudos
Latest Reply
dave-at-plotly
  • 0 kudos

Hey Brad. There is actually ... via a licensed library called databricks-dash which works similar to how jupyter-dash (and now dash itself) works in a jupyter notebook. The library is distributed as part of Plotly's Dash Enterprise. It's a bit of an ...

  • 0 kudos
277745
by New Contributor
  • 9 Views
  • 0 replies
  • 0 kudos

Pandas_Udod max batch size not working in notebook

Hello I am trying to set max batch size for pandas-udf in Databricks notebook, but in my tests it doesn’t have any effect on size. spark.conf.set("spark.sql.execution.arrow.enabled", "true")spark.conf.set('spark.sql.execution.arrow.maxRecordsPerBatch...

  • 9 Views
  • 0 replies
  • 0 kudos
shubham_sanyasi
by Visitor
  • 14 Views
  • 0 replies
  • 0 kudos

Request for Discount voucher for Databricks Data Engineer Associate Exam

Dear community,        I'm eager to enhance my skills in data engineering and analytics by obtaining certification as a Databricks Data Engineer Associate. Would it be possible to receive a free discount voucher for the exam to support my professiona...

  • 14 Views
  • 0 replies
  • 0 kudos
johnboscow
by Visitor
  • 22 Views
  • 0 replies
  • 0 kudos

Passed the Databricks Data Engineer Associate exam

Passed the Databricks Data Engineer Associate exam V3 on Sat 24 feb 2024. With a score of 96%. These questions really help. Please note settle differences between the questions here and the ones in the practice tests. They are small differences that ...

  • 22 Views
  • 0 replies
  • 0 kudos
YuvrajBonde
by Visitor
  • 157 Views
  • 1 replies
  • 0 kudos

Webaccessor Page got stucked

Hi, I am Yuvraj Bonde. Today, I attempted to take the Databricks Certified Data Engineer Associate exam. I followed the instructions properly and launched the exam. However, I encountered an issue with the web accessor portal. After following their i...

  • 157 Views
  • 1 replies
  • 0 kudos
Latest Reply
Cert-Team
Honored Contributor III
  • 0 kudos

Hi @YuvrajBonde as mentioned in your registration confirmation email, If you experience technical difficulties during set-up, launch, or within your Assessment, please contact KRYTERION Technical Support via chat or webform.Please review the followin...

  • 0 kudos
jfl0
by Visitor
  • 29 Views
  • 0 replies
  • 0 kudos

Help! Can't access webinar recording link in email.

Hi, I attended the "Quarterly Product Roadmap Webinar" session this week and got the email to the recording. However, once logged in, I get an error saying it is not available. Please advise how to access.  Thanks

  • 29 Views
  • 0 replies
  • 0 kudos
SirCrayon
by New Contributor
  • 47 Views
  • 0 replies
  • 0 kudos

Do shared clusters have multiple drivers?

Hi,I know that with single clusters, theres a single driver node and one driver per cluster. With shared clusters, multiple jobs can run concurrently. Does this still run on a single driver container or multiple driver containers run per application?...

  • 47 Views
  • 0 replies
  • 0 kudos
ss6
by New Contributor
  • 46 Views
  • 0 replies
  • 0 kudos

Liquid Cluster - SHOW CREATE TABLE error

We've got this table with liquid clustering turned on at first, but then we switch off with below command.ALTER TABLE table_name CLUSTER BY NONE;Now, our downstream process that usually runs "SHOW CREATE TABLE" is hitting a snag. It's throwing this e...

  • 46 Views
  • 0 replies
  • 0 kudos
Mathias
by New Contributor II
  • 1265 Views
  • 3 replies
  • 0 kudos

Different settings per target with Asset bundles

When generating the standard setup with databricks bundle init we will get databricks.yml that references resources/*. The targets are set in the databricks.yml and the resources (pipelines and jobs) are set in different files.I have dlt pipelines th...

  • 1265 Views
  • 3 replies
  • 0 kudos
Latest Reply
137292
New Contributor II
  • 0 kudos

-f is unknown shorthand flag for databricks bundle deploy. Any workaround on how to deploy different jobs with different targets?

  • 0 kudos
2 More Replies
Brad
by New Contributor
  • 88 Views
  • 2 replies
  • 0 kudos

Is there a way to control the cluster runtime version for DLT

Hi team, When I create a DLT job, is there a way to control the cluster runtime version somewhere? E.g. I want to use 14.3 LTS. I tried to add `"spark_version": "14.3.x-scala2.12",` inside cluster default label but not work.Thanks

  • 88 Views
  • 2 replies
  • 0 kudos
Latest Reply
Brad
New Contributor
  • 0 kudos

Thanks. I mean running DLT, not run a cell from notebook sourced for DLT.

  • 0 kudos
1 More Replies
Heisenberg
by New Contributor
  • 47 Views
  • 0 replies
  • 0 kudos

Migrate a workspace from one AWS account to another AWS account

Hi everyone,We have a Databricks workspace in an AWS account that we need to migrate to a new AWS account.The workspace has a lot of managed tables, workflows, saved queries, notebooks which need to be migrated, so looking for an efficient approach t...

Data Engineering
AWS
Databricks Migration
migration
queries
Workflows
  • 47 Views
  • 0 replies
  • 0 kudos
Mumrel
by New Contributor III
  • 56 Views
  • 1 replies
  • 0 kudos

Service Principal can be deleted but permissions not managed

On Azure I added a service principal X to my databricks workspace. I therefore had the Service Prinicpal Manager role on that service principal X. I accidentally downgraded my rights to Service Principal User and now can's get my Managers role back. ...

Mumrel_0-1708705677371.png
  • 56 Views
  • 1 replies
  • 0 kudos
Latest Reply
jamessmith3
New Contributor II
  • 0 kudos

Do you have federated identity enabled on your workspace?

  • 0 kudos
Dhruv_Sinha
by New Contributor
  • 118 Views
  • 2 replies
  • 1 kudos

Parallelizing processing of multiple spark dataframes

Hi all, I am trying to create a collection rd that contains a list of spark dataframes. I want to parallelize the cleaning process for each of these dataframes. Later on, I am sending each of these dataframes to another method. However, when I parall...

  • 118 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @Dhruv_Sinha, The issue you’re encountering—where the Spark context (sc) cannot be accessed from worker nodes—is a common challenge when working with Spark. Let’s explore why this happens and discuss potential workarounds. Spark Context and W...

  • 1 kudos
1 More Replies

Latest from our Blog

Databricks cost analysis and cross charge with Power BI

Authors: Liping Huang (@Liphuan) and Marius Panga (@mariuspc) Introduction Effective cost management is a critical consideration for any cloud data platform. Historically, achieving cost control and i...

820Views 1kudos