cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 
Databricks Learning Festival (Virtual): 15 January - 31 January 2025

Join us for the return of the Databricks Learning Festival (Virtual)! Mark your calendars from 15 January - 31 January 2025! Upskill today across data engineering, data analysis, machine learning, and generative AI. Join the thousands who have el...

  • 109930 Views
  • 248 replies
  • 69 kudos
11-26-2024
Share Your Feedback in Our Community Survey

Your opinion matters! Take a few minutes to complete our Customer Experience Survey to help us improve the Databricks Community. Your input is crucial in shaping the future of our community and ensuring it meets your needs. Take the Survey Now Why p...

  • 973 Views
  • 0 replies
  • 0 kudos
2 weeks ago
Databricks Named a Leader in the 2024 Gartner® Magic Quadrant™ for Cloud Database Management Systems

We’re thrilled to share that Databricks has once again been recognized as a Leader in the 2024 Gartner® Magic Quadrant™ for Cloud Database Management Systems. This acknowledgement underscores our commitment to innovation and our leadership in the dat...

  • 1829 Views
  • 0 replies
  • 4 kudos
3 weeks ago
Milestone: DatabricksTV Reaches 100 Videos!

We are thrilled to announce that DatabricksTV, our growing video hub, has hit a major milestone: 100 videos and counting! What is DatabricksTV?DatabricksTV is a community-driven video hub designed to help data practitioners maximize the Databricks e...

  • 1759 Views
  • 1 replies
  • 4 kudos
12-11-2024
Announcing the new Meta Llama 3.3 model on Databricks

The Meta Llama 3.3 multilingual large language model (LLM) is a pretrained and instruction tuned generative model in 70B (text in/text out). The Llama 3.3 instruction tuned text only model is optimized for multilingual dialogue use cases and outperfo...

  • 2764 Views
  • 0 replies
  • 3 kudos
12-11-2024

Community Activity

RameshRetnasamy
by > Contributor
  • 11602 Views
  • 25 replies
  • 22 kudos

Resolved! Unable to login to Azure Databricks Account Console

I have a personal Azure pay-as-you-go subscription in which I have the 'Global Administrator' role. I am also the databricks account administrator.Until two weeks ago, I was able to access the databricks account console without any issues, but I am f...

Screenshot 2024-08-07 at 12.13.53.png Screenshot 2024-08-07 at 12.15.30.png Screenshot 2024-08-07 at 12.17.20.png Screenshot 2024-08-07 at 12.18.35.png
Administration & Architecture
account-console
Databricks
  • 11602 Views
  • 25 replies
  • 22 kudos
Latest Reply
satyaki_guha
  • 22 kudos

Hi @RameshRetnasamy thanks for your solution.I was able to login to Databricks Account Console but was unable to set the metastore path.Something got broke along the way.It would definitely be beneficial if we could have the same email address for ou...

  • 22 kudos
24 More Replies
sshukla
by > New Contributor III
  • 48 Views
  • 5 replies
  • 0 kudos

External Api not returning any response

import requestsurl = "https://example.com/api"headers = {"Authorization": "Bearer YOUR_TOKEN","Content-Type": "application/json"}Payload = json.dumps({json_data})response = requests.post(url, headers=headers, data=Payload)print(response.status_code)p...

  • 48 Views
  • 5 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Can you DIM the two payload, it has to do with something on endpoint, since it is working on one payload fine, where in the other is not to the same endpoint.

  • 0 kudos
4 More Replies
Brad
by > Contributor II
  • 21 Views
  • 1 replies
  • 0 kudos

How to add shared libs

Hi team,I want to add some shared libs which might be used by many repos, e.g. some util functions which might be used by any repos.1. What is the recommended way to add those libs? E.g. create a separate repo and reference it in another repo?2. How ...

  • 21 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @Brad, You can use cluster libraries so whenever interacting with repos libraries are available. Also the cluster can be shared to others. https://docs.databricks.com/en/libraries/cluster-libraries.html. About permissions you can control them over...

  • 0 kudos
RamaSubbaReddy
by > New Contributor III
  • 956 Views
  • 9 replies
  • 1 kudos

Resolved! Missed the certification exam schedule - Reschedule is required

Hi Team, @data_help @helpdesk I have missed the certification exam schedule. I thought it is 2PM instead of 2AM. Is there a possibility this can be reschedule to today anytime or tomorrow?Thanks in advance for your comments.Regards,Rama

  • 956 Views
  • 9 replies
  • 1 kudos
Latest Reply
Shekhar748
Visitor
  • 1 kudos

Hi Team, @data_help @helpdesk I had registered for the Databricks data engineer associate certification exam.I have missed the certification exam schedule, by mistake I choose 5am slot instead of 5pm. Is there a possibility this can be rescheduled?Lo...

  • 1 kudos
8 More Replies
Dinesh_Negi
by > New Contributor II
  • 769 Views
  • 2 replies
  • 0 kudos

Databricks Certified Data Analyst Associate Exam Suspended

With a great regret ,I am explaining my pain of suspending Databricks Certified Data Analyst Associate Exam scheduled on  August 16, 2024, at 14:00 P.M IST .I was trying to do my exam with honest and loylty .I even did not moved for a second here and...

  • 769 Views
  • 2 replies
  • 0 kudos
Latest Reply
Alwin121
Visitor
  • 0 kudos

Surprising news about the suspension of the Databricks Certified Data Analyst Associate Exam. It was one of the most sought-after certifications in the data analytics space. During my preparation, resources like P2PCerts were incredibly helpful in co...

  • 0 kudos
1 More Replies
Farshid_Jamali
by > New Contributor II
  • 3599 Views
  • 2 replies
  • 2 kudos

Practice Questions for Machine Learning Associate Exam

Are there any Practice Questions for Machine Learning Associate Exam?

  • 3599 Views
  • 2 replies
  • 2 kudos
Latest Reply
Alwin121
Visitor
  • 2 kudos

The Machine Learning Associate Exam focuses on core topics like ML foundations, supervised/unsupervised learning, model evaluation, feature engineering, and ML deployment. P2PCerts material covers these thoroughly, offering concise explanations and p...

  • 2 kudos
1 More Replies
Rishabh_Tiwari
by Databricks Employee
  • 158 Views
  • 5 replies
  • 4 kudos

🎊 2025 Mood Check: Emoji Style 🎊

As we kick off the new year, let’s set the tone for an exciting 2025! Drop an emoji in the comments to share your vibe:  = Ready to build cutting-edge data pipelines! = Excited to master new Databricks features! = Innovating with data and AI solution...

  • 158 Views
  • 5 replies
  • 4 kudos
Latest Reply
hari-prasad
Valued Contributor II
  • 4 kudos

  • 4 kudos
4 More Replies
michaelh
by > New Contributor III
  • 3959 Views
  • 5 replies
  • 4 kudos

Resolved! AWS Databricks Cluster terminated.Reason:Container launch failure

We're developing custom runtime for databricks cluster. We need to version and archive our clusters for client. We made it run successfully in our own environment but we're not able to make it work in client's environment. It's large corporation with...

  • 3959 Views
  • 5 replies
  • 4 kudos
Latest Reply
NandiniN
Databricks Employee
  • 4 kudos

This appears to be an issue with the security group. Kindly review security group inbound/outbound rules.

  • 4 kudos
4 More Replies
franc_bomb
by > New Contributor
  • 62 Views
  • 7 replies
  • 0 kudos

Cluster creation issue

Hello,I just started using Databricks community version for learning purposes.I have been trying to create a cluster but the first time it failed asking me to retry or contact the support, and now it's just running forever.What could be the problem? 

  • 62 Views
  • 7 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Can you please perform one test, check on the cloud provider if you are able to start a node?

  • 0 kudos
6 More Replies
AlexSantiago
by > New Contributor II
  • 3165 Views
  • 17 replies
  • 4 kudos

spotify API get token - raw_input was called, but this frontend does not support input requests.

hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...

  • 3165 Views
  • 17 replies
  • 4 kudos
Latest Reply
avamax44
New Contributor
  • 4 kudos

How do top-followed accounts balance personal and professional content?Authenticity & Relatability: Personal content, such as behind-the-scenes moments, personal stories, and daily life updates, helps influencers connect with their audience on a huma...

  • 4 kudos
16 More Replies
elio
by > Visitor
  • 8 Views
  • 0 replies
  • 0 kudos

gta v

gyty

  • 8 Views
  • 0 replies
  • 0 kudos
KosmaS
by > New Contributor III
  • 814 Views
  • 2 replies
  • 0 kudos

Skewness / Salting with countDistinct

Hey Everyone,I experience data skewness for: df = (source_df .unionByName(source_df.withColumn("region", lit("Country"))) .groupBy("zip_code", "region", "device_type") .agg(countDistinct("device_id").alias("total_active_unique"), count("device_id").a...

Screenshot 2024-08-05 at 17.24.08.png
  • 814 Views
  • 2 replies
  • 0 kudos
Latest Reply
singhvikash86
  • 0 kudos

What about salt function is function on device_id produces mutually exclusive results like hash(device_id) % 101 and then one more aggregation to sum of these counts group by zip_code, region, device_type

  • 0 kudos
1 More Replies
garciargs
by > New Contributor
  • 41 Views
  • 1 replies
  • 0 kudos

Incremental load from two tables

Hi, I am looking to build a ETL process for a incremental load silver table.This silver table, lets say "contracts_silver", is built by joining two bronze tables, "contracts_raw" and "customer".contracts_silverCONTRACT_IDSTATUSCUSTOMER_NAME1SIGNEDPet...

  • 41 Views
  • 1 replies
  • 0 kudos
Latest Reply
hari-prasad
Valued Contributor II
  • 0 kudos

Hi @garciargs ,Yes, in databricks you can do it using DLT (Delta Live Table) and Spark Structured Streaming, where you have to enable CDF (Change Data Feed) on both contracts_raw and customer_raw which would track all DML changes over raw tables.-- N...

  • 0 kudos
kyrrewk
by > New Contributor II
  • 25 Views
  • 2 replies
  • 0 kudos

Monitor progress when using databricks-connect

When using databricks-connect how can you monitor the progress? Ideally, we want something similar to what you get in the Databricks notebook, i.e., information about the jobs/stages. We are using Python.

  • 25 Views
  • 2 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

When you refer to progress, you mean that during the notebook execution you can see the Spark jobs processing for each cell?

  • 0 kudos
1 More Replies
leymariv
by > New Contributor
  • 81 Views
  • 1 replies
  • 0 kudos

Performance issue writing an extract of a huge unpartitionned single column dataframe

I have a huge df (40 billions rows) shared by delta share that has only one column 'payload' which contains json and that is not partitionned:Even if all those payloads are not the same, they have a common col sessionId that i need to extract to be a...

leymariv_2-1737155764713.png leymariv_0-1737155486874.png
  • 81 Views
  • 1 replies
  • 0 kudos
Latest Reply
hari-prasad
Valued Contributor II
  • 0 kudos

Hi @leymariv,You can check the schema of data in delta sharing table, using df.printSchema to better understand the JSON structure. Use from_json function to flatten or normalize the data to respective columns.Additionally, you can understand how dat...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Featured Event

Join Us for an Exclusive Databricks Community Event in San Francisco!

Thursday, January 23, 2025

View Event
Top Kudoed Authors
Read Databricks Data Intelligence Platform reviews on G2

Latest from our Blog

Deep Dive - Streaming Deduplication

In this article we will cover in depth about streaming deduplication using watermarking with dropDuplicates and dropDuplicatesWithinWatermark, how they are different. This blog expects you to have a g...

475Views 1kudos

Data Engineering SQL Holiday Specials

December is the most celebrated time of year in the Data Engineering calendar as we embrace the important holiday: change freeze season.  As we come back to the office to start our new projects, I wan...

2622Views 3kudos