Join the Databricks Learning Festival (Virtual)! Mark your calendars from 10 October - 31 October 2024! Upskill today across data engineering, data analysis, machine learning, and generative AI. Join the thousands who have elevated their career w...
- 27868 Views
- 96 replies
- 38 kudos
3 weeks ago
Databricks Apps, a new way to build and deploy internal data and AI applications, is now available in Public Preview on AWS and Azure. Ideal use cases include data visualization, AI applications, self-service analytics, and data quality monitoring.It...
- 396 Views
- 0 replies
- 3 kudos
Tuesday
Meet Szymon Dybczak, a valued member of our community! Szymon is a Senior Data Engineer at Nordcloud. He brings a wealth of knowledge and expertise to the group, and we're thrilled to have him here. We presented him with a range of questions, and be...
- 947 Views
- 8 replies
- 8 kudos
a week ago
Don't Miss Out on "Making AI-Powered Data Engineering Practical"! Join us for this exciting virtual event where we’ll cut through the hype and explore how AI-powered data intelligence is transforming data engineering. AMER: Nov 4 / 10 AM PT EMEA: ...
- 1647 Views
- 1 replies
- 2 kudos
2 weeks ago
Shifting to customized GenAI that deeply understands your data AMER October 8 / 10 AM PTEMEA October 9 / 9 AM BST / 10 AM CESTAPJ October 10 / 12 PM SGT Click here to check out the agenda and speakers and register now! Why are 9 out of 10 organiz...
- 917 Views
- 2 replies
- 2 kudos
3 weeks ago
Get practical guidance, notebooks, code snippets You know this better than anyone: the best GenAI models in the world will not succeed without good data. That’s why data engineers are even more critical today. The challenge is staying ahead of the ra...
- 1728 Views
- 0 replies
- 3 kudos
3 weeks ago
Community Activity
- 31 Views
- 2 replies
- 1 kudos
Differences among python libraries
I am confused as to the differences between various python libraries for databricks: especially with regard to differences among [databricks-connect](https://pypi.org/project/databricks-connect/), [databricks-api](https://pypi.org/project/databricks-...
- 31 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
@szymon_dybczak,Thank you for typing all that up. It is very clear and helpful.Two follow ups if I may:1. If one's primary goal is to execute SQL queries why prefer databricks sql connector over a generic jdbc or odbc package?2. Did I miss any other ...
- 1 kudos
- 14 Views
- 2 replies
- 0 kudos
Asset Bundles Job If Condition
HiI am trying to use Asset Bundles to deploy a job with email notifications active only on production environment.I tried to use such if statement but it does not seem to do anything.resources: jobs: Master_Load: name: Master Load e...
- 14 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @kunex ,You don't have to use if condition. The recommended approach is to use define "shared" configuration at the top level resources mapping and then when you need environment specific configuration use targets top level-level mapping to overri...
- 0 kudos
- 59 Views
- 3 replies
- 0 kudos
What does DLT INITIALIZING phase do?
In Delta Live Tables, the INITIALIZING phase takes sometimes a minute, sometimes 5 minutes. I'd like to learn what is it doing in the background, and can it be optimized in any way.
- 59 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @antr, In Delta Live Tables (DLT), a feature of Databricks, the "Initializing" phase refers to the first step in the lifecycle of a DLT pipeline run. During this phase, the platform sets up the necessary resources, configurations, and dependencies...
- 0 kudos
- 20 Views
- 1 replies
- 0 kudos
Databricks as a "pure" data streaming software like Confluent
DearsI was wondering if anyone has leveraged Databricks as a "pure" data streaming software in place of Confluent, Flink, Kafka etc.I see the reference architectures placing Databricks on the data processing side mostly once data is made available by...
- 20 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @noorbasha534 ,It depends on what you're asking for. Kafka is primarily a messaging system, optimized for handling high-throughput, distributed message logs. Databricks can read from Kafka as a data source but doesn't replace Kafka's role in messa...
- 0 kudos
- 42 Views
- 1 replies
- 0 kudos
Unity Catalog
Can we set up unity catalog on databricks community edition?If yes, please share the process.Thanks
- 42 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @dyusuf ,No, it's not possible. You can setup free azure trial account though and then you should be able to create UC.
- 0 kudos
- 14 Views
- 0 replies
- 0 kudos
Join Us for the Databricks Pittsburgh User Group Meetup!
Date & Time:Tuesday, October 22, 20244:00 PM - 6:00 PM Location:The Foundry Table381 N Shore Drive, Pittsburgh, PA 15212 Event Details We're excited to invite you to the next Databricks Pittsburgh User Group! This is a community for local users, ...
- 14 Views
- 0 replies
- 0 kudos
- 35 Views
- 1 replies
- 0 kudos
Databricks Exam got Suspended. Need Urgent help
Dear Support Team,I hope this message finds you well. I am writing to express my concern regarding the recent suspension of my Databricks Associate Data Engineer certification exam, which I took on 10 October 2024(16:45 Asia/Calcutta). During the exa...
- 35 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @Chandru13 , Thank you for reaching out to Databricks Community support! We are sorry to hear that your exam was suspended.. Thank you for filing a ticket with our support team. Please allow the support team 24-48 hours for a resolution. In the...
- 0 kudos
- 296 Views
- 2 replies
- 0 kudos
Resolved! Request for Certification Discount Voucher
Hey I am requesting databricks to provide one certification voucher as i have already the taken the exam couple of weeks ago but unfortunately i failed in the exam however i was 2 correct answers away from passing the test.I need to crack this course...
- 296 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Vishalkhode1206 there is this promotion going on now: https://community.databricks.com/t5/events/databricks-learning-festival-virtual-10-october-31-october/ev-p/90710
- 0 kudos
- 30 Views
- 1 replies
- 0 kudos
Can't create table in unity catalog.
Hi all.I have created a Databricks workspace on AWS. I can log into the workspace and successfully perform select operations on files in S3, but I am unable to create tables.The error when creating the table is as follows. "Your request failed with s...
- 30 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
I attach the screenshot when I encounter the error.
- 0 kudos
- 31 Views
- 0 replies
- 0 kudos
Set up compute policy to allow installing python libraries from a private package index
In our organization, we maintain a bunch of libraries we share code with. They're hosted on a private python package index, which requires a token to allow downloads. My idea was to store the token as a secret which would then be loaded into a cluste...
- 31 Views
- 0 replies
- 0 kudos
- 117 Views
- 2 replies
- 1 kudos
Silly question-Easy way to show full notebook path or owner in UI?
We have a few people working in Databricks right now in different clones of the same repository. Occasionally we'll have multiple people with the same branch open- one working, another just has it open to see what it looks like, sort of deal.This has...
- 117 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @Kayla ,I don't know if that's what you looking for, but for sure full path and owner information are available:
- 1 kudos
- 5776 Views
- 7 replies
- 0 kudos
Connect to Databricks from PowerApps
Hi All,Currently I trying to connect databricks Unity Catalog from Powerapps Dataflow by using spark connector specifying http url and using databricks personal access token as specified in below screenshot: I am able to connect but the issue is when...
- 5776 Views
- 7 replies
- 0 kudos
Latest Reply
- 0 kudos
Do you have any firewall restriction? Maybe if the public IP are not allowed for your databricks workspace it is not allowing thew connection, besides that I don't have much clue on how to help. Would suggest open a support ticket.
- 0 kudos
- 47 Views
- 0 replies
- 0 kudos
Optimising Data Integration and Serving patterns with Cosmos DB and Azure Databricks
Introduction In today's data-driven world, organisations aconstantly seekways to optimise their data integration and serving patterns. This blog post delves into the powerful combination of Azure Cosmos DB and Azure Databricks, exploring advanced int...
- 47 Views
- 0 replies
- 0 kudos
- 29 Views
- 0 replies
- 0 kudos
Help to lead the Vancouver usergroup
The Databricks community team needs your help to lead and shape this local community. Why You Should Step Up: Membership: You are already a member of the Databricks community in this region, which gives you a unique perspective on topics relevant to ...
- 29 Views
- 0 replies
- 0 kudos
- 105 Views
- 4 replies
- 0 kudos
Help to lead the bay Area usergroup
The Databricks community team needs your help to lead and shape this local community. Why You Should Step Up: Membership: You are already a member of the Databricks community in this region, which gives you a unique perspective on topics relevant to ...
- 105 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Mike_Gerardi Thank you so much for offering your support and space for hosting meetups—this is fantastic news! We're excited to have both you and @DChoi_WestMonro leading the Bay Area user group together as co-leads. Your collaboration will bri...
- 0 kudos
Latest from our Blog
Optimising Data Integration and Serving patterns with Cosmos DB and Azure Databricks
Introduction In today's data-driven world, organisations aconstantly seekways to optimise their data integration and serving patterns. This blog post delves into the powerful combination of Azure Cosm...
Private and Dedicated Connectivity Patterns for Databricks Serverless Using Private Link (Part 2)
Intro In the first part of our series, we explored the architecture and some typical use cases for a new feature in Databricks that allows customers to connect their Serverless clusters to private res...
SAM2 on Databricks
In this post we’ll walk through getting started with Meta’s latest Segment-Anything-Model 2 (SAM2) on Databricks. We’ll cover experimentation with SAM2 in a Databricks Notebook, expand on the default ...
Dynamic Segmentation in Geospatial Analytics on Databricks - Part 2
This is the second part of a two-part series blog on geospatial data processing on Databricks. In the first part, we covered ingesting and processing Overture Maps data on Databricks. In this second ...
