cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 
Submit your feedback and win a $25 gift card!

Your feedback is crucial to us and directly influences how we innovate and improve our customer experience.  To share your success and feedback with Databricks, please  visit this link.  The first 20 people to submit a completed survey response will ...

  • 289 Views
  • 0 replies
  • 0 kudos
Wednesday
Databricks Unity Catalog Workshop

Thursday, September 12, 2024 | 9:00 AM-10:35 AM PT Effective governance is essential to democratizing data and AI in the age of LLMs and generative AI. Enterprises face challenges with fragmented governance across databases, warehouses, and lakes, c...

  • 1272 Views
  • 1 replies
  • 2 kudos
3 weeks ago
Upskill on Databricks in just an hour

Explore data and AI in 4 short videos In just one hour, you can get the free training you need to start using AI throughout your data processes. Find out how the Databricks Data Intelligence Platform democratizes data and AI for everyone in your orga...

  • 1118 Views
  • 0 replies
  • 2 kudos
3 weeks ago
Join us at the Databricks Generative AI World Cup (Virtual Hackathon)

You're invited to join the Databricks Generative AI World Cup Virtual Hackathon, starting this week! Teams from all over the world, including Asia-Pacific & Japan, will tackle real-world challenges with cutting-edge Generative AI applications.  Why J...

  • 967 Views
  • 6 replies
  • 3 kudos
3 weeks ago
Supernovas, Black Holes and Streaming Data

A Journey in Space With Apache Kafka and the Databricks Data Intelligence Platform This blog post is a follow-up to the session From Supernovas to LLMs at Data + AI Summit 2024, where I demonstrated how anyone can consume and process publicly availab...

  • 1212 Views
  • 0 replies
  • 4 kudos
3 weeks ago
Customizing your Models: RAG, Fine-Tuning, and Pre-Training

While there have been many incredible advancements in AI and Large Language models, there are many techniques that can be applied to make tailor these models and improve performance within a specific domain or task. This talk will guide users along ...

  • 616 Views
  • 0 replies
  • 0 kudos
3 weeks ago
Data + AI World Tour 2024

Over 12,000 companies have adopted data intelligence. Join a tour stop near you to discover how Databricks can unlock your data and AI too. Find your city now!  Explore technologies Explore the latest advancements in the Databricks Data Intellige...

  • 2204 Views
  • 0 replies
  • 4 kudos
07-23-2024

Community Activity

leoarthur
by > New Contributor
  • 4 Views
  • 0 replies
  • 0 kudos

What are the Challenges of Integrating Big Data with Machine Learning Models in Databricks?

Hi everyone,I'm currently working on a project that involves building machine learning models on large datasets in Databricks. While Databricks offers great tools for handling big data, I'm facing some challenges when trying to integrate this data wi...

  • 4 Views
  • 0 replies
  • 0 kudos
rpatel
by > New Contributor III
  • 28236 Views
  • 29 replies
  • 14 kudos

Not received my certificate after passing Data Engineer Associate exam.

I've successfully passed Databricks Data Engineer Associate Certified exam but still have not received the certificate.-----------------------------------------------Hello Rupesh,You have completed the assessment, Databricks Certified Data Engineer A...

  • 28236 Views
  • 29 replies
  • 14 kudos
Latest Reply
Pk4
Visitor
  • 14 kudos

Hi @Nadia1, I have completed my exam on 5th Sept 2024 for DE professional, but i haven't received my certification. Can you please help mw with this. I have attached the screenshot of passing the test.

  • 14 kudos
28 More Replies
Tico23
by > Contributor
  • 12008 Views
  • 13 replies
  • 10 kudos

Connecting SQL Server (on-premise) to Databricks via jdbc:sqlserver

Is it possible to connect to SQL Server on-premise (Not Azure) from Databricks?I tried to ping my virtualbox VM (with Windows Server 2022) from within Databricks and the request timed out.%sh   ping 122.138.0.14This is what my connection might look l...

  • 12008 Views
  • 13 replies
  • 10 kudos
Latest Reply
BharathKumarS
  • 10 kudos

I tried to connect to localhost sql server through databricks community edition, but it failed. I have created an IP rule on port 1433 allowed inbound connection from all public network, but still didn't connect. I tried locally using python its work...

  • 10 kudos
12 More Replies
rg_databricks
by > New Contributor
  • 34 Views
  • 0 replies
  • 0 kudos

Switch from partner to customer academy

Hi there!I have incorrectly signed into Parner Academy portal instead of Customer Academy portal. Because of this issue I am unable to purchase an intended subscription plan. Could you please help me swapping membership to Customer Academy at the ear...

  • 34 Views
  • 0 replies
  • 0 kudos
guangyi
by > Contributor
  • 21 Views
  • 0 replies
  • 0 kudos

Error when query event log: Cannot read more than one event logs in the same query

I try to follow the instructions of Monitor Delta Live Tables pipelines to query dlt.expection log.Here is the simple code version I copied from the Querying the event log section:CREATE TEMPORARY LIVE VIEW event_log_raw AS (SELECT * FROM event_log(T...

  • 21 Views
  • 0 replies
  • 0 kudos
Volker
by > New Contributor III
  • 680 Views
  • 3 replies
  • 0 kudos

Asset Bundles cannot run job with single node job cluster

Hello community,we are deploying a job using asset bundles and the job should run on a single node job cluster. Here is the DAB job definition:resources: jobs: example_job: name: example_job tasks: - task_key: main_task ...

  • 680 Views
  • 3 replies
  • 0 kudos
Latest Reply
AlbertWang
New Contributor III
  • 0 kudos

@Volker @kunalmishra9  Tested the below element is not necessary.num_workers: 0 

  • 0 kudos
2 More Replies
guangyi
by > Contributor
  • 88 Views
  • 4 replies
  • 2 kudos

Resolved! How to create a DLT pipeline with SQL statement

I need a DLT pipeline to create a materialized view for fetching event logs. All the ways below I tried are failed:Attach a notebook with pure SQL inside: No magic cell like `%sql` are failedAttach a notebook with `spark.sql` python code: Failed beca...

  • 88 Views
  • 4 replies
  • 2 kudos
Latest Reply
guangyi
Contributor
  • 2 kudos

After just finishing my last reply, I realized what’s wrong with my code: I should use “file” property instead of “notebook” in the libraries section.It works now. Thank you guys, you are my rubber duck!

  • 2 kudos
3 More Replies
TimB
by > New Contributor III
  • 7276 Views
  • 5 replies
  • 0 kudos

Foreign catalog - Connections using insecure transport are prohibited --require_secure_transport=ON

I have added a connection to a MySql database in Azure, and I have created a foreign catalog in Databricks. But when I go to query the database I get the following error;Connections using insecure transport are prohibited while --require_secure_trans...

  • 7276 Views
  • 5 replies
  • 0 kudos
Latest Reply
J-Bradlee
Visitor
  • 0 kudos

Hi,Is there an update on this? I have the same issue where I try using Lakehouse federation but I cannot input the additional connection information or manually edit the url created when defining a "connection" on the databrick UI required for Lakeho...

  • 0 kudos
4 More Replies
sandy311
by > New Contributor
  • 107 Views
  • 6 replies
  • 2 kudos

Resolved! Databricks asset bundle does not create new job if I change configuration of existing Databricks yam

When deploying multiple jobs using the `Databricks.yml` file via the asset bundle, the process either overwrites the same job or renames it, instead of creating separate, distinct jobs.

  • 107 Views
  • 6 replies
  • 2 kudos
Latest Reply
sandy311
New Contributor
  • 2 kudos

@filipniziolThank you for your valuable feedback. I believe the second approach aligns well with my requirements. I will implement a rollback mechanism in the target environment. Currently, I am performing all these tasks in the development environme...

  • 2 kudos
5 More Replies
Vishalakshi
by > Visitor
  • 48 Views
  • 2 replies
  • 0 kudos

Need to automatically rerun the failed jobs in databricks

Hi all, I need to retrigger the failed jobs automatically in data bricks, can you please help me with all the possible ways to make it possible 

  • 48 Views
  • 2 replies
  • 0 kudos
Latest Reply
filipniziol
New Contributor II
  • 0 kudos

Hi @ranged_coop ,To automatically retrigger failed jobs in Databricks within the last 24 hours, you can use the Databricks REST API to list the jobs, filter out the failed runs, and then retrigger those failed jobs. Below is a Python script that will...

  • 0 kudos
1 More Replies
Ajay-Pandey
by > Esteemed Contributor III
  • 319 Views
  • 4 replies
  • 7 kudos

🚀 Databricks Custom Apps! 🚀

Whether you're a data scientist or a sales executive, Databricks is making it easier than ever to build, host, and share secure data applications. With our platform, you can now run any Python code on serverless compute, share it with non-technical c...

  • 319 Views
  • 4 replies
  • 7 kudos
Latest Reply
PiotrU
Contributor
  • 7 kudos

I do consider that as a good direction - HOWEVER I had a lot of challenges while integrating with Delta Lake on low-code, no-code platforms such as PowerApps on Azure - this is why - I would love to see something like this on Databricks

  • 7 kudos
3 More Replies
Nis
by > New Contributor II
  • 1566 Views
  • 3 replies
  • 1 kudos

can we commit offset in spark structured streaming in databricks.

We are storing offset details in checkpoint location wanted to know is there a way can we commit offset once we consume the message from kafka.

  • 1566 Views
  • 3 replies
  • 1 kudos
Latest Reply
dmytro
New Contributor II
  • 1 kudos

Hi @raphaelblg , thanks a lot for providing an elaborate answer. Do you happen to you, by any chance, of some solutions that developers use to track a consumer lag when streaming with Spark from a Kafka topic? It's a rather essential knowledge to hav...

  • 1 kudos
2 More Replies
ranged_coop
by > Valued Contributor II
  • 35 Views
  • 0 replies
  • 0 kudos

Understanding and loading SQL Server Temp Tables from Databricks

Hi everyone...Came across this question in Stackoverflow and wanted to try my hand in trying it. Unfortunately I have not been able to fix it...https://stackoverflow.com/questions/78953930/create-and-load-sql-server-temp-table-table-or-table-from-dat...

  • 35 Views
  • 0 replies
  • 0 kudos
MarkV
by > New Contributor
  • 72 Views
  • 1 replies
  • 0 kudos

Getting PermissionDenied in SDK When Updating External Location Isolation Mode

Using the Databricks SDK for Python in a notebook in a Databricks workspace, I'm creating an external location and then attempting to update the isolation mode and workspace bindings associated with the external location. The step to create the exter...

  • 72 Views
  • 1 replies
  • 0 kudos
Latest Reply
MarkV
New Contributor
  • 0 kudos

Let me clean-up these cells for better readability:%pip install databricks-sdk --upgradedbutils.library.restartPython()from databricks.sdk import WorkspaceClient from databricks.sdk.service import catalogw = WorkspaceClient()# This works without issu...

  • 0 kudos
Agus1
by > New Contributor III
  • 2474 Views
  • 2 replies
  • 0 kudos

Obtain the source table version number from checkpoint file when using Structured Streaming

Hello!I'm using Structured Streaming to write to a delta table. The source is another delta table written with Structured Streaming as well. In order to datacheck the results I'm attempting to obtain from the checkpoint files of the target table the ...

  • 2474 Views
  • 2 replies
  • 0 kudos
Latest Reply
Agus1
New Contributor III
  • 0 kudos

Hello @Kaniz_Fatma, thank you for your answer.I'm a bit confused here because you seem to be describing the opposite behavior of what I've seen in our checkpoint files.Here I repost my examples to try to understand better.First checkpoint file:{"sour...

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors

Latest from our Blog

Data Validation: The Last Mile of Migration

Introduction Enterprise Data Warehouse (EDW) Migration use cases are always complex in nature. Data architects and Data engineers spend the majority of time in analyzing the source data, identifying t...

301Views 3kudos

MLOps Gym - Getting started with version control

In the halcyon days of data science’s youth, version control was an oft-overlooked aspect of the work of data science teams, a “nice-to-have” that was perhaps the domain of hobbyists and enthusiasts. ...

271Views 1kudos

Re-ingesting Data with Auto Loader

So imagine, you are a Data Engineer using Databricks Auto Loader to ingest data from cloud storage, when you suddenly realize that recently ingested data has some problem and you’d like to simply dele...

416Views 1kudos