Your feedback is crucial to us and directly influences how we innovate and improve our customer experience. To share your success and feedback with Databricks, please visit this link. The first 20 people to submit a completed survey response will ...
- 289 Views
- 0 replies
- 0 kudos
Wednesday
Thursday, September 12, 2024 | 9:00 AM-10:35 AM PT Effective governance is essential to democratizing data and AI in the age of LLMs and generative AI. Enterprises face challenges with fragmented governance across databases, warehouses, and lakes, c...
- 1272 Views
- 1 replies
- 2 kudos
3 weeks ago
Explore data and AI in 4 short videos In just one hour, you can get the free training you need to start using AI throughout your data processes. Find out how the Databricks Data Intelligence Platform democratizes data and AI for everyone in your orga...
- 1118 Views
- 0 replies
- 2 kudos
3 weeks ago
You're invited to join the Databricks Generative AI World Cup Virtual Hackathon, starting this week! Teams from all over the world, including Asia-Pacific & Japan, will tackle real-world challenges with cutting-edge Generative AI applications. Why J...
- 967 Views
- 6 replies
- 3 kudos
3 weeks ago
A Journey in Space With Apache Kafka and the Databricks Data Intelligence Platform This blog post is a follow-up to the session From Supernovas to LLMs at Data + AI Summit 2024, where I demonstrated how anyone can consume and process publicly availab...
- 1212 Views
- 0 replies
- 4 kudos
3 weeks ago
While there have been many incredible advancements in AI and Large Language models, there are many techniques that can be applied to make tailor these models and improve performance within a specific domain or task. This talk will guide users along ...
- 616 Views
- 0 replies
- 0 kudos
3 weeks ago
Over 12,000 companies have adopted data intelligence. Join a tour stop near you to discover how Databricks can unlock your data and AI too. Find your city now! Explore technologies Explore the latest advancements in the Databricks Data Intellige...
- 2204 Views
- 0 replies
- 4 kudos
07-23-2024
Community Activity
- 4 Views
- 0 replies
- 0 kudos
What are the Challenges of Integrating Big Data with Machine Learning Models in Databricks?
Hi everyone,I'm currently working on a project that involves building machine learning models on large datasets in Databricks. While Databricks offers great tools for handling big data, I'm facing some challenges when trying to integrate this data wi...
- 4 Views
- 0 replies
- 0 kudos
- 28236 Views
- 29 replies
- 14 kudos
Not received my certificate after passing Data Engineer Associate exam.
I've successfully passed Databricks Data Engineer Associate Certified exam but still have not received the certificate.-----------------------------------------------Hello Rupesh,You have completed the assessment, Databricks Certified Data Engineer A...
- 28236 Views
- 29 replies
- 14 kudos
Latest Reply
- 14 kudos
Hi @Nadia1, I have completed my exam on 5th Sept 2024 for DE professional, but i haven't received my certification. Can you please help mw with this. I have attached the screenshot of passing the test.
- 14 kudos
- 12008 Views
- 13 replies
- 10 kudos
Connecting SQL Server (on-premise) to Databricks via jdbc:sqlserver
Is it possible to connect to SQL Server on-premise (Not Azure) from Databricks?I tried to ping my virtualbox VM (with Windows Server 2022) from within Databricks and the request timed out.%sh ping 122.138.0.14This is what my connection might look l...
- 12008 Views
- 13 replies
- 10 kudos
Latest Reply
- 10 kudos
I tried to connect to localhost sql server through databricks community edition, but it failed. I have created an IP rule on port 1433 allowed inbound connection from all public network, but still didn't connect. I tried locally using python its work...
- 10 kudos
- 34 Views
- 0 replies
- 0 kudos
Switch from partner to customer academy
Hi there!I have incorrectly signed into Parner Academy portal instead of Customer Academy portal. Because of this issue I am unable to purchase an intended subscription plan. Could you please help me swapping membership to Customer Academy at the ear...
- 34 Views
- 0 replies
- 0 kudos
- 21 Views
- 0 replies
- 0 kudos
Error when query event log: Cannot read more than one event logs in the same query
I try to follow the instructions of Monitor Delta Live Tables pipelines to query dlt.expection log.Here is the simple code version I copied from the Querying the event log section:CREATE TEMPORARY LIVE VIEW event_log_raw AS (SELECT * FROM event_log(T...
- 21 Views
- 0 replies
- 0 kudos
- 680 Views
- 3 replies
- 0 kudos
Asset Bundles cannot run job with single node job cluster
Hello community,we are deploying a job using asset bundles and the job should run on a single node job cluster. Here is the DAB job definition:resources: jobs: example_job: name: example_job tasks: - task_key: main_task ...
- 680 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
@Volker @kunalmishra9 Tested the below element is not necessary.num_workers: 0
- 0 kudos
- 88 Views
- 4 replies
- 2 kudos
Resolved! How to create a DLT pipeline with SQL statement
I need a DLT pipeline to create a materialized view for fetching event logs. All the ways below I tried are failed:Attach a notebook with pure SQL inside: No magic cell like `%sql` are failedAttach a notebook with `spark.sql` python code: Failed beca...
- 88 Views
- 4 replies
- 2 kudos
Latest Reply
- 2 kudos
After just finishing my last reply, I realized what’s wrong with my code: I should use “file” property instead of “notebook” in the libraries section.It works now. Thank you guys, you are my rubber duck!
- 2 kudos
- 7276 Views
- 5 replies
- 0 kudos
Foreign catalog - Connections using insecure transport are prohibited --require_secure_transport=ON
I have added a connection to a MySql database in Azure, and I have created a foreign catalog in Databricks. But when I go to query the database I get the following error;Connections using insecure transport are prohibited while --require_secure_trans...
- 7276 Views
- 5 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi,Is there an update on this? I have the same issue where I try using Lakehouse federation but I cannot input the additional connection information or manually edit the url created when defining a "connection" on the databrick UI required for Lakeho...
- 0 kudos
- 107 Views
- 6 replies
- 2 kudos
Resolved! Databricks asset bundle does not create new job if I change configuration of existing Databricks yam
When deploying multiple jobs using the `Databricks.yml` file via the asset bundle, the process either overwrites the same job or renames it, instead of creating separate, distinct jobs.
- 107 Views
- 6 replies
- 2 kudos
Latest Reply
- 2 kudos
@filipniziolThank you for your valuable feedback. I believe the second approach aligns well with my requirements. I will implement a rollback mechanism in the target environment. Currently, I am performing all these tasks in the development environme...
- 2 kudos
- 48 Views
- 2 replies
- 0 kudos
Need to automatically rerun the failed jobs in databricks
Hi all, I need to retrigger the failed jobs automatically in data bricks, can you please help me with all the possible ways to make it possible
- 48 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @ranged_coop ,To automatically retrigger failed jobs in Databricks within the last 24 hours, you can use the Databricks REST API to list the jobs, filter out the failed runs, and then retrigger those failed jobs. Below is a Python script that will...
- 0 kudos
- 319 Views
- 4 replies
- 7 kudos
🚀 Databricks Custom Apps! 🚀
Whether you're a data scientist or a sales executive, Databricks is making it easier than ever to build, host, and share secure data applications. With our platform, you can now run any Python code on serverless compute, share it with non-technical c...
- 319 Views
- 4 replies
- 7 kudos
Latest Reply
- 7 kudos
I do consider that as a good direction - HOWEVER I had a lot of challenges while integrating with Delta Lake on low-code, no-code platforms such as PowerApps on Azure - this is why - I would love to see something like this on Databricks
- 7 kudos
- 1566 Views
- 3 replies
- 1 kudos
can we commit offset in spark structured streaming in databricks.
We are storing offset details in checkpoint location wanted to know is there a way can we commit offset once we consume the message from kafka.
- 1566 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @raphaelblg , thanks a lot for providing an elaborate answer. Do you happen to you, by any chance, of some solutions that developers use to track a consumer lag when streaming with Spark from a Kafka topic? It's a rather essential knowledge to hav...
- 1 kudos
- 35 Views
- 0 replies
- 0 kudos
Understanding and loading SQL Server Temp Tables from Databricks
Hi everyone...Came across this question in Stackoverflow and wanted to try my hand in trying it. Unfortunately I have not been able to fix it...https://stackoverflow.com/questions/78953930/create-and-load-sql-server-temp-table-table-or-table-from-dat...
- 35 Views
- 0 replies
- 0 kudos
- 72 Views
- 1 replies
- 0 kudos
Getting PermissionDenied in SDK When Updating External Location Isolation Mode
Using the Databricks SDK for Python in a notebook in a Databricks workspace, I'm creating an external location and then attempting to update the isolation mode and workspace bindings associated with the external location. The step to create the exter...
- 72 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Let me clean-up these cells for better readability:%pip install databricks-sdk --upgradedbutils.library.restartPython()from databricks.sdk import WorkspaceClient from databricks.sdk.service import catalogw = WorkspaceClient()# This works without issu...
- 0 kudos
- 2474 Views
- 2 replies
- 0 kudos
Obtain the source table version number from checkpoint file when using Structured Streaming
Hello!I'm using Structured Streaming to write to a delta table. The source is another delta table written with Structured Streaming as well. In order to datacheck the results I'm attempting to obtain from the checkpoint files of the target table the ...
- 2474 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @Kaniz_Fatma, thank you for your answer.I'm a bit confused here because you seem to be describing the opposite behavior of what I've seen in our checkpoint files.Here I repost my examples to try to understand better.First checkpoint file:{"sour...
- 0 kudos
Latest from our Blog
đź’ˇ ML Training Tip Of The Week #1: Optimizing GPU Utilization in Multi-Node Spark Clusters
When running distributed training or batch inference on multi-node GPU clusters with Spark, the GPUs on the Driver node often remain underutilized, resulting in unnecessary waste of GPU resources. The...
Data Validation: The Last Mile of Migration
Introduction Enterprise Data Warehouse (EDW) Migration use cases are always complex in nature. Data architects and Data engineers spend the majority of time in analyzing the source data, identifying t...
MLOps Gym - Getting started with version control
In the halcyon days of data science’s youth, version control was an oft-overlooked aspect of the work of data science teams, a “nice-to-have” that was perhaps the domain of hobbyists and enthusiasts. ...
Re-ingesting Data with Auto Loader
So imagine, you are a Data Engineer using Databricks Auto Loader to ingest data from cloud storage, when you suddenly realize that recently ingested data has some problem and you’d like to simply dele...
