Your feedback is crucial to us and directly influences how we innovate and improve our customer experience. To share your success and feedback with Databricks, please visit this link. The first 20 people to submit a completed survey response will ...
- 115 Views
- 0 replies
- 0 kudos
Wednesday
Thursday, September 12, 2024 | 9:00 AM-10:35 AM PT Effective governance is essential to democratizing data and AI in the age of LLMs and generative AI. Enterprises face challenges with fragmented governance across databases, warehouses, and lakes, c...
- 1167 Views
- 1 replies
- 2 kudos
2 weeks ago
Explore data and AI in 4 short videos In just one hour, you can get the free training you need to start using AI throughout your data processes. Find out how the Databricks Data Intelligence Platform democratizes data and AI for everyone in your orga...
- 1037 Views
- 0 replies
- 2 kudos
3 weeks ago
You're invited to join the Databricks Generative AI World Cup Virtual Hackathon, starting this week! Teams from all over the world, including Asia-Pacific & Japan, will tackle real-world challenges with cutting-edge Generative AI applications. Why J...
- 917 Views
- 6 replies
- 2 kudos
3 weeks ago
A Journey in Space With Apache Kafka and the Databricks Data Intelligence Platform This blog post is a follow-up to the session From Supernovas to LLMs at Data + AI Summit 2024, where I demonstrated how anyone can consume and process publicly availab...
- 1085 Views
- 0 replies
- 4 kudos
3 weeks ago
While there have been many incredible advancements in AI and Large Language models, there are many techniques that can be applied to make tailor these models and improve performance within a specific domain or task. This talk will guide users along ...
- 595 Views
- 0 replies
- 0 kudos
3 weeks ago
Over 12,000 companies have adopted data intelligence. Join a tour stop near you to discover how Databricks can unlock your data and AI too. Find your city now! Explore technologies Explore the latest advancements in the Databricks Data Intellige...
- 2143 Views
- 0 replies
- 4 kudos
07-23-2024
Community Activity
- 80 Views
- 1 replies
- 0 kudos
Failed to read job commit marker error
Recently, we migrate from DBR 11.3 LTS ML to DBR 14.3 LTS ML. We are struggling on one data source where we consume parquet files. New data are appended every 30 minutes to that data source. The data are generated by Databricks notebook which runs on...
- 80 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @ak4, this is an expected behavior with DBIO transactional commit enabled and most likely the issue is when you update a table and query it immediately. You could explicitly invalidate the cache or configure disk cache. If you have a long running ...
- 0 kudos
- 9 Views
- 0 replies
- 0 kudos
Runtime increases exponentially from 11.3 to 13.3
Hello. I am using R on databricks and using the below approach. My Spark version:Single node: i3.2xlarge · On-demand · DBR: 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) · us-east-1a, the job takes 1 hourI install all R packages (including a geo...
- 9 Views
- 0 replies
- 0 kudos
- 2952 Views
- 5 replies
- 9 kudos
Databricks Cluster Web terminal different permissions with tmux and xterm.
I am launching web terminal on my databricks cluster and when I am using the ephemeral xterm instance I am easily able to navigate to desired directory in `Workspace` and run anything... for example `ls ./` When I switch to tmux so that I can preserv...
- 2952 Views
- 5 replies
- 9 kudos
Latest Reply
- 9 kudos
Hey there, fellow data explorer pals! I totally get your excitement when launching that web terminal on your Databricks cluster and feeling the power of running commands like 'ls ./' in the ephemeral xterm instance. It's like traversing the vast univ...
- 9 kudos
- 44 Views
- 2 replies
- 0 kudos
DatabricksApiException Error in Microsoft Azure Databricks
I am doing a course on Machine Learning Associate course , at the starting itseld i am getting error while running in Azure Databricks.Can somebody help me in solving this error.
- 44 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
It sounds like you're facing a Databricks API exception in Microsoft Azure. These errors often stem from issues related to authentication, permissions, or the configuration of your Azure Databricks workspace. Start by checking your Azure Active Direc...
- 0 kudos
- 1246 Views
- 4 replies
- 0 kudos
spotify API get token - raw_input was called, but this frontend does not support input requests.
hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...
- 1246 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
When working with the Spotify API and dealing with authentication issues, especially around token generation, it's important to ensure you're following the correct OAuth flow. Double-check your backend setup for token handling and make sure the raw i...
- 0 kudos
- 25 Views
- 1 replies
- 0 kudos
Databricks asset bundle does not create new job if I change configuration of existing Databricks yam
When deploying multiple jobs using the `Databricks.yml` file via the asset bundle, the process either overwrites the same job or renames it, instead of creating separate, distinct jobs.
- 25 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @sandy311 ,could you share your databricks.yml file?Are you sure you used unique job ids when defining your jobs?
- 0 kudos
- 917 Views
- 6 replies
- 2 kudos
Join us at the Databricks Generative AI World Cup (Virtual Hackathon)
You're invited to join the Databricks Generative AI World Cup Virtual Hackathon, starting this week! Teams from all over the world, including Asia-Pacific & Japan, will tackle real-world challenges with cutting-edge Generative AI applications. Why J...
- 917 Views
- 6 replies
- 2 kudos
- 40 Views
- 2 replies
- 1 kudos
Resolved! alter DLT Materialized View alter column set MASK
I am trying to mask a column on a DLT materialized view - this is created using DLT syntax. I am not able set the column masking after creation. Appreciate any workaround.alter DLT Materialized View alter column set MASK
- 40 Views
- 2 replies
- 1 kudos
- 1469 Views
- 2 replies
- 1 kudos
can we commit offset in spark structured streaming in databricks.
We are storing offset details in checkpoint location wanted to know is there a way can we commit offset once we consume the message from kafka.
- 1469 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @raphaelblg , thanks a lot for providing an elaborate answer. Do you happen to you, by any chance, of some solutions that developers use to track a consumer lag when streaming with Spark from a Kafka topic? It's a rather essential knowledge to hav...
- 1 kudos
- 45 Views
- 0 replies
- 0 kudos
Promicil – Fußpilz – Kosten? Kundenfeedback? (Austria)
Der Verkauf ist jetzt live: https://www.gesundlebenprofi.com/befehl-promicilEine starke Lotion namens Promicil wird häufig zur Behandlung von Pilzinfektionen der Füße verwendet. Die biologische Lösung ist kompliziert. Bei anhaltender Anwendung linder...
- 45 Views
- 0 replies
- 0 kudos
- 32 Views
- 0 replies
- 0 kudos
Real User Insights on Benefits and Ingredients
Although, like my Grandma repeats to me, "The devil has the power to take a pleasing shape." They're just being politically correct. These guys are the ones who check out everyhttps://www.facebook.com/groups/534976209094294 . I wasn't certain what th...
- 32 Views
- 0 replies
- 0 kudos
- 29 Views
- 0 replies
- 0 kudos
Certification Registration clarification
Hi All,I am interested to complete some fundamental databricks certification, for example "Learn Lakehouse Fundamentals". So when I am going for that it's asking for some details before starting with it.So now my only concern is, while registering wh...
- 29 Views
- 0 replies
- 0 kudos
- 1984 Views
- 17 replies
- 9 kudos
The session expires every second in Databricks Community Edition.
Getting irritated by frequent logging in. Every next second it is asking for login"Session expiredYour session has expired, please authenticate again."Very Frustating and pathetic experience with Databricks.
- 1984 Views
- 17 replies
- 9 kudos
Latest Reply
- 9 kudos
I think, the issue seems resolved now. I am no longer getting frequent login tab.
- 9 kudos
- 129 Views
- 2 replies
- 1 kudos
Databricks SQL Agent is failing
Goal :I want to build one Databricks SQL Agent using that any Databricks users can interact with a specified schema in Unity Catalog and generate insights on their data.Followed the url as reference :https://docs.databricks.com/en/large-language-mode...
- 129 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Following up on @Walter_C insights, I encountered the same issue and wanted to share the complete steps I took to fully resolve the issue:1) Upgrade SQLAlchemy: Running the following command before any other code resolved the sqlalchemy.types error: ...
- 1 kudos
- 1230 Views
- 1 replies
- 1 kudos
Unity Catalog
I am new to the Databricks platform, and I am following the LLM course on youtube after being captivated by the excellent conceptual explanations and a longstanding desire to use this platform.Running the first cell in the module -%run ../Includes/Cl...
- 1230 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
I too encountering same problem in Machine Learning coourse
- 1 kudos
Latest from our Blog
đź’ˇ ML Training Tip Of The Week #1: Optimizing GPU Utilization in Multi-Node Spark Clusters
When running distributed training or batch inference on multi-node GPU clusters with Spark, the GPUs on the Driver node often remain underutilized, resulting in unnecessary waste of GPU resources. The...
Data Validation: The Last Mile of Migration
Introduction Enterprise Data Warehouse (EDW) Migration use cases are always complex in nature. Data architects and Data engineers spend the majority of time in analyzing the source data, identifying t...
MLOps Gym - Getting started with version control
In the halcyon days of data science’s youth, version control was an oft-overlooked aspect of the work of data science teams, a “nice-to-have” that was perhaps the domain of hobbyists and enthusiasts. ...
Re-ingesting Data with Auto Loader
So imagine, you are a Data Engineer using Databricks Auto Loader to ingest data from cloud storage, when you suddenly realize that recently ingested data has some problem and you’d like to simply dele...
