cancel
Showing results for 
Search instead for 
Did you mean: 
Page Title

Welcome to the Databricks Community

Discover the latest insights, collaborate with peers, get help from experts and make meaningful connections

107015members
57348posts
cancel
Showing results for 
Search instead for 
Did you mean: 
Earn swag at the User Community Booth at Data & AI Summit 2024!

Be sure to visit the User Community booth in person during the summit to earn swag and an exclusive Summit badge. The Data & AI Summit is happening now! June 10-13 at the Moscone Center in San Francisco. Benefits of Joining the Databricks Community A...

  • 1727 Views
  • 6 replies
  • 6 kudos
a week ago
Get Certified at Data & AI Summit and Earn this Exclusive Databricks Jacket

Agenda at a glance Join us for four days of deep learning around data, AI and LLM technologies. Monday 9:00 AM - 1:00 PM Training and Certification Dive deep into specific topics like data lakehouse architecture, Databricks SQL, MLflow, LLMs and mor...

  • 2824 Views
  • 8 replies
  • 11 kudos
3 weeks ago
Databricks Test Drives - Get Help

Databricks Test Drives! Created exclusively for Data + AI Summit attendees, Databricks Test Drives is your chance to get hands-on with the latest AI-powered features of the Data Intelligence Platform.  With Test Drives, you can: Easily compare DBRX, ...

  • 938 Views
  • 3 replies
  • 6 kudos
a week ago
Databricks Learning Festival (Virtual): 10 July - 24 July 2024

Join the Databricks Learning Festival (Virtual)! Mark your calendars from 10 July to 24 July 2024! Upskill today across data engineering, data analysis, machine learning, and generative AI. Join the thousands who have elevated their career with D...

  • 5433 Views
  • 6 replies
  • 2 kudos
2 weeks ago

Community Activity

manish1987c
by New Contributor III
  • 47 Views
  • 2 replies
  • 0 kudos

Delta Live Table - Flow detected an update or delete to one or more rows in the source table

I have create a pipeline where i am ingesting the data from bronze to silver and using SCD 1, however when i am trying to create gold table as dlt it is giving me error as "Flow 'user_silver' has FAILED fatally. An error occurred because we detected ...

manish1987c_0-1718341166099.png manish1987c_1-1718341206991.png
  • 47 Views
  • 2 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@manish1987c -The Streaming does not handle input that is not an append. you can set skipChangeCommits to true 

  • 0 kudos
1 More Replies
despasito
by Visitor
  • 5 Views
  • 0 replies
  • 0 kudos

Quick Extender Pro Review, User Results, Medical Facts

Quick Extender Pro is one of the best rod based products, that work effectivelly and safely. Why choose Quick Extender Pro.My customer review:1. It's lightweight and easy to hide under your clothes2. It's easy to setup and use3. It uses a DSS(Double ...

best-penis-stretcher.jpg
  • 5 Views
  • 0 replies
  • 0 kudos
MR07
by Visitor
  • 19 Views
  • 0 replies
  • 0 kudos

Optimal Cluster Selection for Continuous Delta Live Tables Pipelines: Bronze and Silver

Hi,I have two Delta Live Tables Pipelines. The first one is the Bronze pipeline, which handles bronze tables. These tables are defined as streaming tables, and this pipeline needs to be executed continuously. The second one is the Silver pipeline, wh...

  • 19 Views
  • 0 replies
  • 0 kudos
MR07
by Visitor
  • 69 Views
  • 2 replies
  • 0 kudos

Databricks Managing Materialized Views in Delta Live Tables: Selective Refresh Behavior

Hi Community,I have 200 complex SQL Queries and I can't create a Streaming tables using these queries. So, I have created as Materialized Views in Delta Live Tables using these SQL queries and the DLT pipeline should be run continuously.My question i...

  • 69 Views
  • 2 replies
  • 0 kudos
Latest Reply
steyler-db
New Contributor III
  • 0 kudos

Hello team, thanks for reaching out us, it will be a pleasure to help you on this ask. That's a great catch to run through a materialized, view and regarding the question: If any record of underlying table is inserted, updated or deleted, the only re...

  • 0 kudos
1 More Replies
OmVispute
by Visitor
  • 24 Views
  • 0 replies
  • 0 kudos

Requesting a Coupon/Voucher for Databricks Certified Data Engineer Associate exam

I am planning on taking the Databricks Certified Data Engineer Associate exam in this upcoming week, however being a student it would be a great support if I could get any coupon or voucher for the exam.

  • 24 Views
  • 0 replies
  • 0 kudos
chevichenk
by New Contributor II
  • 65 Views
  • 3 replies
  • 2 kudos

No userid, username, job when making modifications on tables

Hi, everyone!I'm in this situationI have some jobs that makes changes on a particular table. I use only one user to make this modifications, but then there's a process i can't identify that also makes changes on my table.The question is, there's a re...

chevichenk_1-1718308350095.png
Data Engineering
history
jobs
userid
username
  • 65 Views
  • 3 replies
  • 2 kudos
Latest Reply
chevichenk
New Contributor II
  • 2 kudos

Hi, @shan_chandra, @LuisRSanchez,Just found that there are some .jar that are executed and are writting on this table but this .jar is called through batchSo, we think this is the cause Thanks!Ingrid

  • 2 kudos
2 More Replies
avrm91
by New Contributor III
  • 50 Views
  • 1 replies
  • 0 kudos

How to load xlsx Files to Delta Live Tables (DLT)?

I want to load a .xlsx file to DLT but struggling as it is not available with Autoloader.With the Assistant I tried to load the .xlsx first to a data frame and then send it to DLT.  import dlt from pyspark.sql import SparkSession # Load xlsx file in...

  • 50 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@avrm91  - can try dividing xlsx files into a csv as a preprocessing step and ingest them in to a dataframe using Autoloader. Also, you can use openpyxl to load into a dataframe. refer to this doc for example.  

  • 0 kudos
JeremyH
by New Contributor II
  • 48 Views
  • 3 replies
  • 0 kudos

CREATE WIDGETS in SQL Notebook attached to SQL Warehouse Doesn't Work.

I'm able to create and use widgets using the UI in my SQL notebooks, but they get lost quite frequently when the notebook is reset.There is documentation suggesting we can create widgets in code in SQL: https://learn.microsoft.com/en-us/azure/databri...

  • 48 Views
  • 3 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

Hi @JeremyH - can you please try adding like the below in your query and see if widgets are getting populated? {{parameter_name }}

  • 0 kudos
2 More Replies
Shaimaa
by Visitor
  • 32 Views
  • 1 replies
  • 0 kudos

Running SQL queries against a parquet folder in S3

I need to run sql queries against a parquet folder in S3. I am trying to use "read_files" but sometimes my queries fail due to errors while inferring the schema and sometimes without a specified reason. Sample query:  SELECT SUM(CASE WHEN match_resu...

  • 32 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

 @Shaimaa  - you can divide the query into a nested query to first select all the fields from the s3 by enforcing the schema and build a nested query on top of the below example query (not syntax verified) SELECT * FROM STREAM read_files( 's3...

  • 0 kudos
Jackson1111
by New Contributor II
  • 44 Views
  • 1 replies
  • 0 kudos

Databricks job cluster logs

Hello, how can I enable Databricks to generate a separate spark log for each job run?What parameters should I use in spark configuration? 

  • 44 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@Jackson1111 - If you are talking about workflow jobs, you can try running using a job cluster to generate spark logs for a each of the workflow jobs.  But, If this is of Spark Jobs within the Spark UI, you wanted to separate out the logs. This is a ...

  • 0 kudos
semsim
by New Contributor III
  • 17 Views
  • 0 replies
  • 0 kudos

Init Script Failing

I am getting an error when I try to run the cluster scoped init script. The script itself is as follows:#!/bin/bashsudo apt update && sudo apt upgrade -ysudo apt install libreoffice-common libreoffice-java-common libreoffice-writer openjdk-8-jre-head...

  • 17 Views
  • 0 replies
  • 0 kudos
aozero
by Visitor
  • 48 Views
  • 1 replies
  • 0 kudos

Deleting data programmatically from databricks live delta tables

Hello all, I am relatively new in data engineering and working on a project requiring me to programmatically delete data from delta live tables. However, I found that simply stopping the streaming job and deleting rows from the delta tables caused th...

  • 48 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@aozero - can you please try a FULL REFRESH of the Delta live tables? https://docs.databricks.com/en/delta-live-tables/updates.html#how-delta-live-tables-updates-tables-and-views  

  • 0 kudos
thehonestreview
by New Contributor
  • 252 Views
  • 0 replies
  • 0 kudos

Rpm review

The Rapid Profit Machine (RPM) is a program designed to teach affiliate marketing with an emphasis on using solo ads for traffic generation. While it provides some introductory training on affiliate marketing, there are notable concerns, such as the ...

  • 252 Views
  • 0 replies
  • 0 kudos
kumarsuresh
by New Contributor
  • 63 Views
  • 1 replies
  • 0 kudos

Gen AI course material

Databricks updated the Generative AI course https://partner-academy.databricks.com/learn/lp/315/generative-ai-engineering-pathway but the course material is missing in the partner academy. Does anybody know where to download the course material? 

  • 63 Views
  • 1 replies
  • 0 kudos
Latest Reply
ScottSmithDB
Valued Contributor
  • 0 kudos

Hi @kumarsuresh.  Please try the link again.  I am not sure if there was a delay in the course publication for this learning path, but it appears to be available now.

  • 0 kudos
thehonestreview
by New Contributor
  • 33 Views
  • 0 replies
  • 0 kudos

how does rapid profit machine works?

  The Rapid Profit Machine (RPM) is a program that teaches affiliate marketing with a focus on using solo ads for traffic generation. While it offers some basic training on affiliate marketing, there are red flags to consider, such as the upselling o...

  • 33 Views
  • 0 replies
  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Top Kudoed Authors

Latest from our Blog

MLOps Gym - Evaluating Large Language Models with MLflow

This is the second part of a three-part guide on MLflow in the MLOps Gym series. In Part 1, “Beginners’ Guide to MLflow”, we covered Tracking and Model Registry components. In this article, we will f...

173Views 0kudos

MLOps Gym - 13 Essential Tips for Writing Clean Code

As a data scientist developing ML models in Python on Databricks, you likely utilize notebooks for conducting training experiments.  The ML code you jot down in your notebooks might end up cluttered ...

6127Views 2kudos

How to use Databricks Autoloader across AWS accounts

IntroductionCreate S3 bucket and cross-account instance profileUse Autoloader to create SNS-SQS across accountsManually create SNS-SQS for cross-account AutoloaderTest cross-account Autoloader connect...

5975Views 3kudos