cancel
Showing results for 
Search instead for 
Did you mean: 
Discussions
Engage in dynamic conversations covering diverse topics within the Databricks Community. Explore discussions on data engineering, machine learning, and more. Join the conversation and expand your knowledge base with insights from experts and peers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Browse the Community

Community Discussions

Connect with fellow community members to discuss general topics related to the Databricks platform, ...

1156 Posts

Get Started Discussions

Start your journey with Databricks by joining discussions on getting started guides, tutorials, and ...

698 Posts

Learning Discussion

Engage in vibrant discussions covering diverse learning topics within the Databricks Community. Expl...

320 Posts

Activity in Discussions

despasito
by Visitor
  • 2 Views
  • 0 replies
  • 0 kudos

Quick Extender Pro Review, User Results, Medical Facts

Quick Extender Pro is one of the best rod based products, that work effectivelly and safely. Why choose Quick Extender Pro.My customer review:1. It's lightweight and easy to hide under your clothes2. It's easy to setup and use3. It uses a DSS(Double ...

best-penis-stretcher.jpg
  • 2 Views
  • 0 replies
  • 0 kudos
MR07
by Visitor
  • 6 Views
  • 0 replies
  • 0 kudos

Optimal Cluster Selection for Continuous Delta Live Tables Pipelines: Bronze and Silver

Hi,I have two Delta Live Tables Pipelines. The first one is the Bronze pipeline, which handles bronze tables. These tables are defined as streaming tables, and this pipeline needs to be executed continuously. The second one is the Silver pipeline, wh...

  • 6 Views
  • 0 replies
  • 0 kudos
MR07
by Visitor
  • 68 Views
  • 2 replies
  • 0 kudos

Databricks Managing Materialized Views in Delta Live Tables: Selective Refresh Behavior

Hi Community,I have 200 complex SQL Queries and I can't create a Streaming tables using these queries. So, I have created as Materialized Views in Delta Live Tables using these SQL queries and the DLT pipeline should be run continuously.My question i...

  • 68 Views
  • 2 replies
  • 0 kudos
Latest Reply
steyler-db
New Contributor III
  • 0 kudos

Hello team, thanks for reaching out us, it will be a pleasure to help you on this ask. That's a great catch to run through a materialized, view and regarding the question: If any record of underlying table is inserted, updated or deleted, the only re...

  • 0 kudos
1 More Replies
OmVispute
by Visitor
  • 24 Views
  • 0 replies
  • 0 kudos

Requesting a Coupon/Voucher for Databricks Certified Data Engineer Associate exam

I am planning on taking the Databricks Certified Data Engineer Associate exam in this upcoming week, however being a student it would be a great support if I could get any coupon or voucher for the exam.

  • 24 Views
  • 0 replies
  • 0 kudos
chevichenk
by New Contributor II
  • 65 Views
  • 3 replies
  • 2 kudos

No userid, username, job when making modifications on tables

Hi, everyone!I'm in this situationI have some jobs that makes changes on a particular table. I use only one user to make this modifications, but then there's a process i can't identify that also makes changes on my table.The question is, there's a re...

chevichenk_1-1718308350095.png
Data Engineering
history
jobs
userid
username
  • 65 Views
  • 3 replies
  • 2 kudos
Latest Reply
chevichenk
New Contributor II
  • 2 kudos

Hi, @shan_chandra, @LuisRSanchez,Just found that there are some .jar that are executed and are writting on this table but this .jar is called through batchSo, we think this is the cause Thanks!Ingrid

  • 2 kudos
2 More Replies
avrm91
by New Contributor III
  • 49 Views
  • 1 replies
  • 0 kudos

How to load xlsx Files to Delta Live Tables (DLT)?

I want to load a .xlsx file to DLT but struggling as it is not available with Autoloader.With the Assistant I tried to load the .xlsx first to a data frame and then send it to DLT.  import dlt from pyspark.sql import SparkSession # Load xlsx file in...

  • 49 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@avrm91  - can try dividing xlsx files into a csv as a preprocessing step and ingest them in to a dataframe using Autoloader. Also, you can use openpyxl to load into a dataframe. refer to this doc for example.  

  • 0 kudos
JeremyH
by New Contributor II
  • 48 Views
  • 3 replies
  • 0 kudos

CREATE WIDGETS in SQL Notebook attached to SQL Warehouse Doesn't Work.

I'm able to create and use widgets using the UI in my SQL notebooks, but they get lost quite frequently when the notebook is reset.There is documentation suggesting we can create widgets in code in SQL: https://learn.microsoft.com/en-us/azure/databri...

  • 48 Views
  • 3 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

Hi @JeremyH - can you please try adding like the below in your query and see if widgets are getting populated? {{parameter_name }}

  • 0 kudos
2 More Replies
Shaimaa
by Visitor
  • 32 Views
  • 1 replies
  • 0 kudos

Running SQL queries against a parquet folder in S3

I need to run sql queries against a parquet folder in S3. I am trying to use "read_files" but sometimes my queries fail due to errors while inferring the schema and sometimes without a specified reason. Sample query:  SELECT SUM(CASE WHEN match_resu...

  • 32 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

 @Shaimaa  - you can divide the query into a nested query to first select all the fields from the s3 by enforcing the schema and build a nested query on top of the below example query (not syntax verified) SELECT * FROM STREAM read_files( 's3...

  • 0 kudos
Jackson1111
by New Contributor II
  • 44 Views
  • 1 replies
  • 0 kudos

Databricks job cluster logs

Hello, how can I enable Databricks to generate a separate spark log for each job run?What parameters should I use in spark configuration? 

  • 44 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@Jackson1111 - If you are talking about workflow jobs, you can try running using a job cluster to generate spark logs for a each of the workflow jobs.  But, If this is of Spark Jobs within the Spark UI, you wanted to separate out the logs. This is a ...

  • 0 kudos
semsim
by New Contributor III
  • 17 Views
  • 0 replies
  • 0 kudos

Init Script Failing

I am getting an error when I try to run the cluster scoped init script. The script itself is as follows:#!/bin/bashsudo apt update && sudo apt upgrade -ysudo apt install libreoffice-common libreoffice-java-common libreoffice-writer openjdk-8-jre-head...

  • 17 Views
  • 0 replies
  • 0 kudos
manish1987c
by New Contributor II
  • 43 Views
  • 1 replies
  • 0 kudos

Delta Live Table - Flow detected an update or delete to one or more rows in the source table

I have create a pipeline where i am ingesting the data from bronze to silver and using SCD 1, however when i am trying to create gold table as dlt it is giving me error as "Flow 'user_silver' has FAILED fatally. An error occurred because we detected ...

manish1987c_0-1718341166099.png manish1987c_1-1718341206991.png
  • 43 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@manish1987c -The Streaming does not handle input that is not an append. you can set skipChangeCommits to true 

  • 0 kudos
aozero
by Visitor
  • 48 Views
  • 1 replies
  • 0 kudos

Deleting data programmatically from databricks live delta tables

Hello all, I am relatively new in data engineering and working on a project requiring me to programmatically delete data from delta live tables. However, I found that simply stopping the streaming job and deleting rows from the delta tables caused th...

  • 48 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@aozero - can you please try a FULL REFRESH of the Delta live tables? https://docs.databricks.com/en/delta-live-tables/updates.html#how-delta-live-tables-updates-tables-and-views  

  • 0 kudos
thehonestreview
by New Contributor
  • 205 Views
  • 0 replies
  • 0 kudos

Rpm review

The Rapid Profit Machine (RPM) is a program designed to teach affiliate marketing with an emphasis on using solo ads for traffic generation. While it provides some introductory training on affiliate marketing, there are notable concerns, such as the ...

  • 205 Views
  • 0 replies
  • 0 kudos
kumarsuresh
by New Contributor
  • 62 Views
  • 1 replies
  • 0 kudos

Gen AI course material

Databricks updated the Generative AI course https://partner-academy.databricks.com/learn/lp/315/generative-ai-engineering-pathway but the course material is missing in the partner academy. Does anybody know where to download the course material? 

  • 62 Views
  • 1 replies
  • 0 kudos
Latest Reply
ScottSmithDB
Valued Contributor
  • 0 kudos

Hi @kumarsuresh.  Please try the link again.  I am not sure if there was a delay in the course publication for this learning path, but it appears to be available now.

  • 0 kudos
thehonestreview
by New Contributor
  • 33 Views
  • 0 replies
  • 0 kudos

how does rapid profit machine works?

  The Rapid Profit Machine (RPM) is a program that teaches affiliate marketing with a focus on using solo ads for traffic generation. While it offers some basic training on affiliate marketing, there are red flags to consider, such as the upselling o...

  • 33 Views
  • 0 replies
  • 0 kudos
Top Kudoed Authors