cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 
DAIS 2025 Virtual Learning Festival: 11 June - 02 July 2025

Celebrate Data and AI Summit with us at the Virtual Learning Festival! Mark your calendars: June 11 - July 2, 2025! In honor of our Annual Data and AI Summit, we’re offering an exciting opportunity to upskill in data engineering, data analysis, m...

  • 43328 Views
  • 81 replies
  • 27 kudos
05-15-2025
Welcome to the Greece User Group!

We're excited to announce the launch of the Greece User Group—a dedicated space for data enthusiasts, engineers, scientists, and professionals from across Greece to connect, collaborate, and grow together! What is a User Group? User Groups are region...

  • 93 Views
  • 1 replies
  • 0 kudos
Tuesday
DAIS 2025 Day 1 Highlights

What a buzzing Day 1 at the Databricks Community Booth near the Expo Hall! We were thrilled to welcome 500+ visitors who dropped by to say hi, explore, and get involved! Here's what went down: Attendees joined fun community activities and walked awa...

  • 707 Views
  • 2 replies
  • 4 kudos
a week ago
What an incredible Day 2 at DAIS 2025!

The energy just keeps building, and Day 2 was no exception. We welcomed 600+ attendees to the Databricks Community Booth near the Expo Hall — bringing curiosity, insights, and enthusiasm for all things data. Here’s what made Day 2 memorable: The mor...

  • 402 Views
  • 0 replies
  • 3 kudos
Thursday

Community Activity

shrutikatyal
by > Visitor
  • 7 Views
  • 0 replies
  • 0 kudos

commit time is coming as null in autoloader

As per the databricks new feature in autoloader that we can use archival and move feature in autoloader however I am trying to use that feature using databricks 16.4.x.scala2.12 however commit time is still coming null as its mentioned in the documen...

  • 7 Views
  • 0 replies
  • 0 kudos
pooja_bhumandla
by > New Contributor
  • 41 Views
  • 3 replies
  • 0 kudos

Auto tuning of file size

Why maxFileSize and minFileSize are different from targetFileSize after optimization? What is the significance of targetFileSize? "numRemovedFiles": "2099","numRemovedBytes": "29658974681","p25FileSize": "29701688","numDeletionVectorsRemoved": "0","m...

  • 41 Views
  • 3 replies
  • 0 kudos
Latest Reply
loui_wentzel
New Contributor III
  • 0 kudos

there could be several different reasons, but mainly, it's because grouping arbitrary data into some target file-size is well... arbitrary.Imagine I gave you a large container of sand and some emtpy buckets, and asked you to move the sand from the co...

  • 0 kudos
2 More Replies
Alex79
by > New Contributor
  • 39 Views
  • 1 replies
  • 0 kudos

Get Job Run output through Rest API call

I have a simple notebook reading a dataframe as input and returning another dataframe, which is as follows:from pyspark.sql import SparkSessionimport pandas as pd, jsonspark = SparkSession.builder \    .appName("Pandas to Spark DataFrame Conversion")...

  • 39 Views
  • 1 replies
  • 0 kudos
Latest Reply
Vidhi_Khaitan
Databricks Employee
  • 0 kudos

Hi team,{"error_code": "INVALID_PARAMETER_VALUE","message": "Retrieving the output of runs with multiple tasks is not supported..."}means the job you're triggering (job_id = 'my_job_id') is a multi-task job (even if it has only one task). In such cas...

  • 0 kudos
Rishabh_TiwariD
by Databricks Employee
  • 93 Views
  • 1 replies
  • 0 kudos

Welcome to the Greece User Group!

We're excited to announce the launch of the Greece User Group—a dedicated space for data enthusiasts, engineers, scientists, and professionals from across Greece to connect, collaborate, and grow together! What is a User Group? User Groups are region...

Screenshot 2025-06-17 at 10.01.01 PM.png
  • 93 Views
  • 1 replies
  • 0 kudos
Latest Reply
samuelallen5
New Contributor
  • 0 kudos

Excited to see the Greece User Group coming together—such a great space for sharing ideas and building community! Looking forward to discussions that go beyond the usual, maybe even exploring unique toppings for burgers inspired by Mediterranean flav...

  • 0 kudos
Datamate
by > Visitor
  • 19 Views
  • 0 replies
  • 0 kudos

Databricks Connecting to ADLS Gen2 vs Azure SQL

What is the best approach to connect Databricks with Azure SQL or connect Databricks with ADLS Gen2.I am designing the system where I am planning to Integrate Databricks to Azure.May someone share experience Pros and cons of approach and best practic...

  • 19 Views
  • 0 replies
  • 0 kudos
SreedharVengala
by > New Contributor III
  • 27306 Views
  • 11 replies
  • 7 kudos

PGP Encryption / Decryption in Databricks

Is there a way to Decrypt / Encrypt Blob files in Databricks using Key stored in Key Vault. What libraries need to be used? Any code snippets? Links?

  • 27306 Views
  • 11 replies
  • 7 kudos
Latest Reply
Junpei_Liang
New Contributor II
  • 7 kudos

anyone has update on this?

  • 7 kudos
10 More Replies
Nasd_
by > New Contributor
  • 29 Views
  • 0 replies
  • 0 kudos

Unable to load org.apache.spark.sql.delta classes from JVM pyspark

Hello,I’m working on Databricks with a cluster running Runtime 16.4, which includes Spark 3.5.2 and Scala 2.12.For a specific need, I want to implement my own custom way of writing to Delta tables by manually managing Delta transactions from PySpark....

  • 29 Views
  • 0 replies
  • 0 kudos
Ramki
by > New Contributor
  • 25 Views
  • 1 replies
  • 0 kudos

Lakeflow clarification

Are there options to modify the streaming table after it has been created by the Lakeflow pipeline? In the use case I'm trying to solve, I need to add delta.enableIcebergCompatV2 and delta.universalFormat.enabledFormats to the target streaming table....

  • 25 Views
  • 1 replies
  • 0 kudos
Latest Reply
lingareddy_Alva
Honored Contributor II
  • 0 kudos

Hi @Ramki Yes, you can modify a streaming table created by a LakeFlow pipeline, especially when the pipeline is in triggered mode (not running continuously).In your case, you want to add the following Delta table properties: TBLPROPERTIES ( 'delta....

  • 0 kudos
Yuki
by > New Contributor III
  • 17 Views
  • 0 replies
  • 0 kudos

It's not going well to Connect to Amazon S3 with using Spark

I can't Connect to Amazon S3 well.I'm referencing and following this document: https://docs.databricks.com/gcp/en/connect/storage/amazon-s3But I can't access the S3 well.I believe the credentials are correct because I have verified that I can access ...

  • 17 Views
  • 0 replies
  • 0 kudos
michelleliu
by > New Contributor III
  • 121 Views
  • 3 replies
  • 2 kudos

Resolved! DLT Performance Issue

I've been seeing patterns in DLT process time in all my pipelines, as in attached screenshot. Each data point is an "update" that's set to "continuous". The process time keeps increasing until a point and drops back to what it's desired to be. This w...

  • 121 Views
  • 3 replies
  • 2 kudos
Latest Reply
lingareddy_Alva
Honored Contributor II
  • 2 kudos

Hi @michelleliu This sawtooth pattern in DLT processing times is actually quite common and typically indicates one of several underlying issues. Here are the most likely causes and solutions:Common Causes1. Memory Pressure & Garbage CollectionProcess...

  • 2 kudos
2 More Replies
andre_gonzalez
by > New Contributor
  • 2295 Views
  • 3 replies
  • 0 kudos

SQL wharehouse do not work with power bi online service

Whenever i try to use a SQL Wharehouse serverless cluster on a power bi dataset it does not refresh on the power bi online service. It does work normally for other types of databricks clusters. The catalog is being defined on the power query import.I...

2023-10-17_14-36.png
  • 2295 Views
  • 3 replies
  • 0 kudos
Latest Reply
ChuckyDee25
New Contributor II
  • 0 kudos

Hi,We have the exact same issue, even if we specify the catalog in the connection parameters.However, Oauth authentication through a dataflow (instead of from Power Query Desktop) works fine. In Desktop we are in version 2.122.746.0, but the issue is...

  • 0 kudos
2 More Replies
lezwon
by > New Contributor II
  • 63 Views
  • 1 replies
  • 1 kudos

Install custom wheel from dbfs in serverless enviroment

Hey folks,I have a job that runs on a serverless compute. I have also created a wheel file with custom functions, which I require in this job. I see that from here, we cannot install libraries for a task and must use notebook-scoped libraries. So wha...

  • 63 Views
  • 1 replies
  • 1 kudos
Latest Reply
loui_wentzel
New Contributor III
  • 1 kudos

Is your dbfs mounted?Otherwise, try uploading it to your workspace's "shared" folder - this is a common place to put these sorts of files. dbfs is slowly getting phased out and not really in any best practices.

  • 1 kudos
alau131
by > Visitor
  • 52 Views
  • 2 replies
  • 2 kudos

How to dynamically have the parent notebook call on a child notebook?

Hi! I would please like help on how to dynamically call one notebook from another in Databricks and have the parent notebook get the dataframe results from the child notebook. Some background info is that I have a main python notebook and multiple SQ...

  • 52 Views
  • 2 replies
  • 2 kudos
Latest Reply
jameshughes
New Contributor
  • 2 kudos

What you are looking to do is really not the intent of notebooks and you cannot pass complex data types between notebooks. You would need to persist your data frame from the child notebook so your parent notebook could retrieve the results after the ...

  • 2 kudos
1 More Replies
Abel_Martinez
by > Contributor
  • 17567 Views
  • 10 replies
  • 10 kudos

Resolved! Why I'm getting connection timeout when connecting to MongoDB using MongoDB Connector for Spark 10.x from Databricks

I'm able to connect to MongoDB using org.mongodb.spark:mongo-spark-connector_2.12:3.0.2 and this code:df = spark.read.format("com.mongodb.spark.sql.DefaultSource").option("uri", jdbcUrl)It works well, but if I install last MongoDB Spark Connector ve...

  • 17567 Views
  • 10 replies
  • 10 kudos
Latest Reply
ravisharma1024
New Contributor II
  • 10 kudos

I was facing the same issue, now It is resolved, and thanks to @Abel_Martinez.I am using this like below code:df = spark.read.format("mongodb") \.option('spark.mongodb.read.connection.uri', "mongodb+srv://*****:*****@******/?retryWrites=true&w=majori...

  • 10 kudos
9 More Replies
ESN
by > New Contributor
  • 155 Views
  • 1 replies
  • 0 kudos

Cannot Enable Serverless Compute to Use Row Filters on Databricks Runtime 15.4 LTS Dedicated Cluster

Hi everyone,I want to apply Row Filters to restrict access to Unity Catalog tables using a dedicated cluster running Databricks Runtime 15.4 LTS (Spark 3.5.0, Scala 2.12).I followed the official documentation which says that row filters are supported...

  • 155 Views
  • 1 replies
  • 0 kudos
Latest Reply
loui_wentzel
New Contributor III
  • 0 kudos

Are you the account admin on the workspace?I don't think you can bypass it, so we need to find why you can't see it.

  • 0 kudos
Welcome to the Databricks Community!

Once you are logged in, you will be ready to post content, ask questions, participate in discussions, earn badges and more.

Spend a few minutes exploring Get Started Resources, Learning Paths, Certifications, and Platform Discussions.

Connect with peers through User Groups and stay updated by subscribing to Events. We are excited to see you engage!

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Read Databricks Data Intelligence Platform reviews on G2

Latest from our Blog