cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Srajole
by New Contributor
  • 1654 Views
  • 2 replies
  • 2 kudos

Data load issue

I have a job in Databricks which completed successfully but the data is not been written into the target table, I have checked all the possible ways, each n every thing is correct in the code, target table name, source table name, etc etc. It is a Fu...

  • 1654 Views
  • 2 replies
  • 2 kudos
Latest Reply
cgrant
Databricks Employee
  • 2 kudos

This looks like a misconfigured Query Watchdog, specifically the below config: spark.conf.get("spark.databricks.queryWatchdog.outputRatioThreshold") Please check the value of this config - it is 1000 by default. Also, we recommend using Jobs Comput...

  • 2 kudos
1 More Replies
jano
by New Contributor III
  • 333 Views
  • 1 replies
  • 1 kudos

Delta UniForm

When we save a delta table using the UniForm option we are seeing a 50% drop in table size. When we add UniForm to a delta table in post we are seeing no change in data size. Is this expected or are others seeing this as well? 

Get Started Discussions
Data Size
delta
UniForm
  • 333 Views
  • 1 replies
  • 1 kudos
Latest Reply
BigRoux
Databricks Employee
  • 1 kudos

Re:When we save a delta table using the UniForm option we are seeing a 50% drop in table size What format are you starting with?  e.g. csv -> Delta.   

  • 1 kudos
ChristianRRL
by Valued Contributor III
  • 449 Views
  • 1 replies
  • 2 kudos

Resolved! AutoLoader Pros/Cons When Extracting Data (Cross-Post)

Cross-posting from: https://community.databricks.com/t5/data-engineering/autoloader-pros-cons-when-extracting-data/td-p/127400Hi there, I am interested in using AutoLoader, but I'd like to get a bit of clarity if it makes sense in my case. Based on e...

  • 449 Views
  • 1 replies
  • 2 kudos
Latest Reply
BS_THE_ANALYST
Esteemed Contributor
  • 2 kudos

You’ve already identified data duplication as a potential con of landing the data first, but there are several benefits to this approach that might not be immediately obvious:Schema Inference and Evolution: AutoLoader can automatically infer the sche...

  • 2 kudos
FedeRaimondi
by Contributor
  • 421 Views
  • 3 replies
  • 2 kudos

Resolved! Python module import with Dedicated access mode

I currently have a repo connected in databricks and I was able to correctly import a python module from src folder located in the same root.Since I am using a Machine Learning runtime, I am force to choose a Dedicated (formerly: Single user) access m...

  • 421 Views
  • 3 replies
  • 2 kudos
Latest Reply
FedeRaimondi
Contributor
  • 2 kudos

Thanks @szymon_dybczak ! I confirm that's a permission issue and assigning "CAN MANAGE" solves it.I still find it not really intuitive, since the goal is to use a shared cluster (with ML runtime) for development purposes. I mean, it would make sense ...

  • 2 kudos
2 More Replies
JaydeepKhatri
by New Contributor II
  • 423 Views
  • 1 replies
  • 1 kudos

Using merge Schema with spark.read.csv for inconsistent schemas

The Problem:A common data engineering challenge is reading a directory of CSV files where the schemas are inconsistent. For example, some files might have columns in a different order, or be missing certain columns altogether. The standard behavior o...

  • 423 Views
  • 1 replies
  • 1 kudos
Latest Reply
bogomol312
New Contributor II
  • 1 kudos

Very nice example  and explanation helped me a lot, thanks !

  • 1 kudos
Boban12335
by New Contributor
  • 214 Views
  • 1 replies
  • 0 kudos

Unity Catalog tool function with custom parameters not being used

I have created a UC tool that takes in a few custom STRING parameters. I gave this tool to an ai agent using the mosaic ai agent framework with hardcoded parameter values for testing. The issue is my ai agent hallucinates and injects its own ai gener...

  • 214 Views
  • 1 replies
  • 0 kudos
Latest Reply
Nivethan_Venkat
Contributor III
  • 0 kudos

Hi @Boban12335,Can we get UC function definition to understand your problem better?Best Regards,Nivethan V

  • 0 kudos
ChristianRRL
by Valued Contributor III
  • 378 Views
  • 3 replies
  • 3 kudos

Resolved! AutoLoader - Write To Console (Notebook Cell) Long Running Issue

Hi there,I am likely misunderstanding how to use AutoLoader properly while developing/testing. I am trying to write a simple AutoLoader notebook cell to *read* the contents of a path with json files, and *write* them to console (i.e. notebook cell) i...

ChristianRRL_0-1754403001614.png
  • 378 Views
  • 3 replies
  • 3 kudos
Latest Reply
SP_6721
Contributor III
  • 3 kudos

Hi @ChristianRRL ,It looks like spark.readStream with Auto Loader creates a continuous streaming job by default, which means it keeps running while waiting for new files.To avoid this, you can control the behaviour using trigger(availableNow=True), w...

  • 3 kudos
2 More Replies
Lucas_N
by New Contributor II
  • 2897 Views
  • 2 replies
  • 3 kudos

Resolved! Documentation for spatial SQL public preview - Where is it?

Hi everybody,since DBR 17.1 spatial sql functions (st_point(), st_distancesphere, ... ) are in public preview.The functionality is presented in this talk Geospatial Insights With Databricks SQL: Techniques and Applications or discussed here in the fo...

  • 2897 Views
  • 2 replies
  • 3 kudos
Latest Reply
Geospatial_Gwen
New Contributor III
  • 3 kudos

Is this what you were after?https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-st-geospatial-functions

  • 3 kudos
1 More Replies
Danish1105
by New Contributor II
  • 269 Views
  • 1 replies
  • 1 kudos

Resolved! Run_type has some null

Just wondering — we know that the run_type column in the job run timeline usually has only three values: JOB_RUN, SUBMIT_RUN, and WORKFLOW_RUN. So why do we also see a null value there? Any reason?  

Danish1105_0-1754303528409.jpeg
  • 269 Views
  • 1 replies
  • 1 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 1 kudos

Hi @Danish1105 ,One possible explanation is that you see null values because of the following reason they stated in documentation:"Not populated for rows emitted before late August 2024."In case of my workspace, this seems valid. I have only nulls wh...

  • 1 kudos
ivan7256
by New Contributor II
  • 1173 Views
  • 3 replies
  • 2 kudos

PERMISSION_DENIED: Cannot access Spark Connect. when trying to run serverless databricks connect

I am not able to run a file as "run as workflow" nor "run with databricks connect" when I choose serverless run on my paid account.  However I can perform this action in my free edition account . See error : pyspark.errors.exceptions.connect.SparkCon...

  • 1173 Views
  • 3 replies
  • 2 kudos
Latest Reply
SP_6721
Contributor III
  • 2 kudos

Hi @ivan7256 ,This might be because serverless compute isn't enabled for workflows in your paid workspace.

  • 2 kudos
2 More Replies
devdbk
by New Contributor II
  • 645 Views
  • 3 replies
  • 5 kudos

Databricks Free Edition Needs Transparency About Data Access

When I first discovered the Databricks Free Edition, I thought it was a generous offering for data enthusiasts, researchers, and developers who just needed a personal sandbox. No cost. Easy setup. Promises of productivity. But what caught me off guar...

  • 645 Views
  • 3 replies
  • 5 kudos
Latest Reply
devdbk
New Contributor II
  • 5 kudos

Thanks again for all the perspectives shared so far. I want to re-emphasize that the Databricks Free Edition offers real value. For data enthusiasts, learners, and builders, it’s a genuinely powerful environment to get hands-on without jumping throug...

  • 5 kudos
2 More Replies
florianb
by New Contributor III
  • 4650 Views
  • 3 replies
  • 8 kudos

Resolved! Rss feeds for databricks releases

Hi,are there any rss feeds for the databricks platform, sql & runtime releases? We have a big tech stack so it is sometimes hard to keep up with the ever changing technologies. We are using rss feeds to keep up with all of that.Cant find anything for...

  • 4650 Views
  • 3 replies
  • 8 kudos
Latest Reply
kerem
Contributor
  • 8 kudos

Databricks recently published an RSS feed for all their updates. As far as I can find, it is only for AWS at the moment.https://docs.databricks.com/aws/en/feed.xml 

  • 8 kudos
2 More Replies
mano7438
by New Contributor III
  • 81343 Views
  • 7 replies
  • 7 kudos

Resolved! How to create temporary table in databricks

Hi Team,I have a requirement where I need to create temporary table not temporary view.Can you tell me how to create temporary table in data bricks ?

  • 81343 Views
  • 7 replies
  • 7 kudos
Latest Reply
NandiniN
Databricks Employee
  • 7 kudos

I see, thanks for sharing, can you mark the solution which worked for you @abueno as Accepted.

  • 7 kudos
6 More Replies
holunder
by New Contributor
  • 472 Views
  • 1 replies
  • 0 kudos

CLI: Export-dir provides LatestClone

Hi everyone,I want to download the current databricks codebase out of a workspace and tried viadatabricks databricks workspace export-dir /Sandbox/foo .Surprisingly, some of the subfolders are twice in the export target: One with the expected name (`...

  • 472 Views
  • 1 replies
  • 0 kudos
Latest Reply
SP_6721
Contributor III
  • 0 kudos

Hi @holunder ,This could be because the backend stores both the original and cloned versions of folders, even if only one appears in the web UI. The Databricks CLI exports everything from the backend, not just what's visible in the UI.

  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels