- 424 Views
- 2 replies
- 0 kudos
Community Edition? More Like Community Abandonment - Thanks for NOTHING, Databricks!
To the Databricks Team (or whoever is pretending to care),Let me get this straight. You offer a "Community Edition" to supposedly help people learn, right? Well, congratulations, you've created the most frustrating, useless signup process I've ever s...
- 424 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @sachin_kanchan! I understand the frustration, and I appreciate you sharing your experience. The Community Edition is meant to provide a smooth experience, and this shouldn’t be happening. We usually ask users to drop an email to help@databrick...
- 0 kudos
- 787 Views
- 1 replies
- 1 kudos
Resolved! Access ADLS with serverless. CONFIG_NOT_AVAILABLE error
I have my own Autoloader repo and this repo is responsible for ingestion data from landing layer(ADLS) and load data into raw layer in Databricks. In that repo, I created a couple of workflows, and run these workflows with serverless cluster. and I u...
- 787 Views
- 1 replies
- 1 kudos
- 1 kudos
The recommended approach for accessing cloud storage is to create Databricks storage credentials. These storage credentials can refer to entra service principals, managed identities, etc. After a credential is created, create an external location. Wh...
- 1 kudos
- 482 Views
- 2 replies
- 1 kudos
How to make the write operation faster for writing a spark dataframe to a delta table
So, I am doing 4 spatial join operation on the files with the following sizes:Base_road_file which is 1gigabyteTelematics file which is 1.2 gigsstate boundary file , BH road file, client_geofence file and kpmg_geofence_file which are not too large My...
- 482 Views
- 2 replies
- 1 kudos
- 1 kudos
We recommend using spatial frameworks to speed up things like spatial joins, point-in-polygon, etc, like databricks mosaic or apache sedona. Without these frameworks, many of these operations result in unoptimized and explosive crossjoins.
- 1 kudos
- 478 Views
- 2 replies
- 1 kudos
Resolved! Autoloader streaming table - how to determine if new rows were updated from query?
If I'm running a scheduled batch Autoloader query which read from csv files on S3 and incrementally loads a delta table, how can I determine if new rows were added? I'm currently trying to do this from the streaming query.lastProgress as follows. s...
- 478 Views
- 2 replies
- 1 kudos
- 716 Views
- 0 replies
- 0 kudos
Native geometry Parquet support
Hi there!With the recent GeoParquet 2.0 announcements, I'm curious to understand how this impacts storing geospatial data in Databricks and Delta. For reference:the Parquet specification officially adopting geospatial guidance allowing native storage...
- 716 Views
- 0 replies
- 0 kudos
- 763 Views
- 0 replies
- 0 kudos
(java.util.NoSuchElementException) key not found: date_of_birth#1554
I have a sql function which calls another function. For some reason when I run the notebook that calls that sql function stand alone, it works fine. When I run that notebook as part of a Workflow jobs it fails every time with that error. What is caus...
- 763 Views
- 0 replies
- 0 kudos
- 3834 Views
- 2 replies
- 0 kudos
- 3834 Views
- 2 replies
- 0 kudos
- 0 kudos
Here's the difference a View and Table in the context of a Delta Live Table PIpelineViews are similar to a temporary view in SQL and are an alias for some computation. A view allows you to break a complicated query into smaller or easier-to-understan...
- 0 kudos
- 976 Views
- 3 replies
- 0 kudos
Resolved! Serverless Compute no support for Caching data frames
Can anyone please tell me why df.cache() and df.persist() are not supported in Serevrless compute?Many Thanks
- 976 Views
- 3 replies
- 0 kudos
- 0 kudos
What I do wish was possible was for serverless to warn that caching is not supported, but not error on a call. It makes switching between compute (serverless & all purpose) brittle and prevents code from easily being interoperable, no matter the comp...
- 0 kudos
- 251 Views
- 1 replies
- 0 kudos
Resolved! Using initcap function in materialized view fails
This query works: select order_date, initcap(customer_name), count(*) AS number_of_ordersfrom ... The initcap does as advertised and capitalizes the customer_name column. However, if I wrap the same exact select in a create materialized view I get an...
- 251 Views
- 1 replies
- 0 kudos
- 0 kudos
NOTE: I got it to work by aliasing the customer_name column, it's documented here: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-create-materialized-view#limitationsHowever, it wasn't clear that "Non-column reference expre...
- 0 kudos
- 1830 Views
- 3 replies
- 0 kudos
Re-use jobs as tasks with the same cluster.
Hello,I am facing an issue with my workflow.I have a job (name it main job) that, among others, runs 5 concurrent tasks, which are defined as jobs (not notebooks).Each of these jobs is identical to the others (name them sub-job-1), with the only diff...
- 1830 Views
- 3 replies
- 0 kudos
- 3801 Views
- 3 replies
- 13 kudos
Using foreachBatch within Delta Live Tables framework
Hey there!​I was wondering if there's any way of declaring a delta live table where we use foreachBatch to process the output of a streaming query.​Here's a simplification of my code:​def join_data(df_1, df_2): df_joined = ( df_1 ...
- 3801 Views
- 3 replies
- 13 kudos
- 13 kudos
foreachBatch support in DLT is coming soon, and you now have the ability to write to non-DLT sinks as well
- 13 kudos
- 1153 Views
- 2 replies
- 1 kudos
Resolved! updates on Bring Your Own Lineage (BYOL)?
One of the most exciting things in recent roadmap discussions was the idea of BYOL, so we could import external lineage into Unity Catalog and make it really useful for understanding where our data flows. We're planning some investments for the next ...
- 1153 Views
- 2 replies
- 1 kudos
- 919 Views
- 1 replies
- 0 kudos
LEGACY_ERROR_TEMP_DELTA_0007 A schema mismatch detected when writing to the Delta table.
Need help to resolve the issue Error : com.databricks.sql.transaction.tahoe.DeltaAnalysisException: [_LEGACY_ERROR_TEMP_DELTA_0007] A schema mismatch detected when writing to the Delta table.I am using the below code and my JSON is dynamically changi...
- 919 Views
- 1 replies
- 0 kudos
- 0 kudos
For datasets with constantly changing schemas, we recommend using the Variant type.
- 0 kudos
- 694 Views
- 0 replies
- 0 kudos
Not able to export maps in dashboards
When we export a dashboard with maps, the map background doesn't show up in the pdf.
- 694 Views
- 0 replies
- 0 kudos
- 328 Views
- 1 replies
- 0 kudos
Inconsistant handling of null structs vs strucs with all null values.
Summary:We have a weird behavior with structs that we have been trying (unsuccessfully) to track down. We have a struct column in a silver table that should only have data for 1 in every 500 records. It's normally null. But for about 1 in every 50 r...
- 328 Views
- 1 replies
- 0 kudos
- 0 kudos
Here are some strategies for debugging this: Before you perform each merge, write your source dataframe out as a table, and include the target table's version in the table's nameIf possible, enable the change data feed on your table so as to see chan...
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
