- 4539 Views
- 2 replies
- 0 kudos
Get failed records from Salesforce write target tool in AWS GLUE job
I am working in the AWS GLUE service, where we are trying to migrate data from S3 to salesforce using Salesforce write target tool(Using Salesforce connection). The actual process has to be, once the process is done, the salesforce provides the jobId...
- 4539 Views
- 2 replies
- 0 kudos
- 10972 Views
- 1 replies
- 1 kudos
Ingesting complex/unstructured data
Hi there,my company is reasonably new to using Databricks, and we're running our first PoCs. Some of the data we have structured/reasonably structured, so it drops into a bucket, we point a notebook at it, and all is well and DeltaThe problem is ari...
- 10972 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Toby,Managing diverse, unstructured data can be challenging. At Know2Ledge (ShareArchiver), we specialize in unstructured data management to streamline this process.To handle your scenario efficiently:1️⃣Pre-Process Before Ingestion – Use AI-power...
- 1 kudos
- 3045 Views
- 2 replies
- 0 kudos
Community Edition? More Like Community Abandonment - Thanks for NOTHING, Databricks!
To the Databricks Team (or whoever is pretending to care),Let me get this straight. You offer a "Community Edition" to supposedly help people learn, right? Well, congratulations, you've created the most frustrating, useless signup process I've ever s...
- 3045 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @sachin_kanchan! I understand the frustration, and I appreciate you sharing your experience. The Community Edition is meant to provide a smooth experience, and this shouldn’t be happening. We usually ask users to drop an email to help@databrick...
- 0 kudos
- 5786 Views
- 1 replies
- 1 kudos
Resolved! Access ADLS with serverless. CONFIG_NOT_AVAILABLE error
I have my own Autoloader repo and this repo is responsible for ingestion data from landing layer(ADLS) and load data into raw layer in Databricks. In that repo, I created a couple of workflows, and run these workflows with serverless cluster. and I u...
- 5786 Views
- 1 replies
- 1 kudos
- 1 kudos
The recommended approach for accessing cloud storage is to create Databricks storage credentials. These storage credentials can refer to entra service principals, managed identities, etc. After a credential is created, create an external location. Wh...
- 1 kudos
- 2275 Views
- 2 replies
- 1 kudos
How to make the write operation faster for writing a spark dataframe to a delta table
So, I am doing 4 spatial join operation on the files with the following sizes:Base_road_file which is 1gigabyteTelematics file which is 1.2 gigsstate boundary file , BH road file, client_geofence file and kpmg_geofence_file which are not too large My...
- 2275 Views
- 2 replies
- 1 kudos
- 1 kudos
We recommend using spatial frameworks to speed up things like spatial joins, point-in-polygon, etc, like databricks mosaic or apache sedona. Without these frameworks, many of these operations result in unoptimized and explosive crossjoins.
- 1 kudos
- 2008 Views
- 2 replies
- 1 kudos
Resolved! Autoloader streaming table - how to determine if new rows were updated from query?
If I'm running a scheduled batch Autoloader query which read from csv files on S3 and incrementally loads a delta table, how can I determine if new rows were added? I'm currently trying to do this from the streaming query.lastProgress as follows. s...
- 2008 Views
- 2 replies
- 1 kudos
- 5861 Views
- 2 replies
- 0 kudos
- 5861 Views
- 2 replies
- 0 kudos
- 0 kudos
Here's the difference a View and Table in the context of a Delta Live Table PIpelineViews are similar to a temporary view in SQL and are an alias for some computation. A view allows you to break a complicated query into smaller or easier-to-understan...
- 0 kudos
- 1015 Views
- 1 replies
- 0 kudos
Resolved! Using initcap function in materialized view fails
This query works: select order_date, initcap(customer_name), count(*) AS number_of_ordersfrom ... The initcap does as advertised and capitalizes the customer_name column. However, if I wrap the same exact select in a create materialized view I get an...
- 1015 Views
- 1 replies
- 0 kudos
- 0 kudos
NOTE: I got it to work by aliasing the customer_name column, it's documented here: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-create-materialized-view#limitationsHowever, it wasn't clear that "Non-column reference expre...
- 0 kudos
- 3834 Views
- 3 replies
- 0 kudos
Re-use jobs as tasks with the same cluster.
Hello,I am facing an issue with my workflow.I have a job (name it main job) that, among others, runs 5 concurrent tasks, which are defined as jobs (not notebooks).Each of these jobs is identical to the others (name them sub-job-1), with the only diff...
- 3834 Views
- 3 replies
- 0 kudos
- 6071 Views
- 3 replies
- 13 kudos
Using foreachBatch within Delta Live Tables framework
Hey there!I was wondering if there's any way of declaring a delta live table where we use foreachBatch to process the output of a streaming query.Here's a simplification of my code:def join_data(df_1, df_2): df_joined = ( df_1 ...
- 6071 Views
- 3 replies
- 13 kudos
- 13 kudos
foreachBatch support in DLT is coming soon, and you now have the ability to write to non-DLT sinks as well
- 13 kudos
- 4789 Views
- 1 replies
- 0 kudos
LEGACY_ERROR_TEMP_DELTA_0007 A schema mismatch detected when writing to the Delta table.
Need help to resolve the issue Error : com.databricks.sql.transaction.tahoe.DeltaAnalysisException: [_LEGACY_ERROR_TEMP_DELTA_0007] A schema mismatch detected when writing to the Delta table.I am using the below code and my JSON is dynamically changi...
- 4789 Views
- 1 replies
- 0 kudos
- 0 kudos
For datasets with constantly changing schemas, we recommend using the Variant type.
- 0 kudos
- 2188 Views
- 1 replies
- 0 kudos
Inconsistant handling of null structs vs strucs with all null values.
Summary:We have a weird behavior with structs that we have been trying (unsuccessfully) to track down. We have a struct column in a silver table that should only have data for 1 in every 500 records. It's normally null. But for about 1 in every 50 r...
- 2188 Views
- 1 replies
- 0 kudos
- 0 kudos
Here are some strategies for debugging this: Before you perform each merge, write your source dataframe out as a table, and include the target table's version in the table's nameIf possible, enable the change data feed on your table so as to see chan...
- 0 kudos
- 2270 Views
- 1 replies
- 0 kudos
Install Python libraries on Databricks job cluster
Hello,I am trying to install some wheel file and requirements.txt file from my Unity Catalog Volumes on my Databricks job cluster using an init script but the results are very inconsistent.Does anyone have ever faced that ?What's wrong with my approa...
- 2270 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @sachamourier, Could you please clarify what is the inconsistency? are some packages missing or the incorrect library was loaded?
- 0 kudos
- 814 Views
- 1 replies
- 0 kudos
Running a notebook as 'Run all below' when sheduled via Azure DataFactory
We have a notebook with a lot of subsequent cells that can run independent from each other. When we execute the notebook manually via 'Run all', the runs stops when an error is thrown. When we execute manually via 'Run all below', the run proceeds ti...
- 814 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @LeenB For example each cell execution you can build up with try except command. Example belowtry: print("Hello world") #your code of each cellexcept Exception as e: print("Issue with printing hello world")For sure it is not recommended ...
- 0 kudos
- 1907 Views
- 6 replies
- 1 kudos
Accessing Job parameters using cluster v15.4
After upgrading databricks cluster to version 15.4, is there any way to access job parameters in notebook except the following way ?dbutils.widgets.get("parameter_name")In v15.4, dbutils.notebook.entry_point.getCurrentBindings() has been discontinued...
- 1907 Views
- 6 replies
- 1 kudos
- 1 kudos
For me it is working at 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12)
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
