- 4340 Views
- 3 replies
- 1 kudos
windowing/Join function on streaming data failing in DLT
i have a DLT pipeline, wanted to update a perticular column value from other records, trying it using windowing first_value geting below error.approach#1: joinI tried selfjoin initially, failing with joins are not possible in DLT with below error.org...
- 4340 Views
- 3 replies
- 1 kudos
- 1 kudos
hi @jhonm_839 thanks for your response to the post. I appreciate your time.I have watermark on dataframe in palce and used last(), min() etc.. aggregate functions its still filing.final_df = df.withWatermark("kafka_ts", "1 hour").withColumn("s_featur...
- 1 kudos
- 2264 Views
- 3 replies
- 0 kudos
How are .whl files executed for Python wheel tasks?
Hello,We package a Poetry managed project into a .whl and run it as a Python wheel task. Naturally, many of the dependencies referenced by the .whl file are already present on the Databricks cluster. Is this detected by the task setup (in its virtual...
- 2264 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi David,I can't speak exactly to how Poetry handles the dependency resolution of libraries that are already installed, or how that interacts with the Databricks runtime. However, I can offer you some advice on how my team handles this situtation.It'...
- 0 kudos
- 6035 Views
- 6 replies
- 1 kudos
Variant type table within DLT
Hi,I have a table with Variant type (preview) and works well in 15.3, when i try to run a code that reference this Variant type in a DLT pipeline i get : com.databricks.sql.transaction.tahoe.DeltaUnsupportedTableFeatureException: [DELTA_UNSUPPORTED_F...
- 6035 Views
- 6 replies
- 1 kudos
- 1 kudos
I can indeed confirm that adding some additional table properties to the @Dlt attribute in the DLT pipeline definition resolved the earlier issues. Thanks for pointing this out.
- 1 kudos
- 4863 Views
- 1 replies
- 1 kudos
Resolved! How To Save a File as a Pickle Object to the Databricks File System
I tried running this code:```def save_file(name, obj with open(name, 'wb') as pickle.dump(obj, f)``` One file was saved in the local file system, but the second was too large and so I need to save in the dbfs file system. Unfortunately, I d...
- 4863 Views
- 1 replies
- 1 kudos
- 1 kudos
To save a Python object to the Databricks File System (DBFS), you can use the dbutils.fs module to write files to DBFS. Since you are dealing with a Python object and not a DataFrame, you can use the pickle module to serialize the object and then wri...
- 1 kudos
- 13877 Views
- 9 replies
- 2 kudos
Asset bundle doesn't sync files to workspace
I've created a completely fresh project with a completely empty workspaceLocally I have the databricks CLI version 0.230.0 installedI rundatabricks bundle init default-pythonI have auth set up with a PAT generated by an account which has workspace ad...
- 13877 Views
- 9 replies
- 2 kudos
- 2 kudos
Ok, I feel silly. Despite reading the other messages in this thread, I didn't twig to the fact that I had in fact added the subfolder I had created the DAB in to my top-level project .gitignore since I was just playing around and didn't want to comm...
- 2 kudos
- 5175 Views
- 4 replies
- 0 kudos
Reading excel file using pandas on spark api not rendering #N/A values correctly
I am trying to read a .xlsx file using ps.read_excel() and having #N/A as a value for string type columns. But in the dataframe, i am getting "null" inplace of #N/A . Is there any option , using which we can read #N/A as a string in .xlsx file
- 5175 Views
- 4 replies
- 0 kudos
- 0 kudos
Did you get a solution or workaround for this error, as I am also facing the same even after using dtype = str, na_filter= False, keep_default_na = False ?
- 0 kudos
- 26973 Views
- 18 replies
- 10 kudos
Using init scripts on UC enabled shared access mode clusters
I know that UC enabled shared access mode clusters do not allow init script usage and I have tried multiple workarounds to use the required init script in the cluster(pyodbc-install.sh, in my case) including installing the pyodbc package as a workspa...
- 26973 Views
- 18 replies
- 10 kudos
- 10 kudos
Hello all,Below workaround was efficient to me1) pyodbc-install.sh is uploaded in a Volume 2) the shared cluster is able to navigate to the Volume to select the init script3) the Databricks runtime is 15.4 LTS4) the Allowlist has been updated to allo...
- 10 kudos
- 9026 Views
- 8 replies
- 0 kudos
UC, pyodbc, Shared Cluster, and :Can't open lib 'ODBC Driver 17 for SQL Server' : file not found
Hey Databricks! Trying to use the pyodbc init script in a Volume in UC on a shared compute cluster but receive error: "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)"). I fo...
- 9026 Views
- 8 replies
- 0 kudos
- 0 kudos
Hello all,Below workaround was efficient to me1) pyodbc-install.sh is uploaded in a Volume 2) the shared cluster is able to navigate to the Volume to select the init script3) the Databricks runtime is 15.4 LTS4) the Allowlist has been updated to allo...
- 0 kudos
- 4840 Views
- 2 replies
- 0 kudos
Get failed records from Salesforce write target tool in AWS GLUE job
I am working in the AWS GLUE service, where we are trying to migrate data from S3 to salesforce using Salesforce write target tool(Using Salesforce connection). The actual process has to be, once the process is done, the salesforce provides the jobId...
- 4840 Views
- 2 replies
- 0 kudos
- 3357 Views
- 2 replies
- 0 kudos
Community Edition? More Like Community Abandonment - Thanks for NOTHING, Databricks!
To the Databricks Team (or whoever is pretending to care),Let me get this straight. You offer a "Community Edition" to supposedly help people learn, right? Well, congratulations, you've created the most frustrating, useless signup process I've ever s...
- 3357 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @sachin_kanchan! I understand the frustration, and I appreciate you sharing your experience. The Community Edition is meant to provide a smooth experience, and this shouldn’t be happening. We usually ask users to drop an email to help@databrick...
- 0 kudos
- 6745 Views
- 1 replies
- 1 kudos
Resolved! Access ADLS with serverless. CONFIG_NOT_AVAILABLE error
I have my own Autoloader repo and this repo is responsible for ingestion data from landing layer(ADLS) and load data into raw layer in Databricks. In that repo, I created a couple of workflows, and run these workflows with serverless cluster. and I u...
- 6745 Views
- 1 replies
- 1 kudos
- 1 kudos
The recommended approach for accessing cloud storage is to create Databricks storage credentials. These storage credentials can refer to entra service principals, managed identities, etc. After a credential is created, create an external location. Wh...
- 1 kudos
- 2618 Views
- 2 replies
- 1 kudos
How to make the write operation faster for writing a spark dataframe to a delta table
So, I am doing 4 spatial join operation on the files with the following sizes:Base_road_file which is 1gigabyteTelematics file which is 1.2 gigsstate boundary file , BH road file, client_geofence file and kpmg_geofence_file which are not too large My...
- 2618 Views
- 2 replies
- 1 kudos
- 1 kudos
We recommend using spatial frameworks to speed up things like spatial joins, point-in-polygon, etc, like databricks mosaic or apache sedona. Without these frameworks, many of these operations result in unoptimized and explosive crossjoins.
- 1 kudos
- 2334 Views
- 2 replies
- 1 kudos
Resolved! Autoloader streaming table - how to determine if new rows were updated from query?
If I'm running a scheduled batch Autoloader query which read from csv files on S3 and incrementally loads a delta table, how can I determine if new rows were added? I'm currently trying to do this from the streaming query.lastProgress as follows. s...
- 2334 Views
- 2 replies
- 1 kudos
- 6268 Views
- 2 replies
- 0 kudos
- 6268 Views
- 2 replies
- 0 kudos
- 0 kudos
Here's the difference a View and Table in the context of a Delta Live Table PIpelineViews are similar to a temporary view in SQL and are an alias for some computation. A view allows you to break a complicated query into smaller or easier-to-understan...
- 0 kudos
- 1367 Views
- 1 replies
- 0 kudos
Resolved! Using initcap function in materialized view fails
This query works: select order_date, initcap(customer_name), count(*) AS number_of_ordersfrom ... The initcap does as advertised and capitalizes the customer_name column. However, if I wrap the same exact select in a create materialized view I get an...
- 1367 Views
- 1 replies
- 0 kudos
- 0 kudos
NOTE: I got it to work by aliasing the customer_name column, it's documented here: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-create-materialized-view#limitationsHowever, it wasn't clear that "Non-column reference expre...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
