- 110 Views
- 4 replies
- 4 kudos
Resolved! Least Common Type is different in Serverless and All Purpose Cluster.
The following statement gives different outputs in different computes.In Databricks, 15.4 LTS%sqlSELECT typeof(coalesce(5, '6'));-- OutputstringIn Serverless, environment version 4%sqlSELECT typeof(coalesce(5, '6'));-- OutputbigintThere are other cas...
- 110 Views
- 4 replies
- 4 kudos
- 4 kudos
@Dhruv-22 Regarding your 1st question, I'm not sureYou can refer to https://docs.databricks.com/aws/en/sql/language-manual/parameters/ansi_mode#system-default to understand what happens when ansi mode is disabled
- 4 kudos
- 80 Views
- 2 replies
- 3 kudos
Regarding : How to use Row_number() in dlt pipelines
We have two streaming tables : customer_info and customer_info_history and we joined them using full join to create temp table in pyspark and now we want to eliminate the de-duped records from this temp table. Tried using row_number() but facing bel...
- 80 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello @anusha98 , You’re hitting a real limitation of Structured Streaming: non-time window functions (like row_number() over (...)) aren’t allowed on streaming DFs. You need to use agg().max() to get the “latest value per key” @dlt.table(name="temp_...
- 3 kudos
- 71 Views
- 1 replies
- 1 kudos
Lakeflow Pipelines Trying to Read accented file with spark.readStream but failure
Trying to read a accented file(French characters) but the spark.readStream function is not working and special characters turn into something strange(ex. �) spark.readStream .format("cloudfiles") .option("cloudFiles....
- 71 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @AmarKap , When Spark decodes CP1252 bytes as UTF-8/ISO-8859-1, you’ll see the replacement char like � Can you read the file as : df = (spark.readStream.format("cloudFiles").option("cloudFiles.format", "text").option("encoding", "windows-1252")...
- 1 kudos
- 1637 Views
- 1 replies
- 0 kudos
Doubt with range_join hints optimization, using INSERT INTO REPLACE WHERE
HelloIm optmizing a big notebook and have encountered many times the tip from databricks that says "Unused range join hints". Reading the documentation for reference, I have been able to supress that warning in almost all cells, but some of then rema...
- 1637 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Gustavo_AzTry to use explain to understand what's happening. https://spark.apache.org/docs/latest/sql-ref-syntax-qry-explain.html
- 0 kudos
- 2169 Views
- 1 replies
- 1 kudos
Replay stream to migrate to liquid cluster
The documentation is sparse about how to migrate a partition table to a liquid cluster as the Alter table suggested in the documentation doesnt work when its a partitioned table.The comments on this forum suggest replaying the stream. And this is wha...
- 2169 Views
- 1 replies
- 1 kudos
- 1 kudos
Greetings @EndreM , I did some digging internally and I have come up with some helpful tips/tricks to help guide you through this issue: Based on your situation, you're encountering several common challenges when migrating a partitioned table to liqu...
- 1 kudos
- 2036 Views
- 1 replies
- 1 kudos
Unable to create databricks group and add permission via terraform
I have the following terraform code to create a databricks group and add permission to a workflow: resource "databricks_group" "dbx_group" { display_name = "ENV_MONITORING_TEAM" } resource "databricks_permissions" "workflow_permission" { job_id ...
- 2036 Views
- 1 replies
- 1 kudos
- 1 kudos
Greetings @soumiknow , I did some digging internally and found something that may help: Based on the information gathered, I can now draft a comprehensive response to this Databricks Community question about the Terraform authentication issue. ## Dra...
- 1 kudos
- 189 Views
- 2 replies
- 2 kudos
Resolved! How to make FOR cycle and dynamic SQL and variables work together
I am working on a testing notebook where the table that is tested can be given as a widget. I wanted to write it in SQL. The notebook does the following steps in a cycle that should run 10 times:1. Store the starting version of a delta table in a var...
- 189 Views
- 2 replies
- 2 kudos
- 2 kudos
Thank you! I realised that the example I gave was bad. However, what I was missing is that I did not know how to set a variable in SQL scripting. Including the SET command within the sql string does not work, you have to use the EXECUTE IMMEDIATE ......
- 2 kudos
- 186 Views
- 4 replies
- 1 kudos
Resolved! Problem in VS Code Extention
Until a few days ago, I was working with Databricks Connect using the VS Code extension, and everything worked perfectly.In my .databrickscfg file, I had authentication configured like this: [name]host:token: When I ran my code, everything worked fi...
- 186 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @DatabricksEngi1 - Please ensure you have a Python Venv set up for each Python version that you use with Databricks Connect. Also, I have given step-by-step ways to debug the issue, clear the cache, etc [Read the files and instructions carefully b...
- 1 kudos
- 18800 Views
- 12 replies
- 2 kudos
Fuzzy text matching in Spark
I have a list of client provided data, a list of company names. I have to match those names with an internal database of company names. The client list can fit in memory (its about 10k elements) but the internal dataset is on hdfs and we use Spark ...
- 18800 Views
- 12 replies
- 2 kudos
- 2 kudos
+1 for rapidfuzz, I have used it in production pipelines. Better than just levenshtein function, as rapidfuzz provides a couple of other algorithms as well. I will warn you to not do what 2024 me attempted, which is use LLM to solve for this. It soun...
- 2 kudos
- 87 Views
- 1 replies
- 1 kudos
Resolved! TransformWithState is not emitting for live streams
Hi Team, For one of my custom logics i went with transformwithState processor. However it is not working for live stream inputs., I have a start date filter on my df_base and when I give start date that is not current, the processor computes df_loss ...
- 87 Views
- 1 replies
- 1 kudos
- 1 kudos
I managed to solve this. The issue was with how I handled the value state in the def init method. It was handled as a dataframe which caused the state to never materialize nor update therefore emitting nulls.I changed them to a tuple of values and th...
- 1 kudos
- 2456 Views
- 1 replies
- 0 kudos
Delta sharing view and cached data in DSFF
I've created a view with row level access based on CURRENT_RECIPIENT() function in the where clause. And I have 100s of clients as recipients that query this view.The problem is, when I modify this view CREATE OR REPLACE with a new sql code, and reci...
- 2456 Views
- 1 replies
- 0 kudos
- 0 kudos
Have you tried something like this already? Force Cache Invalidation (Recommended) -- After CREATE OR REPLACE VIEW, execute: ALTER SHARE <share_name> REMOVE TABLE <schema>.<view_name>; ALTER SHARE <share_name> ADD TABLE <schema>.<view_name>; Thi...
- 0 kudos
- 1996 Views
- 1 replies
- 1 kudos
Lakebridge code conversion | Permission issue
Hi,I’ve successfully installed the transpile module from Lakebridge and tried the tool to convert Informatica mappings into PySpark code. However, I’m encountering a PermissionError during execution. I’ve provided the relevant environment details and...
- 1996 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @abhijit007 - I see that this has been resolved in the 0.10.5 release. Can you please retest and confirm?
- 1 kudos
- 14084 Views
- 20 replies
- 4 kudos
spotify API get token - raw_input was called, but this frontend does not support input requests.
hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...
- 14084 Views
- 20 replies
- 4 kudos
- 179 Views
- 2 replies
- 1 kudos
Results from the spark application to driver
I tried to read many articles but still not clear on this:The executors complete the execution of tasks and have the results with them.1. The results(output data) from all executors is transported to driver in all cases or executors persist it if tha...
- 179 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @raghvendrarm1 , Below are the answers to your questions: Do executors always send “results” to the driver? No. Only actions that return values (e.g., collect, take, first, count) bring data back to the driver. collect explicitly “returns al...
- 1 kudos
- 267 Views
- 3 replies
- 2 kudos
Resolved! Cannot import pyspark.pipelines module
Question: What could be a cause of the following error of my code in a Databricks notebook, and how can we fix the error? I'm using latest Free Edition of Databricks that has runtime version 17.2 and PySpark version 4.0.0.Error:ImportError: cannot im...
- 267 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Saf4Databricks - Are you trying to use it from a standalone Databricks notebook? You should only use it from with Lakeflow Declarative Pipeline (LDP). The link you shared is about LDP. Here is an example where I used it.
- 2 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 467 | |
| 349 | |
| 287 |
