- 2215 Views
- 2 replies
- 1 kudos
Resolved! dropDuplicates inside foreachBatch
Hi,If I use dropDuplicates inside foreachBatch, the dropDuplicates will become stateless and no state. It just drop duplicates for the current micro batch so I don't have to specify watermark. Is this true?Thanks
- 2215 Views
- 2 replies
- 1 kudos
- 1 kudos
Yes, you're correct! When using dropDuplicates within foreachBatch, it operates only on the current micro-batch, so it removes duplicates in a stateless manner for each batch independently. Since there's no continuous state tracking across batches, y...
- 1 kudos
- 2077 Views
- 2 replies
- 0 kudos
Fetching queries submitted for review via Genie
Hi All,I wanted to know if there is a way to export the list of queries submitted for review via Genie using an API call?I know there is an API to fetch the query run history but what i need to fetch the list of reviews via Genie.I would be great if ...
- 2077 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi Subhodeep, this new system table "system.access.assistant_events" is closer to what you are looking for ? But it looks like with this new system table introduced, seems like we can expect more around this sooner (in future release).More informatio...
- 0 kudos
- 2333 Views
- 1 replies
- 0 kudos
How to use SQL Streaming tables
We have been testing the usage of Streaming Tables in our pipelines with different results depending on the streaming sourceFor Streaming Tables reading from read_files everything works as expectedFor Streaming Tables reading from read_kafka we have ...
- 2333 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi - Pls note:1) Structure streaming & Delta live table are two different options and have different syntax's.2) You cannot execute DLT code on notebook directly. It can be run as job.Pls refer belowhttps://docs.databricks.com/en/delta-live-tables/sq...
- 0 kudos
- 11331 Views
- 4 replies
- 0 kudos
Deploy lock force acquired error when deploying asset bundle using databricks cli
I'm running this command on a DevOps pipeline.databricks bundle deploy -t devI receive this error and have tried using --force-lock but it still doesn't work.Error: deploy lock force acquired by name@company.com at 2024-02-20 16:38:34.99794209 +0000 ...
- 11331 Views
- 4 replies
- 0 kudos
- 0 kudos
you can use below commanddatabricks bundle deploy -t dev --force-lock
- 0 kudos
- 1553 Views
- 1 replies
- 0 kudos
EXECUTE IMMEDIATE works with JDBC connection ???
Hello, i need help I am trying to use the EXECUTE IMMEDIATE command to perform DELETE or DROP operations on a table located on a remote SQL server (on-premises) using a JDBC connection from a notebook in the Databricks environment.While I can succes...
- 1553 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi - what error are you getting ? Do you have sufficient permission to drop / delete a table ?
- 0 kudos
- 2157 Views
- 2 replies
- 0 kudos
Inconsistent behaviour in group by and order by
Consider this minimal example:with t as (select explode(sequence(1,10,1)) as id)select (id%2) as id from tgroup by idorder by idI would expect an ambiguous column name exception, since the grouping and sorting could apply to 2 different `id` columns....
- 2157 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi,This is not an issue, pls understand order of execution of SQL queries. "Order by" clause will always refer to columns selected / displayed (as you are referring as id everywhere I guess there is a confusion).Ambiguous column name exception occurs...
- 0 kudos
- 2020 Views
- 2 replies
- 0 kudos
NullpointerException when creating a notebook widget
To reproduce, execute this line in a notebook (runtime 15.3):dbutils.widgets.multiselect("foo", None, [None])Exception raised: Py4JJavaError: An error occurred while calling o427.createMultiselectWidget. : java.lang.NullPointerException at com.databr...
- 2020 Views
- 2 replies
- 0 kudos
- 13653 Views
- 2 replies
- 1 kudos
- 13653 Views
- 2 replies
- 1 kudos
- 12977 Views
- 4 replies
- 2 kudos
Unit Testing with PyTest in Databricks - ModuleNotFoundError
Dear all,I am following the guide in this article: https://docs.databricks.com/en/notebooks/testing.htmlhowever I am unable to run pytest due to the following error: ImportError while importing test module '/Workspace/Users/deadmanhide@gmail.com/test...
- 12977 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi,After trying a lot I could able to see some success , see if this is what you all are looking for :notebook_test.py (this is python code file)from pyspark.sql import functions as Fdef sum_values(df return df.agg(F.sum("value")).first()[0]def ...
- 2 kudos
- 1986 Views
- 3 replies
- 0 kudos
Debugging difference between "task time" and execution time for SQL query
I have a pretty large SQL query that has the following stats from the query profiler:Tasks total time: 1.93sExecuting: 27sBased on the information in the query profiler this can be due to tasks waiting for available nodes.How should I approach this t...
- 1986 Views
- 3 replies
- 0 kudos
- 0 kudos
@nengen Try using EXPLAIN EXTENDED: This provides a detailed breakdown of the logical and physical plan of a query in Spark SQL.Based on the EXPLAIN EXTENDED output, here are a few things to consider:Broadcast Exchange: If the join causes data skew,...
- 0 kudos
- 5226 Views
- 2 replies
- 2 kudos
Merging data into table using temp view
I am trying to append data into table which already exists with some data in it.I need to create view by joining multiple tables which later will be used to append data in final table .I am able to alter table schema and then run query to insert data...
- 5226 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @filipniziol Thanks for your reply My issue is resolved as my fellow developer ran same commands with different name after sometime it worked successfully.FYI - I was running query in same notebook just in different cells also I was running cells ...
- 2 kudos
- 43859 Views
- 16 replies
- 15 kudos
Problems with pandas.read_parquet() and path
I am doing the "Data Engineering with Databricks V2" learning path.I cannot run "DE 4.2 - Providing Options for External Sources", as the first code cell does not run successful:%run ../Includes/Classroom-Setup-04.2Screenshot 1: Inside the setup note...
- 43859 Views
- 16 replies
- 15 kudos
- 6947 Views
- 5 replies
- 7 kudos
How to handle schema validation for Json file. Using Databricks Autoloader?
Following are the details of the requirement:1. I am using databricks notebook to read data from Kafka topic and writing into ADLS Gen2 container i.e., my landing layer.2. I am using Spark code to read data from Kafka and write into landing...
- 6947 Views
- 5 replies
- 7 kudos
- 7 kudos
just to clarify, are you reading kafka and writing into adls in json files? like for each message from kafka is 1 json file in adls ?
- 7 kudos
- 9377 Views
- 6 replies
- 0 kudos
How dynamically pass a string parameter to a Delta Live Table Pipeline when calling from Azure Data Factory using REST API
I want to pass some context information to the delta live tables pipeline when calling from Azure Data Factory. I know the body of the API call supports Full Refresh parameter but I wonder if I can add my own custom parameters and how this can be re...
- 9377 Views
- 6 replies
- 0 kudos
- 0 kudos
In case this helps anyone, I only could use the refresh_selection parameter setting it to [] by default. Then, in the notebook, I derived the custom parameter values from the refresh_selection value.
- 0 kudos
- 2020 Views
- 1 replies
- 0 kudos
Resolved! AWS Config price raised after change to Job Compute
I was looking for opportunities to decrease the cost of my Databricks ETLs and, following the documentation, I started to use Job Computes on my ETLs.Earlier, I used only all-purpose compute to do the ETLs because I needed them to run every 10minutes...
- 2020 Views
- 1 replies
- 0 kudos
- 0 kudos
If someone have this problem in the future, the solution is simple: just disable AWS Config. That's all.
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
