04-03-2023 10:52 PM
For example, consider this table. This is just an example of the understanding of what we would like to do.
user_id phone_number status is_deleted
123456 9123474387 ACTIVE 0
123456 9123474388 VERIFYING 0
123456 9123474388 VERIFYING 0
I would like to UPSERT rows based on the primary key (user_id + phone_number) using the data given below
user_id phone_number status is_deleted
123456 9123474387 ACTIVE 0
123456 9123474388 ACTIVE 0
123456 9123474388 VERIFY_FAILED 0
04-09-2023 08:28 AM
@Harikrishnan G :
To create a live pipeline from an in-house Golang backend service to Databricks tables, you can use the Databricks API to read and write data in real-time. The API provides endpoints for various operations such as creating tables, inserting data, updating data, and deleting data.
Here's an example of how you can use the Databricks API to insert data into a table in real-time:
To perform UPSERT/MERGE operations, you can use the MERGE command in SQL. Here's an example of how you can use the MERGE command to UPSERT data based on the primary key:
MERGE INTO table_name t
USING (
VALUES
(123456, 9123474387, 'ACTIVE', 0),
(123456, 9123474388, 'ACTIVE', 0),
(123456, 9123474388, 'VERIFY_FAILED', 0)
) s (user_id, phone_number, status, is_deleted)
ON t.user_id = s.user_id AND t.phone_number = s.phone_number
WHEN MATCHED THEN
UPDATE SET t.status = s.status, t.is_deleted = s.is_deleted
WHEN NOT MATCHED THEN
INSERT (user_id, phone_number, status, is_deleted) VALUES (s.user_id, s.phone_number, s.status, s.is_deleted)
This SQL statement will merge the data from the "s" table into the "t" table based on the primary key (user_id + phone_number), performing an UPSERT operation.
Regarding Gorm, looks like there is no implementation of a Databricks driver for Gorm. However, you can use the Databricks API to interact with the tables directly from your Golang code. Alternatively, you can use a Golang SQL library such as "database/sql" to execute SQL statements directly against the Databricks tables.
04-12-2023 12:56 AM
Hi @Harikrishnan G
Thank you for posting your question in our community! We are happy to assist you.
To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?
This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.