11-15-2022 05:40 AM
Hello Experts,
Is there any API in databricks that allows to write the data in the Databricks tables. I would like to send small size Logs information to Databricks tables from other service. What are my options?
Thank you very much.
11-15-2022 06:55 AM
Hi @Viren Devi ,
I guess one option of writing data into the Databricks tables or Delta Tables to be specific is to use for example Delta Tables writer: https://delta-io.github.io/delta-rs/python/usage.html#writing-delta-tables
I am not sure how this will work with the Databricks Unity Catalog managing access to the data, TBH.
Depends on your requirements. You can write the data into the landing location, then load it into Delta Tables every hour/day.
Thanks,
Pat.
11-18-2022 12:20 AM
thank you Pat. I will check this and test. Appreciate the support.
12-19-2022 08:23 AM
Hello,
I think I was not exactly clear in my requirement. I would like to write into Delta tables from outside the databricks i.e other cloud services. I believe write API works when we are writing the data when in the databricks instance itself?
11-17-2022 07:48 AM
and what about use the jdbc/odbc driver, either programatically or using a tool like dbeaver?
11-18-2022 12:24 AM
we do have a custom driver to pull the data out but we would need to change it for writing it in Databricks. Will try HTTP API first and if unsuccessful go with driver approach. Thanks for the nice suggestion.
12-16-2022 11:09 AM
Hi @Viren Devi, It would mean a lot if you could select the "Best Answer" to help others find the correct answer faster.
This makes that answer appear right after the question, so it's easier to find within a thread.
It also helps us mark the question as answered so we can have more eyes helping others with unanswered questions.
Can I count on you?
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.