I'm using Azure Event Hubs Connector https://github.com/Azure/azure-event-hubs-spark to connect an Even Hub.When I install this library from Maven , then everything works, I can access lib classes using JVM:connection_string = "<connection_string>"
s...
Hi @blackcoffee AR​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
For a KPI dashboard, we need to know the exact size of the data in a catalog and also all schemas inside the catalogs. What is the best way to do this? We tried to iterate over all tables and sum the sizeInBytes using the DESCRIBE DETAIL command for ...
Hi @Anant Pingle​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
Hello,May i know when can i expect The instructor-led Advanced Data Engineering with Databricks course for Databricks Certified Data Engineer Professional will be available in 2023..or any references to prepare would be great.
@sarath endluri​ I saw an ILT training through my Partners Academy, scheduled for April 24-27. Today I just got an e-mail saying it was canceled because they needed to update the content. I asked for further info and got told it's going to be a while...
I am consuming an IoT stream with thousands of different signals using Structured Streaming. During processing of the stream, I need to know the previous timestamp and value for each signal in the micro batch. The signal stream is eventually written ...
@Suteja Kanuri​ Tried the above on streaming DFBut facing the below errorAttributeError: 'DataFrame' object has no attribute 'groupByKey'Can you please let me know DBR runtime
Support of running multiple cells at a time in databricks notebookHi all,Now databricks notebook supports parallel run of commands in a single notebook that will help run ad hoc queries simultaneously without creating a separate notebook.Once you run...
Hi @Ajay Pandey​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so w...
Hi @Rituparna Das​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so...
I know scheduling options are there, but it doesn't consider interdependency like don't execute job2 unless job1 has been executed.I know, script based, or API based triggers are also there, but I am looking for UI based triggers like how we can orch...
Hi @satyam rastogi​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...
I want to make a connection to Databricks with KNIME. To do this I am using "Create Databricks Environment" node. I have made the following configuration:1. Installed Databricks Simba JDBC driver 2. Made the necessary configuration in Create Databric...
Hi @Geethanjali Nataraj​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell...
Using autoloader, I'm reading daily data partitioned by well. The data has a specific schema, but if there's no value for a column it isn't present in the json. For a specific column on a specific table I'm getting an error like:​Cannot convert long ...
Hi @Jordan Fox​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we...
Hi everyone!I would like to know how spark stops the connection when reading from a sql database using the JDBC format.Also, if there is a way to check when the connection is active or manually stop it, I also would like to know.Thank you in advance!
Hi @João Peixoto​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...
Hi team, our application for partner onboarding was processed a year ago. The last email we received was from Procurement Operations Specialist confirming our information has been added to the system. However, we never received any advise on the next...
Hi @Siddhesh Gaikwad​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...
I have mounted my path from Databricks to AzureADLS Gen1. using SPN as service accuntuntill yesterday everything was ok, but today I see, I can view all older deleted folders. I can not see them in ADLS . but my databricks dbutils.fs.ls() shows them....
Hi @pankaj bhatt​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...
Hi @Rituparna Das​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so...
I want to define a column with null values in my dataframe using pyspark. This column will later be used for other calculations.What is the difference between creating it in these two different ways?df.withColumn("New_Column", lit(None))df.withColumn...
Hi @Sara Corral​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback ...
Dear DB community!As I know, when the resource creator is out of the project all resources they create will be deleted as well. So the question is that: can I assign the owner role to a group, that will help protect the resource from deletion or not...
Hi @trung nguyen​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...
