The schedule button isn't saving my schedule information for a databricks sql query. After I hit save and open the schedule again it has reverted to 'Never'. The query itself according to the past executions pane is not running according to the sched...
Hi @Wally Plourde​ Does @Rohit Rajendran​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
Hey there @IssacClose​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
I know that I can get a list of all of the table names in a given 'database' by using (if the 'database' was named "scratch"): show tables from scratchHow do I get a list just like that, but that only lists the tables that I created?
Hi @Barb Krienke​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
Hi Team,My python dataframe is as below.The raw data is quite a long series of approx 5000 numbers. My requirement is to go through each row in RawData column and calculate 2 metrics. I have created a function in Python and it works absolutely fine. ...
Hello @Kausthub NP​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
Hi @Hariharan Sambath​, We haven't heard from you on the last response from @Debayan Mukherjee​ and @Hubert Dudek​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as...
Hi Team,Can anyone please help me figure out how to configure Delta Live Tables Cluster accessing AWS S3 bucket without Instance profile defined in Cluster's JSON?The idea is, the user who is running the DLT Cluster has Storage Credentials and Extern...
Hi @Jason Nam​ , DLT and unity catalog are not integrated yet. The cluster-notebook setup uses UC and can access S3 but not the DLT jobs. Please check the limitations in this document (7th point):https://docs.databricks.com/release-notes/unity-catalo...
Hi,I would like to capture notebook custom log exceptions(python) from ADF pipeline based on the exceptions pipeline should got succeed or failed.Is there any mechanism to implement it. In my testing ADF pipeline is successful irrespective of the log...
Hi SailajaB,Try this out.Notebook, once executed successfully return a long JSON formatted output. We need to specify appropriate nodes to fetch the output. In below screenshot we can see that when notebook ran it returns empName & empCity as output....
What was the established architectural pattern for doing streaming ETL with Delta Lake before DLT was a thing? And incidentally, what approach would you take in the context of delta-oss today? The pipeline definitions would not have had to be declara...
Hi @Veli-Jussi Raitila​ Does @Shanmugavel Chandrakasu​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
Hello community;I am using Databricks on Azure and I am curious to know the implications of changing the IP address of the underlying VM's from dynamic to static - that is, the IP addresses that the clusters get, I would just like them to always have...
Hi @Jonathan Stern​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
Unable to login to https://community.cloud.databricks.com/login.htmlYesterday only I registered, able to login and create cluster.Today unable to login with same credential. I do not receive any email also while trying Password reset.
Hi @SHESADEV SAMAL​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
Failed to launch pipeline cluster 0802-171503-4m02lexd: The operation could not be performed on your account with the following error message: azure_error_code: OperationNotAllowed, azure_error_message: Operation could not be completed as it results ...
Hi there @Aaron LeBato​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you...
Heyooooo!I'm using Azure Databricks and sparklyr to do some geospatial analysis.Before I actually work with Spark Dataframes, I've been using the R packagesstarsandsfto do some preprocessing on my data so that it's easier to interact with later.In or...
Yay! It's September! On September 22nd we are hosting another Community Social - we're doing these monthly ! We want to make sure that we all have the chance to connect as a community often. Come network, talk data, and just get social! Join us for o...
One of my clients has been orchestration Databricks notebooks using Airflow + REST API. They're curious about the pros/cons of switching these jobs to Databricks jobs with Task Orchestration.I know there are all sorts of considerations - for example,...
@Kaniz Fatma​ Hello Kaniz, I'm currently working with a major Enterprise Client looking to make the choice between the Airflow vs Databricks for Jobs scheduling. Our Entire code base is in Databricks and we are trying to figure out the complexities t...
