- 2927 Views
- 5 replies
- 2 kudos
Resolved! Is there a way to list the dlt maintenance jobs through the API?
After creating the delta pipeline, I would like to get details from the dlt maintenance job automatically created by Databricks, like the scheduled time when the dlt maintenance tasks will be executed. However, it seems the Job API 2.1 doesn't cover ...
- 2927 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Debayan Mukherjee ,Actually the Databricks Jobs API documentation has not been fixed yet. The parameter `job_type` should be included in the list endpoint request documentation. Please do this in order to avoid unnecessary questions here in the ...
- 2 kudos
- 3407 Views
- 2 replies
- 1 kudos
Clear Cache From a Notebook, not from a Cluster
Hi, I'm running all my jobs on one big cluster, I'm just concerned is there a solution on how we could clear cache resulted by a notebook in the end of the job when its done? hence it does not causing any memory problem sometime from one to another, ...
- 3407 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @krisna math We haven't heard from you since the last response from @Debayan Mukherjee , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to...
- 1 kudos
- 998 Views
- 1 replies
- 1 kudos
Azure-EventHub Schema Registry with Spark-Scala
Hi all,Is there a way to consume the schemas from schema registry defined in Azure EventHub using apache spark and scala.
- 998 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @sreeranjani thevan Great to meet you, and thanks for your question!Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 6744 Views
- 1 replies
- 2 kudos
try and except in DLT pipelines
Good Morning,I am having some issues with my DLT pipeline. I have a scenario where I am loading in bronze-silver tables programatically from a SQL database (each row corresponds to a table to create). This leaves me in a situation where sometimes onl...
- 6744 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Robert Pearce Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 2 kudos
- 1570 Views
- 1 replies
- 2 kudos
Installing private python Azure DevOps repository without revealing personal access token in pyproject.toml
I want to install a .whl file on my Databricks cluster which includes a private Azure DevOps repository as a dependency in its pyproject.toml file, i.e.:[project] name = "test" description = "test_description." version = "0.1.0" authors = [ { name ...
- 1570 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Aaron Kaijser Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 2 kudos
- 976 Views
- 1 replies
- 2 kudos
Jobs Data Pipeline Runtime Increase Significantly
Hi, I am facing an issue where one of my jobs taking so long since certain time, previously its only needs less than 1 hour to run a batch job that load json data and do a truncate and load to a delta table, but since june 2nd, it become so long that...
- 976 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @krisna math Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 2 kudos
- 1023 Views
- 1 replies
- 1 kudos
Does JDBC and ODBC connection to Databricks Cluster or SQL Warehouse encrypted?
Hello, our consumers (Dell Boomi and other apps) needs data from Databricks. Databricks provides JDBC and ODBC drivers. Does JDBC and ODBC connection to Databricks Cluster or SQL Warehouse encrypted? I am talking about the data in-transit between Dat...
- 1023 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Rajesh Paul Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 1088 Views
- 1 replies
- 1 kudos
how to execute "Build your Chat Bot with Dolly Demo" in my own VM?
I am trying to execute Build your Chat Bot with Dolly Demo using my own VM. At the first steps they are executing this command %run ./_resources/00-init $catalog=hive_metastore $db=dbdemos_llm which is -as I understand- calling another python script...
- 1088 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @alaa migdady Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 858 Views
- 1 replies
- 2 kudos
apply a function across multiple smaller dataframes created from one big dataframe in scala
The dataframe 'big_df' looks like the below| id| index| timestamp||:---- |:------:| -----:|| abc| 1| 11:00:00|| abc| 1| 11:00:10|| abc| 1| 11:00:20|| abc| 1| 11:00:30|| abc| 1| 11:00:40|| abc| 1| 11:00:50|| abc| 2| 11:01:00|| abc| 2| 11:01:10|| abc| ...
- 858 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Sandesh Puligundla Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 2 kudos
- 992 Views
- 1 replies
- 2 kudos
How to change databricks email alert notifications subject line?
Currently, our email notification subject shows for error in below format:<[AWS Account]> Error in run <RUN ID> of <Job Name>In our current databricks environment we have multiple environment jobs such as Dev/QA/UAT/Prod and it is very hard for us to...
- 992 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Sanath Pabba Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 2 kudos
- 949 Views
- 1 replies
- 1 kudos
Alternative to the inbuilt Scan method of the Databricks SDK for Golang
The implementation of Scan method in the Databricks SDK .If the default value of any column is NULL, in that case the Scan method return exception.What can we use as the alternative to this?While implementing the logic with Scan method as per the doc...
- 949 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Shivansh Bhatnagar Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 1117 Views
- 1 replies
- 0 kudos
Alternative to Scan method in Databricks SDK for Golang
We want to fetch multiple rows/records with 100+ columns(that are dynamic as per the request) but in the doc the way shown is to scan each column(through Scan method) one by one rather scanning the complete object at once which is making it very tedi...
- 1117 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Shivansh Bhatnagar Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 693 Views
- 1 replies
- 0 kudos
Is it possible to create sql statements (after return) with cte's in databricks scalar functions?
The function returns an integer as a result of the query
- 693 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Luis Lopez Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 1827 Views
- 4 replies
- 2 kudos
error: at least one column must be specified for the table.
error: at least one column must be specified for the table.
- 1827 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @anand R Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ca...
- 2 kudos
- 5335 Views
- 5 replies
- 0 kudos
Resolved! Incremental Load without Last Modified Date and Primary Key field in Azure Data Factory to create bronze data in data bricks
I am trying to do incremental load in azure data factory. Most of the tables in the Oracle database doesn't have last modified date and Primary key column. Is there any way to do incremental loading without last modified date and primary key column?
- 5335 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @Selva Kumar Ponnusamy Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please te...
- 0 kudos