- 2959 Views
- 3 replies
- 1 kudos
CICD for Jobs @ WorkFlows
I had created Jobs to trigger the respective notebooks in Databricks Workflow.Now I need to move them to further environments.Would you please help me with an CICD process to promote jobs to further environments.
- 2959 Views
- 3 replies
- 1 kudos
- 1 kudos
Please use jobs API 2.1 You can get job and save JSON with that jobs to git.In git then set variables defining databricks workspaces (URL and token) and after push define that API call is triggered with your json stored in git.
- 1 kudos
- 8522 Views
- 1 replies
- 2 kudos
Resolved! Using widget values in a shell script cell
I have a Databricks notebook containing a mix of SQL, Python, and shell script cells. I know I can retrieve and use values of widgets in Python cells using dbutils.widgets.get('key') and in SQL cells using ${key}.How can I use widget values in shell ...
- 8522 Views
- 1 replies
- 2 kudos
- 2 kudos
For those interested, I found and am for now using this workaround:https://stackoverflow.com/questions/54662605/how-to-pass-a-python-variables-to-shell-script-in-azure-databricks-notebookbleswhile I wait for a more direct method.
- 2 kudos
- 28284 Views
- 6 replies
- 15 kudos
Job cluster vs All purpose cluster
Environment: AzureI've a workflow that takes approximately a minute to execute and I want to run the job every 2 minutes.. All purpose cluster:On attaching all purpose cluster to the job, it takes approx. 60 seconds to execute.Using job cluster:On at...
- 28284 Views
- 6 replies
- 15 kudos
- 6647 Views
- 3 replies
- 1 kudos
How to change the audit log delivery Service Account?
Hi Team,I'm trying to set up Audit log delivery based on the documentation "https://docs.gcp.databricks.com/administration-guide/account-settings-gcp/log-delivery.html". As per the document, I've created a multi-region storage bucket however I'm not ...
- 6647 Views
- 3 replies
- 1 kudos
- 17181 Views
- 3 replies
- 4 kudos
Resolved! What are differences between Materialized view and Streaming table in delta live table?
Hi,I was wondering that what are differences between Materialized view and Streaming table? which one should I use when I extract data from bronze table to silver table since I found that both CREATE LIVE TABLE and CREATE STREAMING LIVE TABLE could a...
- 17181 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Mike Chen​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback wi...
- 4 kudos
- 6564 Views
- 2 replies
- 4 kudos
Resolved! Can you use %run or dbutils.notebook.run in a Delta Live Table pipeline?
Hi there, Can you use a %run or dbutils.notebook.run() in a Delta Live Table (DLT) pipeline?When I try, I get the following error: "IllegalArgumentException: requirement failed: To enable notebook workflows, please upgrade your Databricks subscriptio...
- 6564 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi all.@Kaniz Fatma​ thanks for your answer. I am on the premium pricing tier in Azure.After digging around the logs it would seem that you cannot run magic commands in a Delta Live Table pipeline. Therefore, you cannot use %run in a DLT pipeline - w...
- 4 kudos
- 7327 Views
- 6 replies
- 4 kudos
Resolved! Getting a key mismatch error with Delta Live Tables.
I am attempting to create a streaming delta live table. The main issue I am experiencing is the error below.com.databricks.sql.cloudfiles.errors.CloudFilesIllegalStateException: Found mismatched event: keyI have an aws appflow that is creating a fold...
- 7327 Views
- 6 replies
- 4 kudos
- 4 kudos
Hi, I am also facing the same issue, but I found nothing on the documentation to fix it.
- 4 kudos
- 9498 Views
- 3 replies
- 0 kudos
How to extract the start and end time of the command line cell of the notebook using REST API in Azure Databricks?
HiI have a notebook with many command line cells in it.I want to extract the execution time of each cell using Databricks REST API? How can I do that?Please note - I managed to get the Start & End time of the Job using REST API (/2.1/jobs/runs/get) f...
- 9498 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Milind Keer​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 0 kudos
- 3106 Views
- 3 replies
- 0 kudos
data is not written back to data lake
I have this strange case where data is not written back to data lake. I have 3 container- . Bronze, Silver and Gold. I have done the mounting and have not problem to read the source data and write it Bronze layer ( using hive meta store catalog). T...
- 3106 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Givi Salu​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...
- 0 kudos
- 8462 Views
- 4 replies
- 6 kudos
Resolved! Error When Starting the Cluster
I am having this error when running my cluster, any idea why?
- 8462 Views
- 4 replies
- 6 kudos
- 6 kudos
@Werner Stinckens​ , I checked again, you cannot change them after your workspace is deployed. The only way right now is to recreate the workspace and migrate. It’s not possible to update CIDR range right now without migration.
- 6 kudos
- 3268 Views
- 2 replies
- 0 kudos
- 3268 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Deepak Kini​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 0 kudos
- 2160 Views
- 0 replies
- 4 kudos
Databricks marketplace is here. With Databricks Marketplace, you can easily browse and discover data that suits your needs and get instant access with...
Databricks marketplace is here. With Databricks Marketplace, you can easily browse and discover data that suits your needs and get instant access with Delta Sharing. Once you find a dataset you like, you can add it to your Databricks catalog.
- 2160 Views
- 0 replies
- 4 kudos
- 2664 Views
- 2 replies
- 3 kudos
loading incremental data
I want to load incremental data to the delta live table, I wrote function to load data for 10 tables, every time that I run the pipe line, some tables are empty and have a schema, and when I run again, the other tables are empty and the previous tabl...
- 2664 Views
- 2 replies
- 3 kudos
- 3 kudos
@zahra Jalilpour​ How the DLT tables and views are updated depends on the update type:Refresh all: All live tables are updated to reflect the current state of their input data sources. For all streaming tables, new rows are appended to the table.Full...
- 3 kudos
- 11060 Views
- 2 replies
- 2 kudos
Deltalkake vs Delta table
Can somebody give me good definition of delta lake vs delta table? What are the use cases of each, similarities and differences? Sorry I’m new to databricks ans trying to learn.
- 11060 Views
- 2 replies
- 2 kudos
- 2 kudos
Delta Lake and Delta table are related concepts in the Apache Delta Lake project. which extends Apache Spark with ACID (Atomicity, Consistency, Isolation, Durability) capabilities for data lakes. Delta Lake provides a storage layer that enables trans...
- 2 kudos
- 3636 Views
- 1 replies
- 3 kudos
Display Command Not showing the Result, Rather giving the Dataframe Schema
Display Command Not showing the Result, Rather giving the Dataframe Schema
- 3636 Views
- 1 replies
- 3 kudos
- 3 kudos
hey ,can you try you sql query with this methodselect * from (your sql query )
- 3 kudos
| User | Count |
|---|---|
| 1626 | |
| 790 | |
| 506 | |
| 349 | |
| 287 |
