- 1348 Views
- 1 replies
- 0 kudos
how to view row produced from rest API in databricks for long running queries in Running state
print(f"Query ID: {query['query_id']} ,Duration: {query['duration']} ms,user :{query['user_display_name']},Query_execute :{query['query_text']},Query_status : {query['status']},rw:{query['rows_produced']}"") U am able to get rows_produced only for...
- 1348 Views
- 1 replies
- 0 kudos
- 0 kudos
https://{databricks_instance}.cloud.databricks.com/api/2.0/sql/history/queries?include_metrics=true
- 0 kudos
- 1187 Views
- 1 replies
- 0 kudos
How to stop subsequent iterations in Databricks loop feature?
How to stop subsequent iterations in Databricks loop feature? sys.exit(0) or dbutils.notebook.exit() only marks the current task and series of tasks in sequence as failed, but continues with subsequent iterations.
- 1187 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @surajitDECurrently, there is no out of the box feature to achieve that. What you can do, is to try to implement notebook logic that in case of error will cancel for each task run using REST API or python sdk:- use /api/2.1/jobs/runs/cancel endpoi...
- 0 kudos
- 2238 Views
- 1 replies
- 0 kudos
Resolved! Redacted check constraint condition in Delta Table
Hello! I have a delta table with a check constraint - it's one of many that a config-driven ETL pipeline of mine generates. When someone edits the config file and deploys the change, I'd like for the check constraint to be updated as well if it's dif...
- 2238 Views
- 1 replies
- 0 kudos
- 0 kudos
Figured this out with the help of @SamDataWalk 's post https://community.databricks.com/t5/data-engineering/databricks-bug-with-show-tblproperties-redacted-azure-databricks/m-p/93546It happens because Databricks thinks certain keywords in the constra...
- 0 kudos
- 6281 Views
- 5 replies
- 2 kudos
Resolved! Databricks bug with show tblproperties - redacted - Azure databricks
I am struggling to report what is a fairly fundamental bug. Can anyone help? Ideally someone from Databricks themselves. Or others who can confirm they can replicate it.There is a bug where databricks seems to be hiding “any” properties which have th...
- 6281 Views
- 5 replies
- 2 kudos
- 2 kudos
Like your example that redaction behavior seemed to pick up on the column name: a condition that included a column named "URL" was redacted, but one that included a "modifiedDateTime" was not
- 2 kudos
- 2068 Views
- 1 replies
- 1 kudos
Is it possible to have an alert when a row is added to a table?
I currently have a table that periodically adds rows in (sometimes daily, sometimes over a month). I was hoping to have an alert for when a row is added into this. The table has date fields of when rows are loaded in.I have an alert working, but it j...
- 2068 Views
- 1 replies
- 1 kudos
- 1 kudos
You could set your alert with query as: -- Custom alert to monitor for new rows added to a table SELECT COUNT(*) AS new_rows FROM your_table WHERE event_time > current_timestamp() - interval '1' hour In this example, the query checks for rows added t...
- 1 kudos
- 4409 Views
- 2 replies
- 2 kudos
How to enforce a cleanup policy on job cluster logs
We have a number of jobs on our databricks workspaces. All job clusters are configured with a dbfs location to save the respective logs (configured from Job cluster -> "Advanced options" -> "Logging").However, the logs are retained in the dbfs indefi...
- 4409 Views
- 2 replies
- 2 kudos
- 2 kudos
@EDDatabricks Thanks for reaching out to us. I think you should explore our Purge option is enabled or not. https://docs.databricks.com/en/administration-guide/workspace/settings/storage.html
- 2 kudos
- 3251 Views
- 2 replies
- 2 kudos
Unity Catalog
Can we set up unity catalog on databricks community edition?If yes, please share the process.Thanks
- 3251 Views
- 2 replies
- 2 kudos
- 2 kudos
Another option you can try it to setup open source version of unity catalog with apache spark if you don't have possibility to create Azure trial account/
- 2 kudos
- 5023 Views
- 3 replies
- 2 kudos
Resolved! Asset Bundles Job If Condition
HiI am trying to use Asset Bundles to deploy a job with email notifications active only on production environment.I tried to use such if statement but it does not seem to do anything.resources: jobs: Master_Load: name: Master Load e...
- 5023 Views
- 3 replies
- 2 kudos
- 2 kudos
Okay i made it work with a variable, it was pretty easy. The variable accepted multiline text easily.Thanks!databricks.ymltargets: dev: mode: development ... variables: notifications: default: - user1@smth.com ...
- 2 kudos
- 4880 Views
- 3 replies
- 0 kudos
What does DLT INITIALIZING phase do?
In Delta Live Tables, the INITIALIZING phase takes sometimes a minute, sometimes 5 minutes. I'd like to learn what is it doing in the background, and can it be optimized in any way.
- 4880 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @antr, In Delta Live Tables (DLT), a feature of Databricks, the "Initializing" phase refers to the first step in the lifecycle of a DLT pipeline run. During this phase, the platform sets up the necessary resources, configurations, and dependencies...
- 0 kudos
- 5079 Views
- 1 replies
- 0 kudos
Databricks as a "pure" data streaming software like Confluent
DearsI was wondering if anyone has leveraged Databricks as a "pure" data streaming software in place of Confluent, Flink, Kafka etc.I see the reference architectures placing Databricks on the data processing side mostly once data is made available by...
- 5079 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @noorbasha534 ,It depends on what you're asking for. Kafka is primarily a messaging system, optimized for handling high-throughput, distributed message logs. Databricks can read from Kafka as a data source but doesn't replace Kafka's role in messa...
- 0 kudos
- 1566 Views
- 1 replies
- 0 kudos
how to get node and executor id and log
Hi Team,We have df has 70 M row count and we are calling an API for 6000 rows per set using df.repartition(rep_count).foreachpartition(func_name) so in func_name we are calling API post req for that partition but when we are trying to print/log somet...
- 1566 Views
- 1 replies
- 0 kudos
- 0 kudos
any update on this???
- 0 kudos
- 1217 Views
- 1 replies
- 1 kudos
Streaming Live Table - What is actually computed?
Can anyone please share in a DLT or structured streaming task, what group of rows are computed?Specific scenarios:1. when a streaming table A joining a delta table B. Is each of the minibatches in A joining the whole delta table? Does Spark compute t...
- 1217 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @tliuzillow ,1. Stream-static Join: Each minibatch from the streaming table (A) is joined with the entire Delta table (B). 2. Stream-stream Join: Each minibatch from the streaming table(A) is joined with minibatch from the streaming table(B). Howe...
- 1 kudos
- 3901 Views
- 4 replies
- 2 kudos
Resolved! Network bottleneck
Within a script, I noticed that the network connection between driver and the mounted network drives is often a huge bottleneck. It seems that the network through speed is unreasonable low for being an Azure Single node: Standard_DS12_v2 · DBR: 14.3....
- 3901 Views
- 4 replies
- 2 kudos
- 2 kudos
You are right. I am creating 200 small files with the size of roughly 6 MB (in the quality system) and a few 100000s files in production. The partition is motivated by the original business need and further processing. Let me test with a the differen...
- 2 kudos
- 3494 Views
- 1 replies
- 2 kudos
Resolved! Is there a way to use name prefixes for asset bundle deployments to prevent collisions
Hi, we are using asset bundles to deploy artefacts in Databricks. We use a service account and github workflows for that. Currently, we deploy in dev mode whenever a PR is created against the main branch of our gh repo. So far so good. However, we ar...
- 3494 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Mathias_Peters ,Yes, it is possible to do so.1. Configure variable that will be added to the path.variables: feature_name: description: This will be added to the path2. Configure workspace root_path:workspace: root_path: /Shared/.bundle/...
- 2 kudos
- 2047 Views
- 1 replies
- 0 kudos
post /api/2.0/sql/statements/sql statement id/cancel forbidden error in databricks
when i tried executing this requestpost /api/2.0/sql/statements/${SQL_STATEMENT_ID}/cancelI am getting forbidden error.how do we get access to execute this
- 2047 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @slakshmanan ,The 403 Forbidden error usually means that the API request is not authorized, meaning you likely do not have the proper permissions or your authentication credentials are not being recognized. Here are 2 main reasons:1. Missing or In...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
