- 17259 Views
- 7 replies
- 2 kudos
Resolved! How do I clear all output results in a notebook?
I'm building notebooks for tutorial sessions and I want to clear all the output results from the notebook before distributing it to the participants. This functionality exists in Juypter but I can't find it in Databricks. Any pointers?
- 17259 Views
- 7 replies
- 2 kudos
- 2 kudos
Yes! Run > Clear > Clear all cell outputs Fun fact, this feature was made ~10 years ago when we realised all our customer demos looked very messy and had lots of spoilers in them!
- 2 kudos
- 747 Views
- 1 replies
- 1 kudos
Concurrent append exception - Two streaming sources writing to same record on the delta table
Hi All, I have a scenario where there are 2 streaming sources Stream1( id, col1, col2) and Stream 2( id, col3, col4) and my delta table has columns (id, col1, col2, col3, col4). My requirement is to insert the record into the delta table if the corre...
- 747 Views
- 1 replies
- 1 kudos
- 1 kudos
I would keep both write operations separate, i.e. they should write in own tables/partitions. In later stages (e.g. silver), you can easily merge them.
- 1 kudos
- 795 Views
- 1 replies
- 0 kudos
error in notebook execution
Error in callback <bound method UserNamespaceCommandHook.post_run_cell of <dbruntime.DatasetInfo.UserNamespaceCommandHook object at 0x7f5790c07070>> (for post_run_cell)
- 795 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you show us your code?
- 0 kudos
- 1889 Views
- 1 replies
- 2 kudos
node_timeline system table is not producing any output in UC
node_timeline system table is not producing any output in UC.In system tables node_timeline table is not having any data .Do we need to enable to any other services for UC
- 1889 Views
- 1 replies
- 2 kudos
- 1205 Views
- 1 replies
- 0 kudos
Enable serverless in Delta Live Tables in Azure Databricks?
I'm trying to create a serverless DLT, as far I understood is in Public Review as listed here, Azure Databricks regions - Azure Databricks | Microsoft Learn. I've created a workspace in North Europe, but I cannot see the feature in preview.Could you ...
- 1205 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod,Thanks for the quick response. I have checked my region is on Public Preview for DLT, North Europe. But I am not able to see the "Serverless" checkbox.Am I missing some detail?Thanks!
- 0 kudos
- 2573 Views
- 3 replies
- 0 kudos
Databricks -Terraform- (condition_task)
Hi Team ,I am planning to create IF/ELSE condition task in databricks using terraform code . My requirement is Task A ( Extract records from DB and Count recs) --> Task B ( validate the counts using Condition_task) --> Task c ( load data if Task B va...
- 2573 Views
- 3 replies
- 0 kudos
- 0 kudos
Implementing conditional logic in Databricks using Terraform involves setting up tasks and condition checks between them. Here's how you can structure your Terraform code to achieve the desired workflow:Step 1: Define Databricks Notebooks as TasksAs...
- 0 kudos
- 14767 Views
- 7 replies
- 3 kudos
How can I start SparkSession out of Notebook?
Hi community,How can I start SparkSession out of Notebook?I want to split my Notebook into small Python modules, and I want to let some of them to call Spark functionality.
- 14767 Views
- 7 replies
- 3 kudos
- 3 kudos
Just overtake Databricks sparksession.from pyspark.sql import SparkSession spark = SparkSession.getActiveSession()
- 3 kudos
- 1111 Views
- 0 replies
- 0 kudos
Get Block Size
Hi guys,How can I get the block size ? have any idea ?Thank you
- 1111 Views
- 0 replies
- 0 kudos
- 5041 Views
- 6 replies
- 5 kudos
Workaround for GraphFrames not working on Delta Live Table?
According to this page, the GraphFrames package is included in the databricks runtime since at least 11.0. However trying to run a connected components algorithm inside a delta live table notebook yields the error java.lang.ClassNotFoundException: or...
- 5041 Views
- 6 replies
- 5 kudos
- 5 kudos
I'm also trying to use GraphFrames inside a DLT pipeline. I get an error that graphframes not installed in the cluster. i"m using it successfully in test notebooks using the ML version of the cluster. Is there a way to use this inside a DLT job?
- 5 kudos
- 1464 Views
- 2 replies
- 0 kudos
Classroom setup Error
Encountering error in running classroom setup command. Help me to resolve this issue. Thank you.
- 1464 Views
- 2 replies
- 0 kudos
- 0 kudos
The error happens in the classroom-setup notebook you're running. It is not possible to debug with the information given.
- 0 kudos
- 2373 Views
- 2 replies
- 1 kudos
Vacuum and Streaming Issue
Hi User Community,Requesting some advice on the below issue please:I have 4 Databricks notebooks, 1 That ingests data from a Kafka topic (metric data from many servers) and dumps the data in parquet format into a specified location. My 2nd data brick...
- 2373 Views
- 2 replies
- 1 kudos
- 5595 Views
- 2 replies
- 1 kudos
Resolved! How to stop a Streaming Job based on time of the week
I have an always-on job cluster triggering Spark Streaming jobs. I would like to stop this streaming job once a week to run table maintenance. I was looking to leverage the foreachBatch function to check a condition and stop the job accordingly.
- 5595 Views
- 2 replies
- 1 kudos
- 1 kudos
You could also use the "Available-now micro-batch" trigger. It only processes one batch at a time, and you can do whatever you want in between batches (sleep, shut down, vacuum, etc.)
- 1 kudos
- 618 Views
- 1 replies
- 0 kudos
Delta sharing api?
Can we use s3 api for delta sharing?
- 618 Views
- 1 replies
- 0 kudos
- 0 kudos
As of now Delta Sharing does not directly support the S3 API.
- 0 kudos
- 2758 Views
- 3 replies
- 2 kudos
system.billing.usage table missing data for jobs running in my databricks account
I have some jobs running on databricks. I can obtain their jobId from the Jobs UI or List Job Runs API.However when trying to get DBU usage for the corresponding jobs from system.billing.usage, I do not see the same job_id in that table. Its been mor...
- 2758 Views
- 3 replies
- 2 kudos
- 3256 Views
- 1 replies
- 0 kudos
Unit Testing DLT Pipelines
Now we are moving our DLT Pipelines into production, we would like to start looking at unit testing the transformation logic inside DLT notebooks.We want to know how we can unit test the PySpark logic/transformations independently without having to s...
- 3256 Views
- 1 replies
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 347 | |
| 286 | |
| 252 |
