- 3891 Views
- 2 replies
- 2 kudos
Resolved! Unstable workflow runs lately
Hi!We are using Databricks on Azure on production since about 3 months. A big part of what we use Databricks for is processing data using a workflow with various Python notebooks. We run the workflow on a 'Pools' cluster and on a 'All-purpose compute...
- 3891 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi holly,Thanks for your reply, good to hear that the 403 errors are on the radar and due to be fixed. I will reach out to support in case of further issues.
- 2 kudos
- 2137 Views
- 1 replies
- 0 kudos
Python databricks sql.connect takes 10minutes exact to connect to Serverless SQL Datawarehouse
Have tried this with both the SQL Warehouse up and running and turned off. The connection eventually is made and works without issue, but the initial connect() takes 10minutes to establish. No other connectivity issues between the box and the Warehou...
- 2137 Views
- 1 replies
- 0 kudos
- 0 kudos
@Retired_mod thank you for the reply, it seems the issue came down to the script was running with Python 3.12 and I should have been using 3.11.9
- 0 kudos
- 6841 Views
- 7 replies
- 2 kudos
Resolved! Incremental Loads from a Catalog/DLT
The databricks guide outlines several use cases for loading data via delta live tables:https://docs.databricks.com/en/delta-live-tables/load.html This includes autoloader from cloudfiles, kafka messages, small static datasets, etc.But, one use case i...
- 6841 Views
- 7 replies
- 2 kudos
- 2 kudos
Hello @lprevost , I hope this message finds you well. In order to install this library, you can follow any of the steps below:- First step - search for libraries and versions: Go the cluster;Libraries;Install new;Maven;Search Packages;Change from Spa...
- 2 kudos
- 7445 Views
- 1 replies
- 0 kudos
Resolved! Adding new columns to a Delta Live table in a CDC process
Hi I am new to databricks and still learning.I am trying to do a CDC on a table. APPLY CHANGES INTO LIVE.table1 FROM schema2.table2 KEYS (Id) SEQUENCE BY orderByColumn COLUMNS * EXCEPT (col1, col2) STORED AS SCD TYPE 1 ; table1 is in schema1 and ...
- 7445 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ameya , Scenario 1: Enabling Delta schema evolution in your table or at DLT pipeline level should suffice for the scenario of new fields being added to the schema. Scenario 2: The INSERT statement doesn't support schema evolution as described in...
- 0 kudos
- 2347 Views
- 2 replies
- 2 kudos
Resolved! Can we parameterize the compute in job cluster
I have created a workflow job in databricks with job parameters.I want to run the job same with different workloads and data volume.So I want the compute cluster to be parametrized so that I can pass the compute requirements(driver, executor size and...
- 2347 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @NarenderKumar , Have you considered leveraging autoscaling for the existing cluster?If this does not meet your needs, are the differing volume/workloads known in advance? If so, could different compute be provisioned using Infrastructure as Code ...
- 2 kudos
- 1599 Views
- 1 replies
- 0 kudos
Is it possible to configure a DLT pipeline continue to run when a single table ingestion failed?
Is it possible to configure a DLT pipeline continue to run when a single table ingestion failed?
- 1599 Views
- 1 replies
- 0 kudos
- 9097 Views
- 5 replies
- 3 kudos
Pyspark serialization
Hi,I was looking for comprehensive documentation on implementing serialization in pyspark, most of the places I have seen is all about serialization with scala. Could you point out where I can get a detailed explanation on it?
- 9097 Views
- 5 replies
- 3 kudos
- 3 kudos
Thank you @Retired_mod for the prompt reply. This clears the things and also distinguishes between spark-scala and pyspark. Appreciate your explanation. Will apply this and also share any findings based on this which will help the community!
- 3 kudos
- 28233 Views
- 7 replies
- 2 kudos
Resolved! How do I clear all output results in a notebook?
I'm building notebooks for tutorial sessions and I want to clear all the output results from the notebook before distributing it to the participants. This functionality exists in Juypter but I can't find it in Databricks. Any pointers?
- 28233 Views
- 7 replies
- 2 kudos
- 2 kudos
Yes! Run > Clear > Clear all cell outputs Fun fact, this feature was made ~10 years ago when we realised all our customer demos looked very messy and had lots of spoilers in them!
- 2 kudos
- 1633 Views
- 1 replies
- 1 kudos
Concurrent append exception - Two streaming sources writing to same record on the delta table
Hi All, I have a scenario where there are 2 streaming sources Stream1( id, col1, col2) and Stream 2( id, col3, col4) and my delta table has columns (id, col1, col2, col3, col4). My requirement is to insert the record into the delta table if the corre...
- 1633 Views
- 1 replies
- 1 kudos
- 1 kudos
I would keep both write operations separate, i.e. they should write in own tables/partitions. In later stages (e.g. silver), you can easily merge them.
- 1 kudos
- 1457 Views
- 1 replies
- 0 kudos
error in notebook execution
Error in callback <bound method UserNamespaceCommandHook.post_run_cell of <dbruntime.DatasetInfo.UserNamespaceCommandHook object at 0x7f5790c07070>> (for post_run_cell)
- 1457 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you show us your code?
- 0 kudos
- 2824 Views
- 1 replies
- 2 kudos
node_timeline system table is not producing any output in UC
node_timeline system table is not producing any output in UC.In system tables node_timeline table is not having any data .Do we need to enable to any other services for UC
- 2824 Views
- 1 replies
- 2 kudos
- 2235 Views
- 1 replies
- 0 kudos
Enable serverless in Delta Live Tables in Azure Databricks?
I'm trying to create a serverless DLT, as far I understood is in Public Review as listed here, Azure Databricks regions - Azure Databricks | Microsoft Learn. I've created a workspace in North Europe, but I cannot see the feature in preview.Could you ...
- 2235 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod,Thanks for the quick response. I have checked my region is on Public Preview for DLT, North Europe. But I am not able to see the "Serverless" checkbox.Am I missing some detail?Thanks!
- 0 kudos
- 22160 Views
- 7 replies
- 3 kudos
How can I start SparkSession out of Notebook?
Hi community,How can I start SparkSession out of Notebook?I want to split my Notebook into small Python modules, and I want to let some of them to call Spark functionality.
- 22160 Views
- 7 replies
- 3 kudos
- 3 kudos
Just overtake Databricks sparksession.from pyspark.sql import SparkSession spark = SparkSession.getActiveSession()
- 3 kudos
- 1626 Views
- 0 replies
- 0 kudos
Get Block Size
Hi guys,How can I get the block size ? have any idea ?Thank you
- 1626 Views
- 0 replies
- 0 kudos
- 5348 Views
- 2 replies
- 0 kudos
Databricks -Terraform- (condition_task)
Hi Team ,I am planning to create IF/ELSE condition task in databricks using terraform code . My requirement is Task A ( Extract records from DB and Count recs) --> Task B ( validate the counts using Condition_task) --> Task c ( load data if Task B va...
- 5348 Views
- 2 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 577 | |
| 349 | |
| 287 |
