- 2276 Views
- 1 replies
- 0 kudos
Enable serverless in Delta Live Tables in Azure Databricks?
I'm trying to create a serverless DLT, as far I understood is in Public Review as listed here, Azure Databricks regions - Azure Databricks | Microsoft Learn. I've created a workspace in North Europe, but I cannot see the feature in preview.Could you ...
- 2276 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod,Thanks for the quick response. I have checked my region is on Public Preview for DLT, North Europe. But I am not able to see the "Serverless" checkbox.Am I missing some detail?Thanks!
- 0 kudos
- 22364 Views
- 7 replies
- 3 kudos
How can I start SparkSession out of Notebook?
Hi community,How can I start SparkSession out of Notebook?I want to split my Notebook into small Python modules, and I want to let some of them to call Spark functionality.
- 22364 Views
- 7 replies
- 3 kudos
- 3 kudos
Just overtake Databricks sparksession.from pyspark.sql import SparkSession spark = SparkSession.getActiveSession()
- 3 kudos
- 1648 Views
- 0 replies
- 0 kudos
Get Block Size
Hi guys,How can I get the block size ? have any idea ?Thank you
- 1648 Views
- 0 replies
- 0 kudos
- 5466 Views
- 2 replies
- 0 kudos
Databricks -Terraform- (condition_task)
Hi Team ,I am planning to create IF/ELSE condition task in databricks using terraform code . My requirement is Task A ( Extract records from DB and Count recs) --> Task B ( validate the counts using Condition_task) --> Task c ( load data if Task B va...
- 5466 Views
- 2 replies
- 0 kudos
- 7510 Views
- 6 replies
- 5 kudos
Workaround for GraphFrames not working on Delta Live Table?
According to this page, the GraphFrames package is included in the databricks runtime since at least 11.0. However trying to run a connected components algorithm inside a delta live table notebook yields the error java.lang.ClassNotFoundException: or...
- 7510 Views
- 6 replies
- 5 kudos
- 5 kudos
I'm also trying to use GraphFrames inside a DLT pipeline. I get an error that graphframes not installed in the cluster. i"m using it successfully in test notebooks using the ML version of the cluster. Is there a way to use this inside a DLT job?
- 5 kudos
- 2318 Views
- 2 replies
- 0 kudos
Classroom setup Error
 Encountering error in running classroom setup command. Help me to resolve this issue. Thank you.
- 2318 Views
- 2 replies
- 0 kudos
- 0 kudos
The error happens in the classroom-setup notebook you're running. It is not possible to debug with the information given.
- 0 kudos
- 3944 Views
- 2 replies
- 1 kudos
Vacuum and Streaming Issue
Hi User Community,Requesting some advice on the below issue please:I have 4 Databricks notebooks, 1 That ingests data from a Kafka topic (metric data from many servers) and dumps the data in parquet format into a specified location. My 2nd data brick...
- 3944 Views
- 2 replies
- 1 kudos
- 7145 Views
- 2 replies
- 1 kudos
Resolved! How to stop a Streaming Job based on time of the week
I have an always-on job cluster triggering Spark Streaming jobs. I would like to stop this streaming job once a week to run table maintenance. I was looking to leverage the foreachBatch function to check a condition and stop the job accordingly.
- 7145 Views
- 2 replies
- 1 kudos
- 1 kudos
You could also use the "Available-now micro-batch" trigger. It only processes one batch at a time, and you can do whatever you want in between batches (sleep, shut down, vacuum, etc.)
- 1 kudos
- 1195 Views
- 1 replies
- 0 kudos
Delta sharing api?
Can we use s3 api for delta sharing?
- 1195 Views
- 1 replies
- 0 kudos
- 0 kudos
As of now Delta Sharing does not directly support the S3 API.
- 0 kudos
- 4399 Views
- 3 replies
- 2 kudos
system.billing.usage table missing data for jobs running in my databricks account
I have some jobs running on databricks. I can obtain their jobId from the Jobs UI or List Job Runs API.However when trying to get DBU usage for the corresponding jobs from system.billing.usage, I do not see the same job_id in that table. Its been mor...
- 4399 Views
- 3 replies
- 2 kudos
- 4101 Views
- 1 replies
- 0 kudos
Unit Testing DLT Pipelines
Now we are moving our DLT Pipelines into production, we would like to start looking at unit testing the transformation logic inside DLT notebooks.We want to know how we can unit test the PySpark logic/transformations independently without having to s...
- 4101 Views
- 1 replies
- 0 kudos
- 3891 Views
- 2 replies
- 0 kudos
Resolved! Persisting query history data
Hello,I am looking for a way to persist query history data. I have not have direct access to the system tables. I do have access to a query_history view created by selecting from the system.query.history and system.access.audit system tables. I want ...
- 3891 Views
- 2 replies
- 0 kudos
- 20194 Views
- 9 replies
- 3 kudos
Resolved! Invalid configuration fs.azure.account.key trying to load ML Model with OAuth
Hi Community,i was trying to load a ML Model from a Azure Storageaccount (abfss://....) with: model = PipelineModel.load(path) i set the spark config: spark.conf.set("fs.azure.account.auth.type", "OAuth") spark.conf.set("fs.azure.account.oauth.provi...
- 20194 Views
- 9 replies
- 3 kudos
- 3 kudos
@daniel_sahal any possible reason you know of why it works in OSS spark while it does not work in databricks notebook ? Why is there a disparity.
- 3 kudos
- 1293 Views
- 0 replies
- 3 kudos
Parametrizing query for DEEP CLONE
Update: Hey moderator, I've removed the link to the Bobby tables XKCD to reassure that this post is not spam Hi, I'm somehow unable to write a parametrized query to create a DEEP CLONE. I'm trying really hard to avoid using string interpolation (to p...
- 1293 Views
- 0 replies
- 3 kudos
- 15547 Views
- 5 replies
- 5 kudos
Hi Everyone , I was wondering if it is possible to have alerts set up on query level for pyspark notebooks that are run on schedule in databricks so if we have some expected result from it we can receive a mail alert ?
In Above you can see we have 3 workspaces - we have the alert option available in the sql workspace but not in our data science and engineering space , anyway we can incorporate this in our DS and Engineering space ?
- 15547 Views
- 5 replies
- 5 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
