Databricks

Resolved! Databricks Ganglia

Hi, is there any way to get alert automatically from databricks ganglia ? That means that a developer don’t need to review the logs manually but would get notification that resources are underutilized for example.

Resolved! scheduling tasks through CLI

Is there a way to schedule tasks or jobs through the Databricks CLI instead of the GUI? I want to be able to create a job flow with different notebook through the CLI.

Resolved! Problem with network security group (NSG) rules in case of VNet injection

Hi everyone,Our internal company security policy for the Cloud infrastructure requires to have custom outbound NSG rule that denies all traffic. The rules attributes should be as follows: Priority: 4096Port: AnyProtocol: AnySource: AnyDestination: An...

Resolved! Delta, the specified key does not exist error

Hi, I'm having this error too frequently on a few tables, I check on S3 and the partition exists and the file is there on the partition.error: Spectrum Scan Error: DeltaManifestcode: 15005context: Error fetching Delta Lake manifest delta/product/sub_...

Resolved! Cannot open Jobs tab in Databricks Community edition.

Hi,I get the following exception when I try to open jobs tab.Uncaught TypeError: Cannot read properties of undefined (reading 'apply')   Reload the page and try again. If the error persists, contact support. Reference error code: fd9ae37c18c1400cb15...

Resolved! "import pandas as pd" => [Errno 5]

When I type import pandas as pdfrom a Notebook in a Repo I get:--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) /usr/lib/python3.8/importlib/_boots...

Resolved! Spark Streaming - Checkpoint State EOF Exception

I have a Spark Structured Streaming job which reads from 2 Delta tables in streams , processes the data and then writes to a 3rd Delta table. The job is being run with the Databricks service on GCP.Sometimes the job fails with the following exception...

Resolved! Parsing 5 GB json file is running long on cluster

I was creating delta table from ADLS json input file. but the job was running long while creating delta table from json. Below is my cluster configuration. Is the issue related to cluster config ? Do I need to upgrade the cluster config ?The cluster ...

Resolved! HTML Backup Import Into my Account

Hi AllI would like to Import my HTML notebook backup into my databricks account and use it as if it was my master (I am a developer and have many exported HTML backups that I may want to reuse. When you open an .HTML from backup, databricks has, ...

Resolved! Databricks and CD4ML

I would like to use Databricks in a CD4ML way (see also https://martinfowler.com/articles/cd4ml.html). Is this possible? I would like to develop and train models in one environment once qualified, I would like to deploy the model with the application...

Resolved! How many jobs can I create in my Databricks workspace?

Resolved! Show Vacuum operation result (files deleted) without DRY RUN

Hi, I'm runing some scheduled vacuum jobs and would like to know how many files were deleted without making all the computation twice, with and without DRY RUN, is there a way to accomplish this?Thanks!

Resolved! Change Data Feed functionality from SQL Endpoint

I am trying to run command to retrieve change data from sql endpoint. It is throwing below error."The input query contains unsupported data source(s).Only csv, json, avro, delta, parquet, orc, text data sources are supported on Databricks SQL."But th...

Resolved! How to update external metastore cluster configuration on the fly ?

Hello,In my use case, my data is pushed to an adls gen2 container called ingestAfter some data processing on a databricks cluster of the ingest workspace, I declare the associated table in an external metastore for this workspaceAt the end of this pr...