Databricks Community

johnp · 07-17-2024

We just moved our workflow from "all purpose compute cluster" to "job compute cluster". We need to find out the external public IP of the Job Compute cluster. On the all purpose compute cluster, we get the IP by attaching a notebook and run the comm...

johnp · 07-11-2024

I have two structured streaming notebooks running continuously for anomaly detection. Both notebooks import the same python module to mount the Azure blob storage, but each has its own container. Each notebook runs well when it has its own cluster. ...

johnp · 07-11-2024

I am the contributor and owner of my databricks workspace. After a recent spike of expense, I want to check the billing details of my Azure databricks usage. (i.e per cluster, per VM, etc). Databricks provides these information thorough "Admin Conso...

johnp · 04-22-2024

I have an Azure web app running flask web server. From flask server, I want to run some queries on the data stored in ADLS Gen2 storage. I already created Databricks notebooks running these queries. The flask server will pass some parameters in ...

johnp · 04-02-2024

We have an Azure app service written in Django. From databricks notebook we sent curl command to test the connection between databricks and Azure AppService. We got the following repsonse:Error 403 - ForbiddenThe web app you have attempted to reach...

johnp · 07-17-2024

I found the following IPs from the Cluster JSON file:"driver": {"private_ip": "10.*.*.*","public_dns": "172.*.*.*","node_id": "80*****",Similar the executors configuration"executors": [{"private_ip": "10.*.*.*","public_dns": "172.*.*.*","node_id": "7...

johnp · 07-12-2024

@szymon_dybczak Thanks for the help! In my case, the two notebooks are totally separated, but both imports (Not a Run) the same python module to get the variable adls_path. From your explanation, they should not interfere with each other. I will run...

johnp · 03-23-2024

I test with some simple applications, it works as you described. However, the application I am debugging uses the pyspark structured streaming, which runs continuously. After inserting pdb.set_trace(), the application paused at the breakpoint, but t...

Databricks Community

User Stats

User Activity

Get the external public IP of the Job Compute cluster

cluster sharing between different notebooks

Access to "Admin Console" and "System Tables"

Call databricks notebook from azure flask app

Access Azure App service failed with 403 response

Re: Get the external public IP of the Job Compute cluster

Re: cluster sharing between different notebooks

Re: pdb debugger on databricks