Not able to call external API when using Databricks free edition
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-26-2025 08:06 AM
Hello guys , I'm new to Databricks and trying to create ETL pipelines for my personal projects . I'm using the Databricks free Edition but whenever I try to call an API to an external source to extract Data I always get this error .
I have attached the Image of the error below .
Why this keeps happening , am I not aware of something or doing something wrong ? Can someone please help ?
- Labels:
-
Workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-26-2025 09:18 AM
Hi @AKB_10
This is a common issue with Databricks Community Edition (free tier). Here's what's happening and how to fix it:
Root Cause
Databricks Community Edition has network restrictions that block outbound API calls to external services like Spotify,
social media APIs, etc. This is a security limitation of the free tier.
That’s why you get a DNS resolution error: the environment cannot resolve or connect to accounts.spotify.com.
This is a security and cost-control limitation. Since the free Community Edition runs on a shared infrastructure, Databricks blocks:
- Internet-bound traffic
- Some file system access
- Custom library installations from the internet
Solutions:
1. Upgrade to Databricks Premium or use a cloud-connected workspace
If you want to access external APIs:
- Use Databricks on AWS, Azure, or GCP with your own cloud subscription.
- Set up internet access through NAT gateway if on VPC.
2. Develop locally (e.g., in Jupyter or VS Code)
Run your ETL code on your local machine, where you have full internet access. Then:
- Upload results to Databricks if needed for further processing.
