cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Serverless compute databricks

pinaki1
New Contributor III

1. How to connect s3 bucket to databricks since dbfs mount is not supported.?
2. In serverless compute Spark Context (sc), spark.sparkContext, and sqlContext are not supported?. Does it means it will not leverage power of distributed processing?
3. What is the impact on delta tables in serverless compute?

3 REPLIES 3

saurabh18cs
Contributor II

1. Access the S3 bucket directly using AWS credentials

spark = SparkSession.builder \
    .appName("S3Access") \
    .config("spark.hadoop.fs.s3a.access.key", "<your-access-key-id>") \
    .config("spark.hadoop.fs.s3a.secret.key", "<your-secret-access-key>") \
    .getOrCreate()
 
2. Databricks encourages the use of the SparkSession object, which provides a unified entry point for working with Spark.
from pyspark.sql import SparkSession

# Create a SparkSession
spark = SparkSession.builder.appName("ServerlessComputeExample").getOrCreate()
 
3. Serverless compute automatically scales resources based on workload demands. This can be beneficial for Delta tables as it ensures that sufficient resources are available for large-scale data processing tasks.

pinaki1
New Contributor III

Getting this error
[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SELECT on any file. SQLSTATE: 42501
while trying to read from s3 bucket as per your approach

User16653924625
Databricks Employee
Databricks Employee

please see this documentation for accessing cloud storage by setting Unity Catalog objects: Storage Credential and External Location. 

https://docs.databricks.com/en/connect/unity-catalog/cloud-storage/index.html

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group