topic Re: Serverless compute databricks in Data Engineering

Serverless compute databricks

pinaki1 — Wed, 23 Oct 2024 06:08:31 GMT

1. How to connect s3 bucket to databricks since dbfs mount is not supported.?
2. In serverless compute Spark Context (sc), spark.sparkContext, and sqlContext are not supported?. Does it means it will not leverage power of distributed processing?
3. What is the impact on delta tables in serverless compute?

Re: Serverless compute databricks

saurabh18cs — Wed, 23 Oct 2024 08:35:08 GMT

1. Access the S3 bucket directly using AWS credentials

spark = SparkSession.builder \

.appName("S3Access") \

.config("spark.hadoop.fs.s3a.access.key", "<your-access-key-id>") \

.config("spark.hadoop.fs.s3a.secret.key", "<your-secret-access-key>") \

.getOrCreate()

2. Databricks encourages the use of the SparkSession object, which provides a unified entry point for working with Spark.

from pyspark.sql import SparkSession

# Create a SparkSession

spark = SparkSession.builder.appName("ServerlessComputeExample").getOrCreate()

3. Serverless compute automatically scales resources based on workload demands. This can be beneficial for Delta tables as it ensures that sufficient resources are available for large-scale data processing tasks.

Re: Serverless compute databricks

pinaki1 — Fri, 25 Oct 2024 05:54:24 GMT

Getting this error
[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SELECT on any file. SQLSTATE: 42501
while trying to read from s3 bucket as per your approach

Re: Serverless compute databricks

User16653924625 — Fri, 08 Nov 2024 17:17:29 GMT

please see this documentation for accessing cloud storage by setting Unity Catalog objects: Storage Credential and External Location.

https://docs.databricks.com/en/connect/unity-catalog/cloud-storage/index.html

Re: Serverless compute databricks

Yaswanth057 — Wed, 22 Jan 2025 07:59:15 GMT

Which approach worked for you, we are also facing same error as yours.