Databricks Community

Running SQL Data File through Notebook Python

I am attempting to run larger sql scripts through Databricks Notbook and export data to a file. For the most part the Notebook works when the sql script is a single SELECT statement. However, if the sql file is more complicated such as involving the ...

When should I use ".start()" with writeStream?

Hi,I am practicing with Databricks. In sample notebooks,I have seen different use of writeStream with or without ".start()" method. Samples are below:Without .start() spark.readStream   .format("cloudFiles")   .option("cloudFiles.f...

Question about "foreachBatch" to remove duplicate records when streaming data

Hi,I am practicing with Databricks sample notebook published here:https://github.com/databricks-academy/advanced-data-engineering-with-databricksIn one of the notebooks (ADE 3.1 - Streaming Deduplication) (URL), there is a sample code to remove dupli...

Databricks notebook is not printing image in python

When I run a python command that should display an image. The image is not showing, and I do not get any error message.

Is there any way to mask a column of a table for a user/group without creating dynamic views?

Did not get bedge and 100 points for Databricks Lakehouse Platform Accreditation exam

I have completed databricks Fundamentals of the Databricks Lakehouse Platform Accreditation exam but i did not get bedge and 100 points in my account.Could someone please help.

Unable to access workspace after the trial period in databricks in Google cloud provider.

I was using the trial period in databricks for 14 days and had some important notebooks where I had made all the changes. Now I have extended the service and have subscribed for databricks in GCP. When I enter the workspace section I cannot see the w...

Where can I find API documentation to dbutils.notebook.entry_point?

dbutils.notebook.help only lists "run" and "exit" methods. I could only find references to dbutils.notebook.entry_point spread across the web but there does not seem to be an official Databricks API documentation to its complete APIs anywhere. Can so...

Creating a DLT pipeline that reads from a JDBC source

Hey,I'm trying to create a DLT pipeline that reads from a JDBC source, and the code I'm using looks something like this in python:import dlt @dlt.table def table_name(): driver = 'oracle.jdbc.driver.OracleDriver' url = '...' query = 'SELECT ......

mount s3 bucket with specific endpoint

Environment:AZURE-DatabricksLanguage: PythonI can access my s3 bucket via:boto3.client('s3', endpoint_url='https://gateway.storjshare.io', ... )and it also works via:boto3.resource('s3', endpoint_url='https://gateway.storjshare.io', ... )As a next st...

Delta sharing in Databricks doesn't work

Databricks Delta sharing server seems to be broken. We have a table ~ 10M rows and there is no way for us to query the shared data via any methods (e.g Python/Spark or even another Databricks account that the data was shared with)Any ideas on why thi...

Haven't received Databricks Certificate or any form of correspondence

I passed the Databricks Certified Associate Developer for Apache Spark 3.0 - Python on 10/22/2022 with a score of 85%.My kryterion webassessor account shows a pass. However, I've not yet received any correspondence/badge from Databricks

Community Edition SQL Warehouse is not starting up, is it not free to even trail ?

When i start the sql warehosue , i get this error. Is there a way around to start up, I just want to try some features. Clusters are failing to launch. Cluster launch will be retried.Details for the latest failure: Error: Error code: PublicIPCountLim...

Using large dataframe in-memory (data not allowed to be "at rest") results in driver crash and/or out of memory

I'm having trouble working on Databricks with data that we are not allowed to save off or persist in any way. The data comes from an API (which returns a JSON response). We have a scala package on our cluster that makes the queries (almost 6k queries...