missing-QuestionPost topics

Databricks Certification and Badge not received

abhisheksreddy1 — Wed, 26 Mar 2025 13:44:24 GMT

Hello, I passed DEA-C01 exam on March 25, 2025. I haven't received my certification or credentials yet. Can you please check?

I am not able to view Lakehouse Fundamental training videos, I just see menus, clicking does not open anything

pg1 — Tue, 06 Jun 2023 19:45:49 GMT

I am not able to view Lakehouse Fundamental training videos, I just see menus, clicking does not open anything

Grant Privileges to multiple tables/views in single schema

Randomname — Thu, 08 Jun 2023 15:39:04 GMT

Is there a way to grant privileges to multiple tables/views using wild cards ?

Example like Grant Select on *_views to user?

How to use Databricks Spark with Ibis

ccbeloy — Thu, 08 Jun 2023 19:00:21 GMT

Ibis now includes pyspark backend. However using the databricks-connect doesn't seem to work with Ibis. Here's the sample code the throws an error.

from databricks.connect import DatabricksSession

from databricks.sdk.core import Config

config = Config(profile = "dev")

spark = DatabricksSession.builder.sdkConfig(config).getOrCreate()

df = spark.read.parquet("/somefile.parquet")

df.createOrReplaceTempView("sometable")

import ibis

from ibis import _

ibis_con = ibis.pyspark.connect(spark)

above throws an error:

Python39\site-packages\pyspark\sql\connect\session.py", line 532, in sparkContext

raise NotImplementedError("sparkContext() is not implemented.")

NotImplementedError: sparkContext() is not implemented.

Functional Difference between managed and external tables UC

carlosjrestr — Fri, 09 Jun 2023 01:04:20 GMT

Hi community,

Is there a summary or detail guidance on the functional difference between managed and external tables in unity catalog? Looking at the documentation I can't find any specific functionality in databricks that is supported in managed tables that is not supported in external tables. Assuming I have external tables using delta format what capabilities I will not be able to use if I don't convert them to managed tables?

Thanks

I could not access the deltatable from databricks. Error : org.apache.spark.sql.AnalysisException:

Karthe — Fri, 09 Jun 2023 06:58:10 GMT

Hi all,

I am not able to access the delta table from the datbase.

While I try to read the table via spark.read.table command, I get the below error:

org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

Here is the content from error log:

Fri Jun 9 05:57:38 2023 Connection to spark from PID 1377

Fri Jun 9 05:57:38 2023 Initialized gateway on port 44015

Fri Jun 9 05:57:38 2023 Connected to spark.

Tried to attach usage logger `pyspark.databricks.pandas.usage_logger`, but an exception was raised: <property object at 0x7fdd5dc448b0> is not a callable object

Fri Jun 9 06:04:15 2023 Connection to spark from PID 1629

Fri Jun 9 06:04:15 2023 Initialized gateway on port 37083

Fri Jun 9 06:04:16 2023 Connected to spark.

Tried to attach usage logger `pyspark.databricks.pandas.usage_logger`, but an exception was raised: <property object at 0x7f6e23727770> is not a callable object

Fri Jun 9 06:13:51 2023 Connection to spark from PID 1924

Fri Jun 9 06:13:51 2023 Initialized gateway on port 33139

Fri Jun 9 06:13:51 2023 Connected to spark.

While creating a UDF in databricks SQL, how can I declare a local variable? Something like this? CREATE OR REPLACE FUNCTION len() set myString = 'my value'; RETURNS INT RETURN length(myString);

RV — Fri, 09 Jun 2023 13:26:17 GMT

Databricks slack channel

Oliver_Angelil — Sun, 11 Jun 2023 12:01:38 GMT

I would like to know if there is a Databricks slack channel? And if not, would anyone be interested in joining one?

When will I get Databricks Data Engineer associate voucher.

pradyumn9999 — Sun, 11 Jun 2023 16:55:18 GMT

Its been a day since I passed the exam , still no badge recieved from Databricks.

Certificate not received for Databricks Certified Associate Developer for Apache Spark 3.0.

Gaurav007 — Mon, 12 Jun 2023 11:58:38 GMT

Hi,

I have passed the exam for Databricks Certified Associate Developer for Apache Spark 3.0 with 85% on 10 jun 2023. I received a mail where badge and credentials mentioned but didn't received any certificate with it. I raised a ticket also - #00334153

Please send me the certificate on mail

How is everyone's project coming along?

Michelle_-_Devp — Mon, 12 Jun 2023 15:27:34 GMT

There are a few more days until the deadline.🏇 Have you picked a project/topic? Has anyone started building yet or finishing things up? Don't forget about the demo video for your project submission here: https://devpost.com/submit-to/18245-so-you-think-you-can-hack/manage/submissions

multiple drivers for a sample jdbc client added to the Databricks cluster libraries?

shan_chandra — Mon, 12 Jun 2023 21:05:25 GMT

How Databricks treats a jar when we upload multiple drivers for the same jdbc client (for example - oracle) how would databricks treat that jar and which would be considered in the class path?

Access content in dataframe by loc/iloc or by [ ] [ ]?

AleksandraFrolo — Tue, 13 Jun 2023 08:02:16 GMT

Hello,

Task: I am trying to understand, what approach is better to access the content in DataFrame.

My piece of code:

print("First approach: ", df["Purchase Address"][0])
 
print("Second approach: ", df.loc[0,"Purchase Address"])

These lines are equal to each other. For me more comfortable to use first version. Is there any recommends in pandas how to access the content?

Automate Cluster creation

Vidisha — Tue, 13 Jun 2023 10:41:58 GMT

I am new to databricks and my lead told me that we create clusters for running the notebook manually. So please write one python script to do this automation i.e to create the clusters automatically.

Can anyone help me to write the script using PySpark in Databricks. I have to use Azure Cloud Services for this.

dbutils command keeps on running

ravin619 — Tue, 13 Jun 2023 10:29:06 GMT

Hi All,

Im running below command in my cluster

dbutils.fs.ls("abfss://demo@###.dfs.core.window.net")

I have done the spark.conf.set steps before running above command.

It keeps on running for almost 30 mins and still shows as 'Running command'.

I have restarted the cluster many times and tried changing the resource runtime as well.

Please note I'm using azure free subscription plan

h3 hex ID using Databricks Mosaic is not the same as h3.geo_to_h3

kll — Tue, 13 Jun 2023 18:18:34 GMT

I am testing the Databricks Mosaic Spatial Grid Indexing method to obtain the `h3 hex` of a given lat, long.

# Get the latitude and longitude
latitude = 37.7716736
longitude = -122.4485852
 
# Get the resolution
resolution = 7
 
# Get the H3 hex ID
h3_hex_id = grid_longlatascellid(lit(latitude), lit(longitude), lit(resolution)).hex
 
# Print the H3 hex ID
print(h3_hex_id)
 
Column<'grid_longlatascellid(CAST(37.7716736 AS DOUBLE), CAST(-122.4485852 AS DOUBLE), 7)[hex]'>

How do I see the actual hex id in the code above?

According the docs, the `h3 hex id` returned by `grid_longlatascellid` looks different from what is returned by `h3.geo_to_h3` method.

h3.geo_to_h3(float(latitude), float(longitude), 7)  
 
'872830829ffffff'

df = spark.createDataFrame([{'lon': 30., 'lat': 10.}])
df.select(grid_longlatascellid('lon', 'lat', lit(10))).show(1, False)
+----------------------------------+
|grid_longlatascellid(lon, lat, 10)|
+----------------------------------+
|                623385352048508927|

How do I obtain the `h3 hex id` using Databricks Mosaic library? I have the following imports and configurations:

import h3
from mosaic import enable_mosaic
enable_mosaic(spark, dbutils)
from mosaic import *
spark.conf.set("spark.databricks.labs.mosaic.index.system", "H3")

Unable to perform patch operation on users api which is in preview

karthik_p — Tue, 13 Jun 2023 22:57:41 GMT

Hi team,

we would like to lock users from accessing workspace, we are able to get properties of user like groups etc..but when we are doing patch operation it is throwing 500 error

while during post body in path we are providing entitlements

maxFilesPerTrigger not working in bronze to silver layer

sanjay — Wed, 14 Jun 2023 09:40:04 GMT

Hi,

I am using Matillion architecture where autoloader picks files from AWS S3 and saves in delta lake. Next layer picks the changes from delta lake and does some processing. I am able to set batch size in autoloader and its working. But in bronze to silver layer, unable to set batch limit, its picking all files in one go. Here is my code from bronze to silver layer..

(spark.readStream.format("delta")

.option("useNotification","true")

.option("includeExistingFiles","true")

.option("allowOverwrites",True)

.option("ignoreMissingFiles",True)

.option("maxFilesPerTrigger", 100)

.load(bronze_path)

.writeStream

.option("checkpointLocation", silver_checkpoint_path)

.trigger(processingTime="1 minute")

.foreachBatch(foreachBatchFunction)

.start()

)

Appreciate any help.

Regards,

Sanjay

Connect Databricks with SSAS multidimensional Project

Mehala — Thu, 15 Jun 2023 11:43:12 GMT

Hi,

I just wanted to connect Databricks with SSAS multidimensional Project/

want to use Databricks connection string and provider.

Is that possible to do in SSAS multidimensional Project?

Which Provider I need to use?

Or Is there any other workaround to achieve this scenario?

@Hubert Dudek ,@Werner Stinckens , @Aviral Bhardwaj , @Omkar G , @Taha Hussain , @Adam Pavlacka , @Ananth Arunachalam , @Vidula Khanna , @Jose Alfonso , @Kaniz Fatma

Checklist or Document for Migrating from Dev to Prod

Enzo_Bahrami — Fri, 16 Jun 2023 17:29:11 GMT

Hello Everyone!

So we have a dev environment in Databricks and want to migrate it to prod.

I need it o go over every single table, schema, notebooks, and artifacts in the databricks and make sure nothing is hard-coded for example or that there is nothing compromising the prod environment.

Do you any checklist or resource to help in this regards? Maybe a checklist of what are the best practices and what to look over. I want to prepare a diagnosis of the current status of the project.

thank you all!