cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

AdamStra2
by New Contributor III
  • 755 Views
  • 3 replies
  • 1 kudos

Web terminal and clusters

Hi, I have come across this piece of documentation:Databricks does not support running Spark jobs from the web terminal. In addition, Databricks web terminal is not available in the following cluster types:Job clustersClusters launched with the DISAB...

  • 755 Views
  • 3 replies
  • 1 kudos
Latest Reply
AdamStra2
New Contributor III
  • 1 kudos

Hi @Kaniz ,any update on my question? Thanks.

  • 1 kudos
2 More Replies
sm1274
by New Contributor
  • 1114 Views
  • 1 replies
  • 0 kudos

Creating java UDF for Spark SQL

Hello, I have created a sample java UDF which masks few characters of a string. However I facing couple of issues when uploading and using it.First I could only import it, which for now is OK. But when do the following,create function udf_mask as 'ba...

  • 1114 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @sm1274, The error message you received indicates that the CREATE FUNCTION statement is not supported on a Databricks SQL endpoint. This statement is the specific error message you're seeing indicating that you're trying to run the CREATE FUNCTION...

  • 0 kudos
s_park
by Valued Contributor II
  • 14380 Views
  • 3 replies
  • 4 kudos

Training @ Data & AI World Tour 2023

Join your peers at the Data + AI World Tour 2023! Explore the latest advancements, hear real-world case studies and discover best practices that deliver data and AI transformation. From the Databricks Lakehouse Platform to open source technologies in...

Screenshot 2023-10-09 at 10.42.55 AM.png
Get Started Discussions
DAIWT
DAIWT_2023
Training
User_Group
  • 14380 Views
  • 3 replies
  • 4 kudos
Latest Reply
VjGian15
New Contributor II
  • 4 kudos

Introducing Mini Flush: Your Ticket to Ultimate Casino Thrills!Are you ready to embark on an electrifying journey into the world of online gambling? If so, look no further than Vijaybet Online Casino! Our state-of-the-art platform is your gateway to ...

  • 4 kudos
2 More Replies
sg-vtc
by New Contributor III
  • 933 Views
  • 1 replies
  • 1 kudos

Resolved! Problem creating external delta table on non-AWS s3 bucket

I am testing Databricks with non-AWS S3 object storage.  I can access the non-AWS S3 bucket by setting these parameters:sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", "XXXXXXXXXXXXXXXXXXXX")sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key...

sgvtc_0-1697817308224.png sgvtc_1-1697817308223.png sgvtc_2-1697817308221.png
Get Started Discussions
external delta table
  • 933 Views
  • 1 replies
  • 1 kudos
Latest Reply
sg-vtc
New Contributor III
  • 1 kudos

Found the solution to disable it.  Can close this question.

  • 1 kudos
Data_Analytics1
by Contributor III
  • 620 Views
  • 2 replies
  • 0 kudos

Getting secret from Key Vault of previous version

Hi,I have added secrets in Azure Key Vault and also updated it few times. I need to access current as well as previous version secret in a data pipeline. dbutils.secrete.get(KeyName, SecretScopeName)This gives me the current version of secret.How can...

  • 620 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Data_Analytics1, To access a specific version of a secret in Azure Key Vault using dbutils.secrets.get(), you need to append the version number to the secret name.

  • 0 kudos
1 More Replies
AdamStra2
by New Contributor III
  • 9104 Views
  • 1 replies
  • 3 kudos

Resolved! Schema owned by Service Principal shows error in PBI

Background info:1. We have unity catalog enabled. 2. All of our jobs are run by Service Principal that has all necessary access it needs.Issue:One of the jobs checks existing schemas against the ones it is supposed to create in that given run and if ...

pic.png
  • 9104 Views
  • 1 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @AdamStra2, This may be related to ownership chaining in SQL Server. Ownership chaining is a security feature in SQL Server that's designed to allow users to access objects in a database without requiring explicit permissions on the object itself....

  • 3 kudos
HHol
by New Contributor
  • 418 Views
  • 0 replies
  • 0 kudos

How to retrieve a Job Name from the SparkContext

We are currently starting to build certain data pipelines using Databricks.For this we use Jobs and the steps in these Jobs are implemented in Python Wheels.We are able to retrieve the Job ID, Job Run ID and Task Run Id in our Python Wheels from the ...

  • 418 Views
  • 0 replies
  • 0 kudos
cmilligan
by Contributor II
  • 2606 Views
  • 7 replies
  • 1 kudos

Long run time with %run command

My team has started to see long run times on cells when using the %run commands to run another notebook. The notebook that we are calling with %run only contains variable setting, defining functions, and library imports. In some cases I have seen in ...

  • 2606 Views
  • 7 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @cmilligan ,  - Long run times with %run command could be due to notebook size and complexity, Databricks cluster load, and network latency.- %run command executes another notebook immediately, making its functions and variables available in the c...

  • 1 kudos
6 More Replies
Ajbi
by New Contributor II
  • 1368 Views
  • 2 replies
  • 0 kudos

NATIVE_XML_DATA_SOURCE_NOT_ENABLED

I'm trying to read an xml file and receiving the following error. I've installed the maven library spark xml to the cluster, however I'm receiving the error. is there anything i'm missing?ErrorAnalysisException: [NATIVE_XML_DATA_SOURCE_NOT_ENABLED] N...

  • 1368 Views
  • 2 replies
  • 0 kudos
Latest Reply
Ajbi
New Contributor II
  • 0 kudos

i've tried already  spark.read.format('com.databricks.spark.xml'). it receives the same error.  

  • 0 kudos
1 More Replies
liv1
by New Contributor II
  • 839 Views
  • 2 replies
  • 1 kudos

Structured Streaming from a delta table that is a dump of kafka and get the latest record per key

I'm trying to use Structured Streaming in scala to stream from a delta table that is a dump of a kafka topic where each record/message is an update of attributes for the key and no messages from kafka are dropped from the dump, but the value is flatt...

  • 839 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @liv1 ,  To get the latest message per key in your streaming job and perform stream-stream joins, you can use Databricks Delta's time travel feature in combination with foreachBatch(). You can use Delta's time travel feature to maintain the latest...

  • 1 kudos
1 More Replies
eric2
by New Contributor II
  • 1151 Views
  • 3 replies
  • 0 kudos

Databricks Delta table Insert Data Error

When trying to insert data into the Delta table in databricks, an error occurs as shown below. [TASK_WRITE_FAILED] Task failed while writing rows to abfss://cont-01@dlsgolfzon001.dfs.core.windows.net/dir-db999_test/D_RGN_INFO_TMP.In SQL, the results ...

  • 1151 Views
  • 3 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

seems ok to me, have you tried to display the data from table A and also the B/C join?

  • 0 kudos
2 More Replies
ChaseM
by New Contributor II
  • 469 Views
  • 2 replies
  • 0 kudos

how to make distributed predictions with sklearn model?

So I have a sklearn style model which predicts on a pandas df. The data to predict on is a spark df. Simply converting the whole thing at once to pandas and predicting is not an option due to time and memory constraints.Is there a way to chunk a spar...

  • 469 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @ChaseM, You can chunk a Spark DataFrame, convert each chunk to a Pandas DataFrame, and predict each chunk in parallel using worker nodes in Databricks. 

  • 0 kudos
1 More Replies
sg-vtc
by New Contributor III
  • 992 Views
  • 1 replies
  • 0 kudos

problem with workspace after metastore deleted

I am completely new to Databricks AWS and start working on it a week ago.  Pls excuse me if I ask or did something silly.I created a workspace and a single node cluster for testing. A metastore was created from Databricks quickstart and it was automa...

  • 992 Views
  • 1 replies
  • 0 kudos
Latest Reply
sg-vtc
New Contributor III
  • 0 kudos

I restarted the compute node and this problem went away.ErrorClass=METASTORE_DOES_NOT_EXIST] Metastore 'b11fb1a0-a462-4dfb-b91b-e0795fde10b0' does not exist.New question: I am testing Databricks with non-AWS S3 object storage.  I can access the non-A...

  • 0 kudos
aerofish
by New Contributor II
  • 1086 Views
  • 3 replies
  • 1 kudos

drop duplicates within watermark

Recently we are using structured streaming to ingest data. We want to use watermark to drop duplicated event. But We encountered some wired behavior and unexpected exception. Anyone can help me to explain what is the expected behavior and how should ...

  • 1086 Views
  • 3 replies
  • 1 kudos
Latest Reply
aerofish
New Contributor II
  • 1 kudos

Any maintainer can help me on this question??

  • 1 kudos
2 More Replies
bigt23
by New Contributor II
  • 1416 Views
  • 2 replies
  • 1 kudos

Resolved! Read zstd file from Databricks

I just started to read `zstd` compressed file in Databricks on Azure, Runtime 14.1 on Spark 3.5.0I've set PySpark commands as followspath = f"wasbs://{container}@{storageaccount}.blob.core.windows.net/test-zstd" schema = "some schema" df = spark.read...

  • 1416 Views
  • 2 replies
  • 1 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

The available compression types are format dependent.For json, zstd is not (yet) available, whereas for parquet it is.

  • 1 kudos
1 More Replies
Labels
Top Kudoed Authors