cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
carlosst01
by • New Contributor II
  • 1636 Views
  • 0 replies
  • 0 kudos

java.lang.SecurityException: Could not verify permissions for DeleteFromTable ERROR IN EXTERNAL SYNAPSE TABLE

Hi guys, I am trying to delete an external table in Databricks that is connected to a Synapse table (via the com.databricks.spark.sqldw connector) but got the error shown below:Apart from that, when I try to execute an DELETE and INSERT actions (like...

WhatsApp Image 2023-05-25 at 12.02.52 PM WhatsApp Image 2023-05-25 at 12.08.16 PM WhatsApp Image 2023-05-25 at 12.13.14 PM
Data Engineering
Reply
  • 1636 Views
  • 0 replies
  • 0 kudos
darkraisisi
by • New Contributor
  • 798 Views
  • 0 replies
  • 0 kudos

Is there a way to manually update the cuda required file in the db runtime? There are some rather annoying bugs still in TF 2.11 that have been fixed ...

Is there a way to manually update the cuda required file in the db runtime?There are some rather annoying bugs still in TF 2.11 that have been fixed in TF 2.12.Sadly the latest DB runtime 13.1 (beta) only supports the older TF 2.11 even tho 2.12 was ...

Data Engineering
Reply
  • 798 Views
  • 0 replies
  • 0 kudos
Brian61
by • New Contributor
  • 773 Views
  • 0 replies
  • 0 kudos

I'm getting a strange error whenever I try to execute some unittest tests of a class I have written in python. The error is 'DummyMod' object has no attribute...I don't understand this error and need some assistance to sort it out.

The class is written using Python 3.10 and relies on unittest framework for the tests. I'm running it on DB version 13.0 ML.

Data Engineering
Reply
  • 773 Views
  • 0 replies
  • 0 kudos
pedroHmdo
by • New Contributor II
  • 5696 Views
  • 2 replies
  • 3 kudos

Resolved! Why I did not receive the databricks lakehouse fundamentals accreditation badge?

I have passed the test but did not receive the Badge. I also didn't receive any email.Thank you for you attention.

Data Engineering
Reply
  • 5696 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Pedro Medeiros​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 3 kudos
1 More Replies
Neil
by • New Contributor
  • 5461 Views
  • 1 replies
  • 0 kudos

While trying to save the spark dataframe to delta table is taking too long

While working on video analytics task I need to save the image bytes to the delta table earlier extracted into the spark dataframe. While I want to over write a same delta table over the period of complete task and also the size of input data differs...

Data Engineering
Reply
  • 5461 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

can you check the spark UI, to see where the time is spent?It can be a join, udf, ...

  • 0 kudos
FRG96
by • New Contributor III
  • 5922 Views
  • 0 replies
  • 0 kudos

How to set the ABFSS URL for Azure Databricks Init Scripts that have spaces in directory names?

I want to use an Init Script on ADLS Gen2 location for my Azure Databricks 11.3 and 12.2 clusters. The init_script.sh is placed in a directory that has spaces in it:https://storageaccount1.blob.core.windows.net/container1/directory%20with%20spaces/su...

Data Engineering
Reply
  • 5922 Views
  • 0 replies
  • 0 kudos
Chinu
by • New Contributor III
  • 5464 Views
  • 1 replies
  • 1 kudos

Resolved! How to create a raw data (with filter_by) to pull query history from now to 5 mins ago

Hi Team, Is it possible I can use "query_start_time_range" filter from the api call to get the query data only from now to 5 mins ago?Im using telegraf to call query history api but it looks like Im reaching the max return and I can't find how to use...

Data Engineering
Reply
  • 5464 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
mathan_pillai
Valued Contributor
  • 1 kudos

Have you checked this https://docs.databricks.com/api-explorer/workspace/queryhistory/list you can list the queries based on time range as well. So you can try passing the fields in the filter_by parameter. Then pass the value as (current time - 5 m...

  • 1 kudos
User16783854357
by • New Contributor III
  • 1146 Views
  • 1 replies
  • 0 kudos

Delta Sharing - Who provides the server?

I would like to understand who provides the server when using Delta sharing? If a customer exposes their delta table through Delta sharing, is it the customer who needs to setup a cluster or server to process the incoming requests?

Data Engineering
Reply
  • 1146 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
BigRoux
New Contributor III
  • 0 kudos

The producer does need a cluster to set up Delta Sharing. However, once the handoff happens no cluster is needed, the data will be delivered via storage services.

  • 0 kudos
Shankar
by • New Contributor III
  • 1541 Views
  • 1 replies
  • 2 kudos

Resolved! Is there a Python API for vacuum with dry run?

I have the below sql command where i am doing a dry run with vacuum. ​%sql VACUUM <table_name> RETAIN 500 HOURS DRY RUN;wanted to check if there is a way to achieve this in python api?​I​ tried the below. But, not sure if there is a parameter that we...

Data Engineering
Reply
  • 1541 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
venkatcrc
New Contributor III
  • 2 kudos

Equivalent of sql command "VACUUM <table_name> RETAIN 500 HOURS DRY RUN;" in python is spark.sql("VACUUM <table_name> RETAIN 500 HOURS DRY RUN;")

  • 2 kudos
Retko
by • Contributor
  • 7756 Views
  • 3 replies
  • 3 kudos

Resolved! Data Tab is not showing any databases and tables even though cluster is running (Community edition)

Hi, I have a cluster running: But I dont see anything in Data Tab: As you can see it tells about some error, but error appeared after I deleted clusters which were terminated. Before it said something that cluster is not running, dont remember exactl...

image image image
Data Engineering
Reply
  • 7756 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Rajani
Contributor II
  • 3 kudos

@Retko Okter​ You need to enable DBFS File Browser from the Admin Settings hope this helps.

  • 3 kudos
2 More Replies
horatiug
by • New Contributor III
  • 719 Views
  • 0 replies
  • 0 kudos

Can the databricks_mount timeout be changed. ?

I am using terrafom to do databricks workspace configuration and while mounting 6 buckets if duration of mount is bigger than 20 min I get timeout. Is it possible to change the timeout ? thanksHoratiu

Data Engineering
Reply
  • 719 Views
  • 0 replies
  • 0 kudos
cblock
by • New Contributor III
  • 1920 Views
  • 3 replies
  • 3 kudos

Unable to run jobs with git notebooks

So, in this case our jobs are deployed from our development workspace to our isolated testing workspace via an automated Azure DevOps pipeline. As such, they are created (and thus run as) a service account user.Recently we made the switch to using gi...

Data Engineering
Reply
  • 1920 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Chris Block​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 3 kudos
2 More Replies
swatish0395
by • New Contributor III
  • 979 Views
  • 0 replies
  • 0 kudos

i am working on the parquet file level column encryption and decryption on user specific permission

i am able to encrypt and decrypt the daat in multiple ways and able to save the encrypted parquet file, but i want to decrypt the data if the user has specific permission otherwise he will get the encrypted data,.is there any permanent solution to de...

Data Engineering
Reply
  • 979 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels