cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Retko
by • Contributor
  • 8868 Views
  • 3 replies
  • 3 kudos

Resolved! Data Tab is not showing any databases and tables even though cluster is running (Community edition)

Hi, I have a cluster running: But I dont see anything in Data Tab: As you can see it tells about some error, but error appeared after I deleted clusters which were terminated. Before it said something that cluster is not running, dont remember exactl...

image image image
Data Engineering
Reply
  • 8868 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Rajani
Contributor II
  • 3 kudos

@Retko Okter​ You need to enable DBFS File Browser from the Admin Settings hope this helps.

  • 3 kudos
2 More Replies
horatiug
by • New Contributor III
  • 855 Views
  • 0 replies
  • 0 kudos

Can the databricks_mount timeout be changed. ?

I am using terrafom to do databricks workspace configuration and while mounting 6 buckets if duration of mount is bigger than 20 min I get timeout. Is it possible to change the timeout ? thanksHoratiu

Data Engineering
Reply
  • 855 Views
  • 0 replies
  • 0 kudos
cblock
by • New Contributor III
  • 2151 Views
  • 3 replies
  • 3 kudos

Unable to run jobs with git notebooks

So, in this case our jobs are deployed from our development workspace to our isolated testing workspace via an automated Azure DevOps pipeline. As such, they are created (and thus run as) a service account user.Recently we made the switch to using gi...

Data Engineering
Reply
  • 2151 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Chris Block​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 3 kudos
2 More Replies
swatish0395
by • New Contributor III
  • 1062 Views
  • 0 replies
  • 0 kudos

i am working on the parquet file level column encryption and decryption on user specific permission

i am able to encrypt and decrypt the daat in multiple ways and able to save the encrypted parquet file, but i want to decrypt the data if the user has specific permission otherwise he will get the encrypted data,.is there any permanent solution to de...

Data Engineering
Reply
  • 1062 Views
  • 0 replies
  • 0 kudos
grazie
by • Contributor
  • 2349 Views
  • 2 replies
  • 2 kudos

how to get dbutils in Runtime 13

We're using the following method (generated by using dbx) to access dbutils, e.g. to retrieve parameters from secret scopes: @staticmethod def _get_dbutils(spark: SparkSession) -> "dbutils": try: from pyspark.dbutils import...

Data Engineering
Reply
  • 2349 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
colt
New Contributor III
  • 2 kudos

We have something similar in our code. This worked using runtime 13 until last week. Also the Machine Learning DBR doesn't work either.

  • 2 kudos
1 More Replies
96286
by • Contributor
  • 6459 Views
  • 4 replies
  • 3 kudos

Resolved! Autoloader works on compute cluster, but does not work within a task in workflows

I feel like I am going crazy with this. I have tested a data pipeline on my standard compute cluster. I am loading new files as batch from a Google Cloud Storage bucket. Autoloader works exactly as expected from my notebook on my compute cluster. The...

Data Engineering
Reply
  • 6459 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
96286
Contributor
  • 3 kudos

I found the issue. I describe the solution in the following SO post. https://stackoverflow.com/questions/76287095/databricks-autoloader-works-on-compute-cluster-but-does-not-work-within-a-task/76313794#76313794

  • 3 kudos
3 More Replies
g96g
by • New Contributor III
  • 1009 Views
  • 1 replies
  • 0 kudos

Function in databricks

Im having a hard time to convert below function from SSMS to databricks function. Any help would be appreciated! CREATE FUNCTION [dbo].[MaxOf5Values] (@D1 [int],@D2 [int],@D3 [int],@D4 [int],@D5 [int]) RETURNS int AS BEGIN DECLARE @Result int   ...

Data Engineering
Reply
  • 1009 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 0 kudos

Hi @Givi Salu​ ,​Please refer to this link that will help you convert this function.

  • 0 kudos
HappySK
by • New Contributor
  • 5447 Views
  • 1 replies
  • 0 kudos

Execute COPY INTO query with source as temp view

We are having the files that needs to be loaded into the delta tableNow we want to perform some transformation on the files and load that into the tableWhat we didCreate a Spark DF from that fileApply transformation on the DFCreate a temp view from t...

Data Engineering
Reply
  • 5447 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 0 kudos

Hi @Sravan Kumar Mohanraj​ ,Yes, you can use copy query in this case your temp_view will be source.For more info, please visit these links.

  • 0 kudos
Ismail1
by • New Contributor III
  • 2424 Views
  • 2 replies
  • 3 kudos

Resolved! API Authentication

I am trying to run some API calls to the account console. I tried with every syntax possible encoded and decoded to get the call successfully but it returns a "user not authenticated" error. But when I tried with the Account admin it worked. I need t...

Data Engineering
Reply
  • 2424 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Ismail1
New Contributor III
  • 3 kudos

Hi Venkat, that sounds like a good idea. Thanks

  • 3 kudos
1 More Replies
Sujitha
by • Databricks Employee
  • 578 Views
  • 0 replies
  • 1 kudos

�� Welcome to Chennai User Group Community! �� We are thrilled to have you join our vibrant and enthusiastic community of users! Whether...

Welcome to Chennai User Group Community! We are thrilled to have you join our vibrant and enthusiastic community of users! Whether you are a seasoned expert or a newcomer, this group is the perfect place for you to connect, learn, and grow alongside...

Data Engineering
Reply
  • 578 Views
  • 0 replies
  • 1 kudos
swatish0395
by • New Contributor III
  • 3203 Views
  • 3 replies
  • 4 kudos

Resolved! how to create a scala jar using db notebook and save it in a file path inside databricks

I have scala function as below, i am unable to understand how to write a scala jar with the same, please find below code i have used Enforcing Column-Level Encryption - Databrick %scala import com.macasaet.fernet.{Key, StringValidator, Token}import o...

Data Engineering
Reply
  • 3203 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
swatish0395
New Contributor III
  • 4 kudos

I had to finally create the jar using teh intellij and sbt iconfiguration on the same env. and then installed the jar in the cluster it worked

  • 4 kudos
2 More Replies
Bhawna_bedi
by • New Contributor II
  • 5243 Views
  • 5 replies
  • 5 kudos

Resolved! I want to run a streaming job from morning 6a.m to evening 5p.m how can I schedule this window in databricks. Or how can u stop my stream at 5pm?

Data Engineering
Reply
  • 5243 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
merca
Valued Contributor II
  • 5 kudos

If you are streaming to delta, not much, the micro batch will fail and in next time the stream will pick up from last successful write (due to ACID). I don't know about other formats, what happens if the stream is aborted in mid micro batch.

  • 5 kudos
4 More Replies
Prashant777
by • New Contributor II
  • 2671 Views
  • 2 replies
  • 0 kudos

UnsupportedOperationException: Cannot perform Merge as multiple source rows matched and attempted to modify the same

My Code:-- CREATE OR REPLACE TEMPORARY VIEW preprocessed_source ASSELECT  Key_ID,  Distributor_ID,  Customer_ID,  Customer_Name,  ChannelFROM integr_masterdata.Customer_Master;-- Step 2: Perform the merge operation using the preprocessed source table...

Data Engineering
Reply
  • 2671 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Prashant Joshi​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...

  • 0 kudos
1 More Replies
THIAM_HUATTAN
by • Valued Contributor
  • 3676 Views
  • 6 replies
  • 6 kudos

Resolved! Delta Lake’s CDF Feature

https://www.databricks.com/notebooks/delta-lake-cdf.htmlI am trying to understand the above article. Could someone explain to be the below questions?a) From SELECT * FROM table_changes('gold_consensus_eps', 2)why is consensus_eps values of 2.1 and 2....

Data Engineering
Reply
  • 3676 Views
  • 6 replies
  • 6 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @THIAM HUAT TAN​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 6 kudos
5 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels