cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
marchino
by • New Contributor II
  • 2913 Views
  • 4 replies
  • 1 kudos

Can I change Service Principal's OAuth token's expiration date?

Hi,since I have to read from a Databricks table from an external API I created a Service Principal that would start a cluster and perform the operation, to authenticate the request on behalf of the Service Principal I generate the OAuth token followi...

Community Discussions
Reply
  • 2913 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
NandiniN
Honored Contributor
  • 1 kudos

Hello @marchino Please check if this is of your interest https://kb.databricks.com/en_US/security/set-an-unlimited-lifetime-for-service-principal-access-token 

  • 1 kudos
3 More Replies
Henrik
by • New Contributor III
  • 1540 Views
  • 3 replies
  • 1 kudos

Data lineage on views

I do not know if this is intended behavior of data lineage but for me it is weird.When I create a view based on two tables the data lineage upstream looks correct. But when I replace the view to only use one of the tables, then data lineage upstream ...

Community Discussions
Reply
  • 1540 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Henrik
New Contributor III
  • 1 kudos

After some thoughts, i have come to this conclusion:Data lineage on views is working as one should expect. I strongly recommend that this feature is redesigned so it shows the result of the lastest view.

  • 1 kudos
2 More Replies
Chalki
by • New Contributor III
  • 3578 Views
  • 3 replies
  • 0 kudos

Iterative read and writes cause java.lang.OutOfMemoryError: GC overhead limit exceeded

I have an iterative algorithm which read and writes a dataframe iteration trough a list with new partitions, like this: for p in partitions_list:df = spark.read.parquet("adls_storage/p")df.write.format("delta").mode("overwrite").option("partitionOver...

Community Discussions
Reply
  • 3578 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Chalki
New Contributor III
  • 0 kudos

@daniel_sahalI've attached the wrong snip/ Actually it is FULL GC Ergonomics, which was bothering me. Now I am attaching the correct snip.  But as you said I scaled a bit. The thing I forgot to mention is that the table is wide - more than 300 column...

  • 0 kudos
2 More Replies
Dekova
by • New Contributor II
  • 1460 Views
  • 1 replies
  • 3 kudos

Resolved! Using DeltaTable.merge() and generating surrogate keys on insert?

I'm using merge to upsert data into a table:DeltaTable.forName(DESTINATION_TABLE).as("target").merge(merge_df.as("source") ,"source.topic = target.topic and source.key = target.key").whenMatched().updateAll().whenNotMatched().insertAll().execute()Id ...

Community Discussions
Reply
  • 1460 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
daniel_sahal
Esteemed Contributor
  • 3 kudos

@Dekova 1) uuid() is non-deterministic meaning that it will give you different result each time you run this function2) Per the documentation "For Databricks Runtime 9.1 and above, MERGE operations support generated columns when you set spark.databri...

  • 3 kudos
Phani1
by • Valued Contributor
  • 2817 Views
  • 4 replies
  • 1 kudos

Databricks Job Failure + Service now Integration

Hi Team,Could you please suggest how to raise the service now ticket, in case of Databricks job failure?Regards ,Phanindra

Community Discussions
Reply
  • 2817 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Swastik_Mishra
New Contributor II
  • 1 kudos

Hi @Phani1, You can use the webhook method to integrate Databricks job failure notifications with ServiceNow. This allows Databricks to send an HTTP POST request (webhook) to a designated endpoint in ServiceNow whenever a job fails. By doing so, you ...

  • 1 kudos
3 More Replies
kurtrm
by • New Contributor III
  • 2087 Views
  • 4 replies
  • 0 kudos

Import dbfs file into workspace using Python SDK

Hello,I am looking to replicate the functionality provided by the databricks_cli Python package using the Python SDK. Previously, using the databricks_cli WorkspaceApi object, I could use the import_workspace or import_workspace_dir methods to move a...

Community Discussions
Reply
  • 2087 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Kratik
New Contributor III
  • 0 kudos

Even, I am looking for a way to bring files present in S3 to Workspace programmatically. 

  • 0 kudos
3 More Replies
alesventus
by • New Contributor III
  • 377 Views
  • 0 replies
  • 0 kudos

Big time differences in reading tables

When I read managed table in #databricks# i can see big differences in time spent. Small test table with just 2 records is once loaded in 3 seconds and another time in 30 seconds. Reading table_change for this tinny table took 15 minutes. Don't know ...

Community Discussions
performance issue
Reply
  • 377 Views
  • 0 replies
  • 0 kudos
yzhang
by • New Contributor III
  • 1288 Views
  • 2 replies
  • 4 kudos

Resolved! Is there a plan to support workflow jobs to be stored in a subfolder?

I have many workflow jobs created and they all in a flat list. Is there a way to create (kind of) sub folders that I can category my databricks workflow jobs into it (kind of organizer)...

Community Discussions
Reply
  • 1288 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
yzhang
New Contributor III
  • 4 kudos

@Anonymous thanks for the suggestion. And thanks @Vinay_M_R a lot for answering the question. The solution mentioned is doable but less optimized way to do. Everyone in the team has to follow the same rules especially for shared jobs, and sometimes n...

  • 4 kudos
1 More Replies
GrahamBricks
by • New Contributor
  • 1490 Views
  • 0 replies
  • 0 kudos

terraform jobs depends_on

I am attempting to automate Jobs creation using Databrick Terraform provider.    I have a number of task that will "depends_on" each other and am trying to use dynamic content to do this.  Each task name is stored in a string array so looping over th...

Community Discussions
Reply
  • 1490 Views
  • 0 replies
  • 0 kudos
CraiMacl_23588
by • New Contributor
  • 346 Views
  • 0 replies
  • 0 kudos

Init scripts in legacy workspace (pre-E2)

Hello,I've got a legacy workspace (not E2) and I am trying to move my cluster scoped init script to the workspace area (from DBFS). It doesn't seem to be possible to store a shell script in the workspace area (Accepted formats: .dbc, .scala, .py, .sq...

Community Discussions
Reply
  • 346 Views
  • 0 replies
  • 0 kudos
Ninad
by • New Contributor
  • 935 Views
  • 3 replies
  • 1 kudos

Databricks on AWS

I want to host databricks on AWS. I want to know if we create databricks on top of AWS, will it be created in same account's VPC or will it be created out of my AWS account?If it is going to be created in my account, will it create a new VPC for me?T...

Community Discussions
Reply
  • 935 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Siebert_Looije
Contributor
  • 1 kudos

Hi, If you want to know more about the how to properly setup the databricks on top of AWS. I would really recommend to do the AWS platform administrator course of Databricks. In this everything is explained what you need to know. Hopes this helps.Kin...

  • 1 kudos
2 More Replies
zbowden2010
by • New Contributor II
  • 1043 Views
  • 1 replies
  • 0 kudos

503 Error from Databricks when Cluster Inactive/Starting Up via Alteryx

Hello,I have been connecting to Databricks via Alteryx. It works fine when our cluster is active, but returns a 503 Service Unavailable error if the Cluster is inactive/starting up. I have previously reached out to Alteryx, but they have told me this...

Community Discussions
Reply
  • 1043 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
zbowden2010
New Contributor II
  • 0 kudos

I should have mentioned in the original post, we are using Microsoft Azure and a Simba Spark ODBC Driver.

  • 0 kudos
rohit-1989
by • New Contributor
  • 440 Views
  • 0 replies
  • 0 kudos

How to access ADLS Gen2 hdfs from a databricks cluster which has credential passthrough enabled?

When executing through a Databricks cluster with credential passthrough enabled, I wish to obtain supplementary file attributes in ADLS, such as the file's last modified time, which are currently unavailable in the databricks dbutils.fs.ls function.W...

Community Discussions
credential-passthrough
Databricks
Reply
  • 440 Views
  • 0 replies
  • 0 kudos
KVNARK
by • Honored Contributor II
  • 2018 Views
  • 1 replies
  • 1 kudos

No points shown in databricks new community page

There are no points displayed in Databricks new community page. Is it the same for all or only for me if I have done something wrong.

Community Discussions
Reply
  • 2018 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
KaKa
Contributor
  • 1 kudos

same concerns with you. on my account also didnot find where place display how many point in my account 

  • 1 kudos
Obulreddy
by • New Contributor
  • 2122 Views
  • 3 replies
  • 1 kudos

Unable to access S3 objects from Databricks using IAM access keys in both AWS and Azure Databricks

Hi Team,We are trying to connect to Amazon S3 bucket from both Databricks running on AWS and Azure using IAM access keys directly through Scala code in Notebook and we are facing com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; with stat...

Community Discussions
Reply
  • 2122 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Obulreddy  We haven't heard from you since the last response from @KaKa ​, and I was checking back to see if her suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful to others.  Also,...

  • 1 kudos
2 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors
View all