cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Phani1
by • Valued Contributor II
  • 5870 Views
  • 8 replies
  • 8 kudos

Delta Live Table name dynamically

Hi Team,Can we pass Delta Live Table name dynamically [from a configuration file, instead of hardcoding the table name]? We would like to build a metadata-driven pipeline.

Data Engineering
Reply
  • 5870 Views
  • 8 replies
  • 8 kudos
View Replies
Latest Reply
Vic01
New Contributor II
  • 8 kudos

Hello, I wonder if there is any update for this feature?Thanks

  • 8 kudos
7 More Replies
thiagoawstest
by • Contributor
  • 396 Views
  • 0 replies
  • 0 kudos

change network/vpc workspace

Hello, I have two workspaces, each workspace pointing to a VPC in AWS, in one of the accounts we need to remove a subnet, after removing the InvalidSubnetID.NotFound AWS error when starting the clueter, checked in Manager Account, the networl is poin...

thiagoawstest_0-1720808852626.png
Data Engineering
AWS
Reply
  • 396 Views
  • 0 replies
  • 0 kudos
AWS1567
by • New Contributor III
  • 17918 Views
  • 9 replies
  • 5 kudos

We've encountered an error logging you in.

I'm trying to login for past two days and i'm still facing this error: "We've encountered an error logging you in." I've tried to reset the password multiple times and nothing happened. My friend is also not able to login. I request you to resolve t...

Databricks_login_issue
Data Engineering
Reply
  • 17918 Views
  • 9 replies
  • 5 kudos
View Replies
Latest Reply
AlxMares
New Contributor II
  • 5 kudos

I had to sign up again using the same email. In my case, the error seemed to be related to my Azure account. So, when Databricks asks you to choose your cloud provider to sync with, you should select the community version instead.

  • 5 kudos
8 More Replies
Avinash_Narala
by • Contributor
  • 337 Views
  • 0 replies
  • 0 kudos

Tracking Serverless cluster cost

Hi,I just explored serverless feature in databricks and wondering how can i track cost associated with it. Is it stored in system tables? If yes, then where can i find it?And also how can i prove that it's cost is relatively less compared to classic ...

Data Engineering
Reply
  • 337 Views
  • 0 replies
  • 0 kudos
Avinash_Narala
by • Contributor
  • 449 Views
  • 0 replies
  • 0 kudos

File Trigger VS Autoloader

Hi,I recently came across File Trigger in Databricks and find mostly similar to Autoloader. My 1st question is why file trigger as we have autoloader.In which scenarios I can go with file triggers and autoloader.Can you please differentiate?

Data Engineering
Reply
  • 449 Views
  • 0 replies
  • 0 kudos
Avinash_Narala
by • Contributor
  • 445 Views
  • 1 replies
  • 0 kudos

Custom Endpoints for AI functions In Databricks

Hi Community.Recently I gone through the AI Functions and amazed by the results.I just wanted to know whether can we use our custom endpoints(instead of databricks foundational models) and leverage this AI Functions(ai_classify, ai_mask, etc)https://...

Data Engineering
Reply
  • 445 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi Community.Recently I gone through the AI Functions and amazed by the results.I just wanted to know whether can we use our custom endpoints(instead of databricks foundational models) and leverage this AI Functions(ai_classify, ai_mask, etc)https://...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
rimaissa
by • New Contributor II
  • 462 Views
  • 1 replies
  • 0 kudos

Autoloader file notification mode error using UC

We have a DLT pipeline we've created that is using autoloader file notification mode. The pipeline ran fine before moving it to UC. Now that we're using UC, we are getting an AWS permissions issue when the autoloader file notification mode is set to ...

Data Engineering
Reply
  • 462 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
We have a DLT pipeline we've created that is using autoloader file notification mode. The pipeline ran fine before moving it to UC. Now that we're using UC, we are getting an AWS permissions issue when the autoloader file notification mode is set to ...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
FennVerm_60454
by • New Contributor II
  • 3032 Views
  • 4 replies
  • 1 kudos

Resolved! How to clean up extremely large delta log checkpoints and many small files?

AWS by the way, if that matters. We have an old production table that has been running in the background for a couple of years, always with auto-optimize and auto-compaction turned off. Since then, it has written many small files (like 10,000 an hour...

Data Engineering
Reply
  • 3032 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
siddhathPanchal
Contributor
  • 1 kudos

Sometime, if we have less commit versions for a delta table, it won't create checkpoint files in the table. Checkpoint file is responsible to trigger the log cleanup activities. In case, you observe that there are no checkpoint files available for th...

  • 1 kudos
3 More Replies
Kayla
by • Valued Contributor
  • 796 Views
  • 3 replies
  • 1 kudos

Resolved! Datadog Installation

Is anyone familiar with installing the Datadog agent on clusters? We're not having much luck.  We honestly might not be having the init script run since we're not seeing it in the log, but we can get just a generic "hellow world" init script to run a...

Data Engineering
Reply
  • 796 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Kayla
Valued Contributor
  • 1 kudos

Responding here with the solution I found. Hopefully it'll help anyone with similar issues.First, the Datadog install script is practically a matryoshka doll- the script creates another script which creates a YAML file.One of the consequences of that...

  • 1 kudos
2 More Replies
erigaud
by • Honored Contributor
  • 1373 Views
  • 4 replies
  • 0 kudos

Pass Dataframe to child job in "Run Job" task

Hello,I have a Job A that runs a Job B, and Job A defines a globalTempView and I would like to somehow access it in the child job. Is that in anyway possible ? Can the same cluster be used for both jobs ? If it is not possible, does someone know of a...

Data Engineering
Reply
  • 1373 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
rahuja
New Contributor III
  • 0 kudos

Hi @ranged_coop Yes, we are using the same job compute for using different workflows. But I think different tasks are like different docker containers so that is why it becomes an issue. It would be nice if you can explain a bit about the approach yo...

  • 0 kudos
3 More Replies
pavansharma36
by • New Contributor III
  • 812 Views
  • 4 replies
  • 1 kudos

Resolved! Databricks Workspace import api size limitation

As libraries from dbfs is deprecated and support is going to be removed. We are moving libs from dbfs to workspace files.But while uploading libraries using https://docs.databricks.com/api/azure/workspace/workspace/import api there seems to be limit ...

Data Engineering
Reply
  • 812 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
pavansharma36
New Contributor III
  • 1 kudos

Looks like volumes not supported for Standard tier.{ "error_code": "BAD_REQUEST", "message": "Unity Catalog is not available for feature tier STANDARD_TIER.", "details": [ { "@type": "type.googleapis.com/google.rpc.Err...

  • 1 kudos
3 More Replies
tf32
by • New Contributor II
  • 1000 Views
  • 2 replies
  • 1 kudos

Resolved! ERROR com.databricks.common.client.DatabricksServiceHttpClientException: DEADLINE_EXCEEDED

Hi,I got this error "com.databricks.WorkflowException: com.databricks.common.client.DatabricksServiceHttpClientException: DEADLINE_EXCEEDED" during the run of a job workflow with an interactive cluster, at the start of this. It's a job that has been ...

Data Engineering
Reply
  • 1000 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
tf32
New Contributor II
  • 1 kudos

Yes, subsequent runs have been successful.Thank you for the explanation. 

  • 1 kudos
1 More Replies
Avinash_Narala
by • Contributor
  • 976 Views
  • 2 replies
  • 2 kudos

Resolved! Databricks AI Assistant Cost Implications

I'm worried about how much the Databricks AI assistant will cost me.I need to understand what I'll be charged for, especially when I give a prompt to the AI Assistant Pane and how it will operate in the background.

Data Engineering
Reply
  • 976 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Avinash_Narala
Contributor
  • 2 kudos

Is there any token limit? like in response or the prompt we send?

  • 2 kudos
1 More Replies
brian999
by • Contributor
  • 868 Views
  • 4 replies
  • 2 kudos

Resolved! Managing libraries in workflows with multiple tasks - need to configure a list of libs for all tasks

I have workflows with multiple tasks, each of which need 5 different libraries to run. When I have to update those libraries, I have to go in and make the update in each and every task. So for one workflow I have 20 different places where I have to g...

Data Engineering
Reply
  • 868 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
brian999
Contributor
  • 2 kudos

Actually I think I found most of a solution here in one of the replies: https://community.databricks.com/t5/administration-architecture/installing-libraries-on-job-clusters/m-p/37365/highlight/true#M245It seems like I only have to define libs for the...

  • 2 kudos
3 More Replies
praful
by • New Contributor
  • 886 Views
  • 0 replies
  • 0 kudos

Recover Lost Notebook

Hi Team, I was using Databricks community edition for learning purpose. I had an account https://community.cloud.databricks.com/?o=6822095545287159 where I stored all my learning notebooks. Unfortunately, this account suddenly stopped working, and I ...

Data Engineering
Reply
  • 886 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels