- 2303 Views
- 2 replies
- 1 kudos
When you use Workflows to orchestrate standard notebooks, they can share a single cluster. It will be awesome if we can achieve the same for DLT pipelines orchestrated in a Workflows Job.I understand DLT pipelines utilising their own special clusters...
- 2303 Views
- 2 replies
- 1 kudos
Latest Reply
Hi @Bo Zhu Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
1 More Replies
- 2031 Views
- 1 replies
- 0 kudos
Orchestration of when DLT runs is handled by Azure Data Factory. There are scenario's a table within a DLT pipeline needs to run on a different schedule.Is there a pipeline configuration option to be set to allow the same table to be ran by two diff...
- 2031 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @Kristian Foster , It should not be possible. Every pipeline owns its table and multiple pipelines cannot write to the same table.
- 4151 Views
- 4 replies
- 0 kudos
Is there a way to limit how many retries DLT Pipelines run when in "Production" mode? What is the key value pair I use in the configuration?
- 4151 Views
- 4 replies
- 0 kudos
Latest Reply
Hi @Kristian Foster,Yes. We have two spark configuration that can be set. pipelines.numStreamRetryAttempts and pipelines.numUpdateRetryAttempts. The former configures how many times we retry each flow before failing the update. The latter configures...
3 More Replies
- 2325 Views
- 3 replies
- 3 kudos
Hey everyone! I'm building a DLT pipeline that reads files from S3 (or tries to) and then writes them into different directories in my s3 bucket. The problem is I usually access S3 with an instance profile attached to a cluster, but DLT does not give...
- 2325 Views
- 3 replies
- 3 kudos
Latest Reply
{
"clusters": [
{
"label": "default",
"aws_attributes": {
"instance_profile_arn": "arn:aws:..."
}
},
{
"label": "maintenance",
"aws_attributes": {
"instance_profile_arn": "arn:aws:..."
...
2 More Replies
