- 3503 Views
- 4 replies
- 3 kudos
Migrating source directory in an existing DLT Pipeline with Autoloader
I have a DLT pipeline that reads data in S3 into an append-only bronze layer using Autoloader. The data sink needs to be changed to a new s3 bucket in a new account, and data in the existing s3 bucket migrated to the new one.Will Autoloader still be ...
- 3503 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi SamAdams,How are doing today? , Really appreciate you sharing what worked and what didn’t in your case! You're absolutely right—when switching buckets, not just folders within a bucket, that spark.databricks.cloudFiles.checkSourceChanged config be...
- 3 kudos
- 8108 Views
- 1 replies
- 0 kudos
Trying to connect SFTP directly in Databricks
Hi, As a proof of concept, I have created an ADLS and enabled SFTP> Created Local User and SSH Private key. Now, I am trying to connect this SFTP connection directly in Databricks to create table or data frame. Below is the code snipper I have used a...
- 8108 Views
- 1 replies
- 0 kudos
- 0 kudos
```scala// For Spark 3.x with Scala 2.12 (common in newer Databricks runtimes)com.springml:spark-sftp_2.12:1.1.5``` ```scalacom.springml:spark-sftp_2.11:1.1.5``` ```scalaval df = spark.read.format("com.springml.spark.sftp").option("host", "your-sft...
- 0 kudos
- 1606 Views
- 2 replies
- 0 kudos
Compute Launch Failed
Hi Team,Facing this error while we are creating a compute in DatBricksCannot launch the cluster because the user specified an invalid argument.Instance ID: failed-4d70d115-f338-43a0-9Internal error message: The VM launch request to AWS failed, please...
- 1606 Views
- 2 replies
- 0 kudos
- 0 kudos
@Renu_ I have the necessary permissions, even the admin permissions as well and moreover with the same policy in another account it works fine but it not working for specific account
- 0 kudos
- 3276 Views
- 5 replies
- 1 kudos
Resolved! Attempt to use a custom container with an instance pool fails
I am trying to run a job with (1) custom containers, and (2) via an instance pool. Here's the setup:The custom container is just the DBR-provided `databricksruntime/standard:12.2-LTS`The instance pool is defined via the UI (see screenshot, below).At ...
- 3276 Views
- 5 replies
- 1 kudos
- 1 kudos
If anyone from DBR is monitoring this thread, can y'all confirm my understanding and--if so--update the docs to reflect this requirement?
- 1 kudos
- 5202 Views
- 1 replies
- 0 kudos
Resolved! Job deploy with git source using Asset Bundles
Hi, I'm trying to deploy a job with a notebook task based on a git source. But I'm facing an error when I try to deploy.This is the YAML file:resources:jobs:data_quality_pipelines_job:name: schedule_data_quality_jobschedule:quartz_cron_expression: "0...
- 5202 Views
- 1 replies
- 0 kudos
- 0 kudos
I found the solution for this one:When we’re running a job that needs to consume files from a Git repo, in addition to declaring the git_source clause, we also need to declare the source clause within the task configuration.The image below demonstrat...
- 0 kudos
- 2517 Views
- 4 replies
- 0 kudos
Databricks Access Management
Hi, Can someone suggest me what's the least level of access we can provide to any external business user to able to run a notebook and download the results. I tried with giving read and run level permissions at notebook level and also with a user gro...
- 2517 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello @Niki_M , Just adding one more check : did you added the user group to the cluster with minimal access ? Saran
- 0 kudos
- 2217 Views
- 2 replies
- 0 kudos
Hitting 504 Error with Databricks Delta Share open sharing protocol
When trying to query a particularly large table that my team has been given access to in a share (we are querying the table using a profile file with bearer token), we are continually hitting the following error:io.delta.sharing.client.util.Unexpecte...
- 2217 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @nbrisson This could be due to large metadata in the Delta table being queried via Delta Sharing. There are some known limitations, you can refer to these docs for more details:Troubleshoot common sharing issues in Delta SharingRESOURCE_LIMIT_EXCE...
- 0 kudos
- 3112 Views
- 2 replies
- 0 kudos
Data loss with spark streaming and kafka
Hi guys!I'm facing a problem, and I have no idea where it came from. My process is not appending all the topic data into my bronze table. I checked the topic, and the data is there.For example, I have some rows that are still in my Kafka topic but do...
- 3112 Views
- 2 replies
- 0 kudos
- 0 kudos
1)To Troubleshoot: Add metrics collection to your job to track the number of records processed vs. dropped.You can check the kafka offset.Compare Kafka offsets with checkpoint committed offsets. 2) Consider modifying your code to use continuous proce...
- 0 kudos
- 3472 Views
- 2 replies
- 1 kudos
Change Data Feed And Column Masks
Hi there,Wondering if anyone can help me. I have had a job set up to stream from one change data feed enabled delta table to another delta table and has been executing successfully. I then added column masks to the source table from which I am stream...
- 3472 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi mh177,How are you doing today?, As per my understanding, it sounds like everything was working fine until you added column masks to your source table. The error you're seeing basically means that once a table has row or column-level security polic...
- 1 kudos
- 1386 Views
- 2 replies
- 1 kudos
Private package installation using DAB on job cluster
I'm using a .whl job to upload and install a package on a job cluster using DAB. However, I'm facing an issue with a private package from Azure Artifacts. It works fine when running in CI or Azure DevOps pipelines because the PAT token is available a...
- 1386 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Brahmareddy Thanks for details, could you please provide examples if possible?
- 1 kudos
- 5451 Views
- 2 replies
- 0 kudos
Switching from All-Purpose to Job Compute – How to Reuse Cluster in Parent/Child Jobs?
I’m transitioning from all-purpose clusters to job compute to optimize costs. Previously, we reused an existing_cluster_id in the job configuration to reduce total job runtime.My use case:A parent job triggers multiple child jobs sequentially.I want ...
- 5451 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi satyam-verma,How are you doing today?, As per my understanding, switching from all-purpose clusters to job compute can definitely help with cost optimization. In your case, where a parent job triggers multiple child jobs, it makes sense to want to...
- 0 kudos
- 3214 Views
- 5 replies
- 3 kudos
is there a way to find DLT job and task records in system.lakeflow.job_task_run_timeline?
Hi Guys,We're working on the monetization product and are trying to understand how much costs are coming from our jobs and DLT and all purpose interactive sessions? and are currently exploring the system.lakeflow.job_task_run_timeline table to find t...
- 3214 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi Ankit, Thanks for your follow-up. The pipeline_events table you're referring to is part of the system tables for Delta Live Tables (DLT), and if it’s not showing up in your workspace, it’s likely because system tables haven't been enabled yet in y...
- 3 kudos
- 1271 Views
- 1 replies
- 2 kudos
Events subscription in Databricks delta tables
Dear all, We are maintaining a global/enterprise data platform for a customer.We like to capture events based on data streaming happening on Databricks based delta tables. ((data streams run for at least 15 hrs a day; so, events should be generated b...
- 1271 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi noorbasha534,How are you doing today?, As per my understanding, what you're looking to build is a really powerful and smart setup—kind of like a data-driven event notification system for streaming Delta tables. While there’s no out-of-the-box feat...
- 2 kudos
- 1496 Views
- 1 replies
- 1 kudos
Auto Loader bringing NULL Records
Hi I am using auto loader to fetch some records stored in two files. Please see below my code. It fetches records from two files correctly and then it starts fetching NULL records. I attach option("cleanSource", ) to readStream. But it is ...
- 1496 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi subhas,How are you doing today?, As per my understanding, It looks like the issue is happening because you're using /FileStore, which isn’t fully supported by Autoloader’s cleanSource option. Even though the code looks mostly fine, Autoloader expe...
- 1 kudos
- 1564 Views
- 1 replies
- 2 kudos
Resolved! OPTIMIZE command on heavily nested table OOM error
I'm trying to run the OPTIMIZE command on a table with less than 2000 rows, but it is causing an out of memory issue. The problem seems to come from the fact that it is a heavily nested table in staging between a json file and flattened table. The ta...
- 1564 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi cmathieu,How are you doing today?, As per my understanding, Yeah, this sounds like one of those cases where the data volume is small, but the complexity of the schema is what’s causing the trouble. If your table has deeply nested JSON structures, ...
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
