- 5214 Views
- 4 replies
- 2 kudos
Optimizing Spark Read Performance on Delta Tables with Deletion Vectors Enabled
Hi Databricks Experts,I'm currently using Delta Live Table to generate master data managed within Unity Catalog, with the data stored directly in Google Cloud Storage. I then utilize Spark to read these master data from the GCS bucket. However, I’m ...
- 5214 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Louis_Frolio , thanks for your explaination.In case we can't optimize spark locally as fast as databicks. Do you have any suggestion for us to optimize performance in this scenario?
- 2 kudos
- 8546 Views
- 2 replies
- 1 kudos
Resolved! Incorrect length for `string` returned by the Databricks ODBC driver
Dear Databricks and community,I have been struggling with a bug related to using golang and the Databricks ODBC driver.It turns out that `SQLDescribeColW` consequently returns 256 as a length for `string` columns. However, in Spark, strings might b...
- 8546 Views
- 2 replies
- 1 kudos
- 1 kudos
Is there a permanent solution for this? We are still seeing this issue and it's been years.
- 1 kudos
- 2904 Views
- 5 replies
- 1 kudos
not able to create table from pyspark sql using databricks unity catalog open apis
I was trying to access databricks and do DDL/DML operations using databricks unity catalog open apis. The create schema and select tables are working, but create table is not working due to below issues, could you please help?I was using pyspark sql ...
- 2904 Views
- 5 replies
- 1 kudos
- 1 kudos
Hello @dixonantony Can you try running this command? spark.sql("create table datatest.dischema.demoTab1(id int, name VARCHAR(10), age int)") Ensure that you have the necessary permissions to create tables in Unity Catalog. You need the CREATE TABLE ...
- 1 kudos
- 3224 Views
- 4 replies
- 3 kudos
Migrating source directory in an existing DLT Pipeline with Autoloader
I have a DLT pipeline that reads data in S3 into an append-only bronze layer using Autoloader. The data sink needs to be changed to a new s3 bucket in a new account, and data in the existing s3 bucket migrated to the new one.Will Autoloader still be ...
- 3224 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi SamAdams,How are doing today? , Really appreciate you sharing what worked and what didn’t in your case! You're absolutely right—when switching buckets, not just folders within a bucket, that spark.databricks.cloudFiles.checkSourceChanged config be...
- 3 kudos
- 7632 Views
- 1 replies
- 0 kudos
Trying to connect SFTP directly in Databricks
Hi, As a proof of concept, I have created an ADLS and enabled SFTP> Created Local User and SSH Private key. Now, I am trying to connect this SFTP connection directly in Databricks to create table or data frame. Below is the code snipper I have used a...
- 7632 Views
- 1 replies
- 0 kudos
- 0 kudos
```scala// For Spark 3.x with Scala 2.12 (common in newer Databricks runtimes)com.springml:spark-sftp_2.12:1.1.5``` ```scalacom.springml:spark-sftp_2.11:1.1.5``` ```scalaval df = spark.read.format("com.springml.spark.sftp").option("host", "your-sft...
- 0 kudos
- 1473 Views
- 2 replies
- 0 kudos
Compute Launch Failed
Hi Team,Facing this error while we are creating a compute in DatBricksCannot launch the cluster because the user specified an invalid argument.Instance ID: failed-4d70d115-f338-43a0-9Internal error message: The VM launch request to AWS failed, please...
- 1473 Views
- 2 replies
- 0 kudos
- 0 kudos
@Renu_ I have the necessary permissions, even the admin permissions as well and moreover with the same policy in another account it works fine but it not working for specific account
- 0 kudos
- 3049 Views
- 5 replies
- 1 kudos
Resolved! Attempt to use a custom container with an instance pool fails
I am trying to run a job with (1) custom containers, and (2) via an instance pool. Here's the setup:The custom container is just the DBR-provided `databricksruntime/standard:12.2-LTS`The instance pool is defined via the UI (see screenshot, below).At ...
- 3049 Views
- 5 replies
- 1 kudos
- 1 kudos
If anyone from DBR is monitoring this thread, can y'all confirm my understanding and--if so--update the docs to reflect this requirement?
- 1 kudos
- 5041 Views
- 1 replies
- 0 kudos
Resolved! Job deploy with git source using Asset Bundles
Hi, I'm trying to deploy a job with a notebook task based on a git source. But I'm facing an error when I try to deploy.This is the YAML file:resources:jobs:data_quality_pipelines_job:name: schedule_data_quality_jobschedule:quartz_cron_expression: "0...
- 5041 Views
- 1 replies
- 0 kudos
- 0 kudos
I found the solution for this one:When we’re running a job that needs to consume files from a Git repo, in addition to declaring the git_source clause, we also need to declare the source clause within the task configuration.The image below demonstrat...
- 0 kudos
- 2342 Views
- 4 replies
- 0 kudos
Databricks Access Management
Hi, Can someone suggest me what's the least level of access we can provide to any external business user to able to run a notebook and download the results. I tried with giving read and run level permissions at notebook level and also with a user gro...
- 2342 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello @Niki_M , Just adding one more check : did you added the user group to the cluster with minimal access ? Saran
- 0 kudos
- 2076 Views
- 2 replies
- 0 kudos
Hitting 504 Error with Databricks Delta Share open sharing protocol
When trying to query a particularly large table that my team has been given access to in a share (we are querying the table using a profile file with bearer token), we are continually hitting the following error:io.delta.sharing.client.util.Unexpecte...
- 2076 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @nbrisson This could be due to large metadata in the Delta table being queried via Delta Sharing. There are some known limitations, you can refer to these docs for more details:Troubleshoot common sharing issues in Delta SharingRESOURCE_LIMIT_EXCE...
- 0 kudos
- 2975 Views
- 2 replies
- 0 kudos
Data loss with spark streaming and kafka
Hi guys!I'm facing a problem, and I have no idea where it came from. My process is not appending all the topic data into my bronze table. I checked the topic, and the data is there.For example, I have some rows that are still in my Kafka topic but do...
- 2975 Views
- 2 replies
- 0 kudos
- 0 kudos
1)To Troubleshoot: Add metrics collection to your job to track the number of records processed vs. dropped.You can check the kafka offset.Compare Kafka offsets with checkpoint committed offsets. 2) Consider modifying your code to use continuous proce...
- 0 kudos
- 3308 Views
- 2 replies
- 1 kudos
Change Data Feed And Column Masks
Hi there,Wondering if anyone can help me. I have had a job set up to stream from one change data feed enabled delta table to another delta table and has been executing successfully. I then added column masks to the source table from which I am stream...
- 3308 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi mh177,How are you doing today?, As per my understanding, it sounds like everything was working fine until you added column masks to your source table. The error you're seeing basically means that once a table has row or column-level security polic...
- 1 kudos
- 1297 Views
- 2 replies
- 1 kudos
Private package installation using DAB on job cluster
I'm using a .whl job to upload and install a package on a job cluster using DAB. However, I'm facing an issue with a private package from Azure Artifacts. It works fine when running in CI or Azure DevOps pipelines because the PAT token is available a...
- 1297 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Brahmareddy Thanks for details, could you please provide examples if possible?
- 1 kudos
- 5103 Views
- 2 replies
- 0 kudos
Switching from All-Purpose to Job Compute – How to Reuse Cluster in Parent/Child Jobs?
I’m transitioning from all-purpose clusters to job compute to optimize costs. Previously, we reused an existing_cluster_id in the job configuration to reduce total job runtime.My use case:A parent job triggers multiple child jobs sequentially.I want ...
- 5103 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi satyam-verma,How are you doing today?, As per my understanding, switching from all-purpose clusters to job compute can definitely help with cost optimization. In your case, where a parent job triggers multiple child jobs, it makes sense to want to...
- 0 kudos
- 2985 Views
- 5 replies
- 3 kudos
is there a way to find DLT job and task records in system.lakeflow.job_task_run_timeline?
Hi Guys,We're working on the monetization product and are trying to understand how much costs are coming from our jobs and DLT and all purpose interactive sessions? and are currently exploring the system.lakeflow.job_task_run_timeline table to find t...
- 2985 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi Ankit, Thanks for your follow-up. The pipeline_events table you're referring to is part of the system tables for Delta Live Tables (DLT), and if it’s not showing up in your workspace, it’s likely because system tables haven't been enabled yet in y...
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
