- 2830 Views
- 1 replies
- 2 kudos
Resolved! Trigger table sync from job
When setting up a table sync using the UI, a pipeline is created, but this is not visible through the Pipelines overview–presumably because it's "managed" by the target table (at least this is where you manage the data ingest process.This means that ...
- 2830 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Malthe The recommended method is to manage and trigger the sync via table-level APIs or management interfaces, not the pipeline-level job triggers: For Unity Catalog synced tables (e.g., syncing to Postgres), triggering a sync or refresh is per...
- 2 kudos
- 1977 Views
- 1 replies
- 0 kudos
Resolved! Clarification on Default Socket Timeout Behavior Introduced in v2.6.35
Hi Team,I’m currently using the Databricks JDBC driver version 2.6.40, and I’ve noticed intermittent socket timeout errors during pipeline execution.I came across the release note for version 2.6.35 mentioning the following change:[SPARKJ-688] The co...
- 1977 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Balram-snaplogi The default socket timeout value introduced in Databricks JDBC driver version 2.6.35 is 300 seconds (5 minutes). Please refer to the below kb. https://kb.databricks.com/dbsql/job-timeout-when-connecting-to-a-sql-endpoint-over-jd...
- 0 kudos
- 733 Views
- 1 replies
- 0 kudos
Resolved! Establish connectivity from databricks serverless to Cloud SQL (GCP) Postgres database
Hi team,We have setup workspace using databricks managed network (databricks is setup on GCP). There is a cloud sql db in customer managed network. We want to establish connectivity from databricks serverless to database. Could someone please tell di...
- 733 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @bhargavabasava Can you please refer to the below doc for complete details. https://docs.databricks.com/gcp/en/security/network/classic/private-service-connect
- 0 kudos
- 1057 Views
- 2 replies
- 1 kudos
Resolved! unable to overwrite a table using add data in hive metastore.
I used to see the dropdown of tables under hive_metastore while using add data -> Create or modify table from file upload -> overwrite existing table but not able to from couple days. No change in permissions. I can still see this for unity catalog t...
- 1057 Views
- 2 replies
- 1 kudos
- 1 kudos
hi @kkg33 Thank you for reaching out and for your patience in awaiting a response. Have you tried to query the table directly from a notebook or using the Catalog Explorer? Or is this behavior only happening on the "Upload File UI"? If you are unabl...
- 1 kudos
- 5203 Views
- 4 replies
- 1 kudos
What stuff does UCX can not do?
Hey folks! I want to know what are the limitations of UCX?, means what are the thing specially during migration we have to do manually?UCX is currently in developing mode that means it may have some drawbacks too, I want to know what are thsose?
- 5203 Views
- 4 replies
- 1 kudos
- 1 kudos
Guess you will find some differences in before and after, such as the use of the wildcard to point to folders in ADLS2 for external tables is supported in hive but not in UC catalogs.
- 1 kudos
- 9603 Views
- 6 replies
- 3 kudos
Job Compute
Hey Community, I’m new to this platform and need some guidance.I’ve set up a job on a basic compute configuration: 8GB RAM, 4 Core CPU, 1 Worker (Standard F4), with DLT runtime 16.4.8. However, my job is running slower than expected. When I checked t...
- 9603 Views
- 6 replies
- 3 kudos
- 3 kudos
Hey @Khaja_Zaffer, I’m not using any caching in my code. The cache showing up in the chart is likely from the OS page cache. What do you say?
- 3 kudos
- 4363 Views
- 2 replies
- 2 kudos
Resolved! Export Jobs Schedulation
Hello, I have to export all my notebooks from DEV to PROD.My problem is that I can't find a way to export my jobs (not the outputs, the actual notebook schedulation), is it even possible? I have hundreds of jobs to export and have to keep the same pa...
- 4363 Views
- 2 replies
- 2 kudos
- 2 kudos
You can export the complete job configuration in different ways: You can use the REST API: https://docs.databricks.com/api/azure/workspace/jobs/getYou can use the python SDK: https://github.com/databricks/databricks-sdk-pyYou can use Terraform: http...
- 2 kudos
- 2168 Views
- 6 replies
- 6 kudos
Resolved! Alertsv2 Permissions
Hey team,Is it possible to set permissions on Alertsv2 using API or share it? I did look at the API docs and cannot see as option to do so. In the UI, it's possible to 'share' an alert. The issue is that I currently have SP, which is used to create a...
- 2168 Views
- 6 replies
- 6 kudos
- 6 kudos
Thanks @szymon_dybczak I have asked about it in the github thread.I was able to successfully manage AlertsV2 permissions using the Workspace Permissions API with the correct object type:workspace_client.workspace.set_permissions( workspace_object...
- 6 kudos
- 1075 Views
- 3 replies
- 1 kudos
Azure Databricks External location with Enable File Events using custom azure storage queue
Hi I have found that there are few issues with this feature. 1. To make it work you need to first use 'Automatic' as the value of 'File event type'. Then put the correct value of 'Resource Group' and 'Subscription id'. If you select 'Provided' as the...
- 1075 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @RamenBhar I tried to implement the same, happens to me exactly what you have said. For now move towards automatic only because its recommended Also, there is workaround for this you can find the same over here. someone has posted the solution ...
- 1 kudos
- 2436 Views
- 11 replies
- 6 kudos
Rehouse Postgresql DB into AWS Databricks options ?
Rehouse Postgresql DB into AWS Databricks options ?==========================================Hi Community experts?Thanks for replies to my threads.We need to relocate the Postgresql DB (data + code + on-prem scheduled jobs) into Cloud Databricks data...
- 2436 Views
- 11 replies
- 6 kudos
- 6 kudos
Thanks for checking in. We did not get funding for this to move fwd. Thanks for your help.
- 6 kudos
- 1556 Views
- 2 replies
- 1 kudos
How do we leverage Databricks skills into full stack?
Hi Community experts?Thanks for replies to my threads.We need to understand about full stack in the context of Data bricks. I am hearing from folks like "we doing databricks and full stack development" Is this a new framework or set of new tools?Are...
- 1556 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks for weighing in.I am reviewing the Trainings Catalog | Databricks (Learning Library)Eg . Listed training:Build Data Pipelines with Lakeflow Declarative PipelinesHow/where can I get the note books showing in the session, so I can try the lab se...
- 1 kudos
- 1342 Views
- 1 replies
- 1 kudos
Resolved! How do I efficiently manage common and environment-specific job parameters in DABs
I have a scenario where my Databricks asset bundles require two types of job parameters:Common parameters that apply to all environmentsEnvironment-specific parameters that differ per environment.My current YAML setup is structured like this:# common...
- 1342 Views
- 1 replies
- 1 kudos
- 1 kudos
To merge common parameters with environment-specific parameters in Databricks Asset Bundle (DAB) YAMLs, the most effective approach is to adopt modular YAML files, leveraging the include directive and hierarchical overrides within the targets mapping...
- 1 kudos
- 1203 Views
- 3 replies
- 1 kudos
Error MISSING_EXPECTED_PROPERTY during migration to default publishing mode
We are in the process of migrating a pipeline from legacy publishing mode (LIVE virtual schema) to default publishing mode, following the steps outlined in the documentation:Enable the default publishing mode in a pipeline (Public Preview)However, wh...
- 1203 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @minhhung0507 ,According to troubleshooting guide the "MISSING_EXPECTED_PROPERTY" error indicates that you did not run a recent update prior to adding the pipelines.enableDPMForExistingPipeline configuration. Are you sure that your update was succ...
- 1 kudos
- 1283 Views
- 1 replies
- 2 kudos
Resolved! Can Databricks federation policy support cross-cloud authentication?
Hi!I'm exploring options for workload identity federation and have a question about cross-cloud scenarios.Current Setup:Azure Databricks workspaceWorkloads running in GCP (planning to use GKE/Kubernetes)Need to authenticate GCP-based workloads to Azu...
- 1283 Views
- 1 replies
- 2 kudos
- 2 kudos
Yes, Databricks federation policy can support cross-cloud authentication, allowing the use of external identity providers (IdPs) that may reside in different clouds. This includes scenarios where tokens issued by trusted IdPs—such as those for servic...
- 2 kudos
- 1648 Views
- 5 replies
- 0 kudos
GEOMETRY column type breaks all access to table from Spark
Hello, I have a Databricks table with a column using the new GEOMETRY type. When I try to access this table from a Spark workload, I am not able to describe the table or operate on any of its columns. My Spark config is the following, per the Databri...
- 1648 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @NandiniN,To my understanding, there is no way to use the Databricks SDK as a catalog implementation in an external Spark workload, correct?The table I'm using is a Delta table I created from a Databricks notebook. The rows represent some areas-of...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 573 | |
| 349 | |
| 287 |
