- 1230 Views
- 1 replies
- 1 kudos
PGP decryption in python file
The same decryption code works in notebook, but fails in python file: import gnupgfrom pyspark.dbutils import DBUtilsdbutils = DBUtils(spark)gpg = gnupg.GPG()decryption_key = dbutils.secrets.get(secret_scope, secret_name)gpg.import_keys(decryption_ke...
- 1230 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @TetianaDromova Thank you for reaching out and waiting for a response. Having your code working on a notebook is a significant first step, so you are on the right path, but then moving into a Python file, we must consider specific details: How is...
- 1 kudos
- 283 Views
- 3 replies
- 1 kudos
Available Connectors for ServiceNow to Databricks
Hi Team,What are the available connectors to bring data and metadata from ServiceNow to Databricks, and what are the best options/best practices for integrating ServiceNow with Databricks ?Regards,Phani
- 283 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Phani1 ,There's official databricks service now connection. You can read about it at below link:Configure ServiceNow for Databricks ingestion - Azure Databricks | Microsoft LearnAnd here you have example of how to create ingestion pipeline using ...
- 1 kudos
- 207 Views
- 3 replies
- 1 kudos
- 207 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @khasim76 Good day!!Can i know what is the error msg you are getting? is it legacy community edition or free community edition? Also there are some limitation like - Table results: ≤10,000 rows or 2 MB (whichever first). - Text outputs: Trunc...
- 1 kudos
- 618 Views
- 10 replies
- 4 kudos
Resolved! append using foreach batch autoloader
Hello Guys, when i append i have this error someone knows how to fix it? raise converted from None pyspark.errors.exceptions.captured.AnalysisException: [TABLE_OR_VIEW_ALREADY_EXISTS] Cannot create table or view `s_test` because it already exists. Ch...
- 618 Views
- 10 replies
- 4 kudos
- 471 Views
- 1 replies
- 2 kudos
Resolved! Trigger table sync from job
When setting up a table sync using the UI, a pipeline is created, but this is not visible through the Pipelines overview–presumably because it's "managed" by the target table (at least this is where you manage the data ingest process.This means that ...
- 471 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Malthe The recommended method is to manage and trigger the sync via table-level APIs or management interfaces, not the pipeline-level job triggers: For Unity Catalog synced tables (e.g., syncing to Postgres), triggering a sync or refresh is per...
- 2 kudos
- 548 Views
- 1 replies
- 0 kudos
Resolved! Clarification on Default Socket Timeout Behavior Introduced in v2.6.35
Hi Team,I’m currently using the Databricks JDBC driver version 2.6.40, and I’ve noticed intermittent socket timeout errors during pipeline execution.I came across the release note for version 2.6.35 mentioning the following change:[SPARKJ-688] The co...
- 548 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Balram-snaplogi The default socket timeout value introduced in Databricks JDBC driver version 2.6.35 is 300 seconds (5 minutes). Please refer to the below kb. https://kb.databricks.com/dbsql/job-timeout-when-connecting-to-a-sql-endpoint-over-jd...
- 0 kudos
- 244 Views
- 1 replies
- 0 kudos
Resolved! Establish connectivity from databricks serverless to Cloud SQL (GCP) Postgres database
Hi team,We have setup workspace using databricks managed network (databricks is setup on GCP). There is a cloud sql db in customer managed network. We want to establish connectivity from databricks serverless to database. Could someone please tell di...
- 244 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @bhargavabasava Can you please refer to the below doc for complete details. https://docs.databricks.com/gcp/en/security/network/classic/private-service-connect
- 0 kudos
- 391 Views
- 2 replies
- 1 kudos
Resolved! unable to overwrite a table using add data in hive metastore.
I used to see the dropdown of tables under hive_metastore while using add data -> Create or modify table from file upload -> overwrite existing table but not able to from couple days. No change in permissions. I can still see this for unity catalog t...
- 391 Views
- 2 replies
- 1 kudos
- 1 kudos
hi @kkg33 Thank you for reaching out and for your patience in awaiting a response. Have you tried to query the table directly from a notebook or using the Catalog Explorer? Or is this behavior only happening on the "Upload File UI"? If you are unabl...
- 1 kudos
- 3610 Views
- 4 replies
- 1 kudos
What stuff does UCX can not do?
Hey folks! I want to know what are the limitations of UCX?, means what are the thing specially during migration we have to do manually?UCX is currently in developing mode that means it may have some drawbacks too, I want to know what are thsose?
- 3610 Views
- 4 replies
- 1 kudos
- 1 kudos
Guess you will find some differences in before and after, such as the use of the wildcard to point to folders in ADLS2 for external tables is supported in hive but not in UC catalogs.
- 1 kudos
- 4571 Views
- 6 replies
- 3 kudos
Job Compute
Hey Community, I’m new to this platform and need some guidance.I’ve set up a job on a basic compute configuration: 8GB RAM, 4 Core CPU, 1 Worker (Standard F4), with DLT runtime 16.4.8. However, my job is running slower than expected. When I checked t...
- 4571 Views
- 6 replies
- 3 kudos
- 3 kudos
Hey @Khaja_Zaffer, I’m not using any caching in my code. The cache showing up in the chart is likely from the OS page cache. What do you say?
- 3 kudos
- 3281 Views
- 2 replies
- 2 kudos
Resolved! Export Jobs Schedulation
Hello, I have to export all my notebooks from DEV to PROD.My problem is that I can't find a way to export my jobs (not the outputs, the actual notebook schedulation), is it even possible? I have hundreds of jobs to export and have to keep the same pa...
- 3281 Views
- 2 replies
- 2 kudos
- 2 kudos
You can export the complete job configuration in different ways: You can use the REST API: https://docs.databricks.com/api/azure/workspace/jobs/getYou can use the python SDK: https://github.com/databricks/databricks-sdk-pyYou can use Terraform: http...
- 2 kudos
- 613 Views
- 6 replies
- 6 kudos
Resolved! Alertsv2 Permissions
Hey team,Is it possible to set permissions on Alertsv2 using API or share it? I did look at the API docs and cannot see as option to do so. In the UI, it's possible to 'share' an alert. The issue is that I currently have SP, which is used to create a...
- 613 Views
- 6 replies
- 6 kudos
- 6 kudos
Thanks @szymon_dybczak I have asked about it in the github thread.I was able to successfully manage AlertsV2 permissions using the Workspace Permissions API with the correct object type:workspace_client.workspace.set_permissions( workspace_object...
- 6 kudos
- 287 Views
- 3 replies
- 1 kudos
Azure Databricks External location with Enable File Events using custom azure storage queue
Hi I have found that there are few issues with this feature. 1. To make it work you need to first use 'Automatic' as the value of 'File event type'. Then put the correct value of 'Resource Group' and 'Subscription id'. If you select 'Provided' as the...
- 287 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @RamenBhar I tried to implement the same, happens to me exactly what you have said. For now move towards automatic only because its recommended Also, there is workaround for this you can find the same over here. someone has posted the solution ...
- 1 kudos
- 966 Views
- 11 replies
- 6 kudos
Rehouse Postgresql DB into AWS Databricks options ?
Rehouse Postgresql DB into AWS Databricks options ?==========================================Hi Community experts?Thanks for replies to my threads.We need to relocate the Postgresql DB (data + code + on-prem scheduled jobs) into Cloud Databricks data...
- 966 Views
- 11 replies
- 6 kudos
- 6 kudos
Thanks for checking in. We did not get funding for this to move fwd. Thanks for your help.
- 6 kudos
- 287 Views
- 2 replies
- 1 kudos
How do we leverage Databricks skills into full stack?
Hi Community experts?Thanks for replies to my threads.We need to understand about full stack in the context of Data bricks. I am hearing from folks like "we doing databricks and full stack development" Is this a new framework or set of new tools?Are...
- 287 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks for weighing in.I am reviewing the Trainings Catalog | Databricks (Learning Library)Eg . Listed training:Build Data Pipelines with Lakeflow Declarative PipelinesHow/where can I get the note books showing in the session, so I can try the lab se...
- 1 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 452 | |
| 349 | |
| 287 |
