- 659 Views
- 9 replies
- 1 kudos
How do i Create a workspace object with SP ownership
Hi Team,I have a scenario that i have a jar file(24MB) to be put on workspace directory. But the ownership should be associated to the SP with any Individual ID ownership. Tried the Databricks CLI export option but it has limitation of 10 MB max.Plea...
- 659 Views
- 9 replies
- 1 kudos
- 1 kudos
Inspecting underlying HTTP traffic while using Databricks UI to import files in Workspace, it turns out (as expected) Databricks API is used, with requests similar to:So, @Naveenkumar1811 use Databricks API with SP identity in a similar way as expect...
- 1 kudos
- 308 Views
- 2 replies
- 0 kudos
"Something went wrong, please try again later." On Sync tables for PostgreSQL
I'm using the Sync feature to load up a Snowflake view from a catalog to postgreSQL (to expose data to API's for faster response times).I'm been playing around scripting the creation of the sync. And when I create + delete and recreate the same sync/...
- 308 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you for your response @stbjelcevic ,So tried to refreh the catalog and the schema when the table was deleted in Postgre + unity catalog (the sync one) and removed the pipeline:from databricks.sdk import WorkspaceClient from databricks.sdk.servi...
- 0 kudos
- 566 Views
- 4 replies
- 6 kudos
Resolved! VACUUM vs VACUUM LITE
Hey Team,I have a few questions regarding VACUUM and VACUUM LITE1. How do they work internally, do both of them scan the entire table storage directory?2. How should we use these in our prod jobs..I mean should we always run VACUUM LITE or VACUUM or ...
- 566 Views
- 4 replies
- 6 kudos
- 398 Views
- 3 replies
- 3 kudos
Resolved! List workspace permissions should return identity
Hi there,Looking through the documentation, I noticed the /api/2.0/accounts/{account_id}/workspaces/{workspace_id}/permissionassignments/permissions only returns the permissions but not the identity assigned the permission. This would be helpful for ...
- 398 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @GijsR , For “who has what” today, the most reliable alternatives are the system tables and Unity Catalog information schema views, which do include principals. You can use the information_schema to list the current grants the principals (GRANT...
- 3 kudos
- 376 Views
- 3 replies
- 2 kudos
Lakeflow Connect CDC error, broken links
I get this error, regarding database validation, when setting up a lakeflow connect CDC pipeline (see screenshot). The two links mentioned in the message are broken, they give me a "404 - Content Not Found" when I try to open them.
- 376 Views
- 3 replies
- 2 kudos
- 2 kudos
@Advika Thank you. My reason for this post was to alert the SQL server ingestion team to this bug in the interface. I will file a report about this (didn't know I could do that) and a few other issues with the feature that I've found recently.
- 2 kudos
- 1381 Views
- 4 replies
- 2 kudos
Resolved! Lakeflow Connect - Postgres connector
Hi, I was wondering when is the ETA of the LF Connector for PostgreSQL (even in public/private preview)?
- 1381 Views
- 4 replies
- 2 kudos
- 2 kudos
Ask your workspace administrator if they disabled access to it. Louis
- 2 kudos
- 485 Views
- 2 replies
- 1 kudos
Facing CANNOT_OPEN_SOCKET error after job cluster fails to upsacle to target nodes
This error pops up in my Databricks workflow 1 out of 10 times, and everytime it occurs I see the below message in event logs. Compute upsize complete, but below target size. The current worker count is 1, out of a target of 3.And right after this my...
- 485 Views
- 2 replies
- 1 kudos
- 1 kudos
@ashishCh The [CANNOT_OPEN_SOCKET] failures stem from PySpark’s default, socket‑based data transfer path used when collecting rows back to Python (e.g., .collect(), .first(), .take()), where the local handshake to a JVM‑opened ephemeral port on 127....
- 1 kudos
- 602 Views
- 4 replies
- 1 kudos
Dataframe from SQL query glitches when grouping - what is going on !?!
I have a query with some grouping. I'm using spark.sql to run that query.skus = spark.sql('with cte as (select... group by all) select *, .. from cte group by all')It displays as expected table.This table I want to split into batches for processing, ...
- 602 Views
- 4 replies
- 1 kudos
- 1 kudos
Try to use this code customized in the way you need:Instead of using monotonically_increasing_id function directly, use row_number over the previous result. This will ensure sequential "small" numbers. This was indeed the exact solution I used to sol...
- 1 kudos
- 1042 Views
- 3 replies
- 0 kudos
Moving files from Volume to Workspace
Hello Team,I am trying to move some files from volume to %shdatabricks fs cp dbfs:/Volumes/workspace/default/delc/generated_scripts/*.py Workspace/Shared/Delc_Project/scripts/ I tried all ways , Please help me to move them @DataBricks @Louis_Frolio ...
- 1042 Views
- 3 replies
- 0 kudos
- 0 kudos
@Aviraldb please try the below way,%shcp /dbfs/Volumes/workspace/default/delc/generated_scripts/*.py /Workspace/Shared/Delc_Project/scripts/ Hope it helps.
- 0 kudos
- 785 Views
- 2 replies
- 5 kudos
Resolved! What strategies have you found most effective for optimizing ETL pipelines built on the Databricks L
If you are building data pipelines in Databricks (where data is Extracted, Transformed, and Loaded), what tips, methods, or best practices do you use to make those pipelines run faster, cheaper, and more efficiently?
- 785 Views
- 2 replies
- 5 kudos
- 5 kudos
When I think about optimising ETL on the Databricks Lakehouse, I split it into four layers: data layout, Spark/SQL design, platform configuration, and operational excellence.And above all: you are not building pipelines for yourself, you are building...
- 5 kudos
- 768 Views
- 4 replies
- 2 kudos
Resolved! SkipChangeCommit to True Scenario on Data Loss Possibility
Hi Team,I have Below Scenario,I have a Spark Streaming Job with trigger of Processing time as 3 secs Running Continuously 365 days.We are performing a weekly delete job from the source of this streaming job based on custom retention policy. it is a D...
- 768 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi szymon/Raman,My Question was on the commit it performs with the insert/append via my streaming and the delete operation by the weekly maintenance Job... Is there a way that both transaction will fall into same commit. I need to understand that por...
- 2 kudos
- 230 Views
- 1 replies
- 0 kudos
In databricks custom APP how I can retrieve Genie parameters and use it in app
I have created a databricks custom APP and it is working. I need to pass parameters from Genie to the custom app. Can someone able to suggest on how I can achieve it.
- 230 Views
- 1 replies
- 0 kudos
- 0 kudos
You can pass values between a Genie space and your Databricks App using the Genie Conversation API and by adding the Genie space as an app resource: https://docs.databricks.com/aws/en/dev-tools/databricks-apps/genie Do you want the parameters to orig...
- 0 kudos
- 299 Views
- 2 replies
- 0 kudos
UDFs for working with date ranges
Hi bricklayers,Originally from a Teradata background and relatively new to Databricks, I was in need of brushing up on my Python and Github CI/CD skills so I’ve spun up a repo for a project I’m calling Terabricks.The aim is to provide a space for mak...
- 299 Views
- 2 replies
- 0 kudos
- 0 kudos
Fantastic Initiative @hobrob.I have used Teradata for good 5+ years but pre-2014/5. So I will be closely following it and very happy to contribute to it. Thanks.
- 0 kudos
- 486 Views
- 4 replies
- 3 kudos
Resolved! Using a cluster of type SINGLE_USER to run parallel python tasks in one job
Hi, I have set up a job of multiple spark python tasks running in parallel. I have only set up one job cluster, single node, data security mode SINGLE_USER, using Databricks Runtime version 14.3.x-scala2.12. These parallel spark python tasks share so...
- 486 Views
- 4 replies
- 3 kudos
- 3 kudos
@oye - The variables scope is local to the individual task and do no interfere with other tasks even if the underlying cluster is same. In fact, the issue is normally other way round where if we have to share the variable across tasks - Then the solu...
- 3 kudos
- 1199 Views
- 2 replies
- 0 kudos
Repairing job is useful but disabled if the job is triggered by another Run job task
Hi, I regularly use the repair job option to rerun a subset of a larger data pipeline job. However, this option is disabled if I start this job from another "orchestration job" via Job Run tasks. The "repair" button is disabled in my case. This restr...
- 1199 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @carolsun08 How are you doing today?, Yeah, it's a bit frustrating that the repair job option is disabled when running jobs through Job Run tasks in an orchestration job. While there's no official confirmation on when this might change, it would d...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 1050 | |
| 791 | |
| 553 | |
| 349 |
