- 3723 Views
- 2 replies
- 1 kudos
Machine Type for VACUUM operation
Dear allI have a workflow with 2 tasks : one that does OPTIMIZE, followed by one that does VACUUM. I used a cluster with F32s driver and F64s - 8 workers (auto-scaling enabled). All 8 workers are launched by Databricks as soon as OPTIMIZE starts. As ...
- 3723 Views
- 2 replies
- 1 kudos
- 2416 Views
- 2 replies
- 0 kudos
How to migrate Git repos with DLT configurations
Hi!I want to migrate all my databricks related code from one github repo to another. I knew this wouldn't be straight forward. When I copy my code for one DLT, I get the errororg.apache.spark.sql.catalyst.ExtendedAnalysisException: Table 'vessel_batt...
- 2416 Views
- 2 replies
- 0 kudos
- 0 kudos
Does cloning take considerably less time then recreating the tables?Can I resume append operations to a cloned table?
- 0 kudos
- 1455 Views
- 0 replies
- 0 kudos
Job cluster configuration for 24/7
Hi Team,We intend to activate the job cluster around the clock. We consider the following parameters regarding cost: - Data volumes - Client SLA for job completion- Starting with a small cluster configuration Please advise on any other options we s...
- 1455 Views
- 0 replies
- 0 kudos
- 3640 Views
- 1 replies
- 1 kudos
Masking rules with Delta Sharing
Hi,We tried Delta sharing to PBI which worked fine, But facing issues while trying to apply row, column level filtering or data masking. It fails with error that its not supported.Can anyone please confirm, if delta sharing with masking rules works w...
- 3640 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Anshul_DBX good day! The issue you are encountering is due to a limitation in Delta Sharing. As per the provided information, Delta Sharing does not support row-level security or column masks. This means that you cannot apply row and column level...
- 1 kudos
- 5013 Views
- 1 replies
- 0 kudos
Databricks cli workflow
Is there a way that I can set up and configure a Databricks workflow job and tasks from Databricks cli or api tools by using python? Any help would be appreciated. #databricksworkflow #databricks
- 5013 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello and yes, you can set up and configure a Databricks workflow job and tasks using Databricks CLI or API tools with Python. Here are some resources and steps to guide you: Create and run Databricks Jobs: This document: ( https://docs.databrick...
- 0 kudos
- 4413 Views
- 2 replies
- 1 kudos
Address Validation, Correction and Enrichment with Databricks Spark Engine
Hi all!In our project, we're thinking about "Validation, Correction and Enrichment of Postal Addresses" with Databricks. For sure we'd need some kind of batch processing, because we have millions of addresses in our system.I'm aware of Address Valida...
- 4413 Views
- 2 replies
- 1 kudos
- 1 kudos
Happy to help. Feel free to reach out https://www.linkedin.com/in/saleh-sultan-143ab036?utm_source=share&utm_campaign=share_via&utm_content=profile&utm_medium=android_app
- 1 kudos
- 2471 Views
- 1 replies
- 0 kudos
udf in databricks
Hi Team,Is there a particular reason why we should avoid using UDF and instead convert to DataFrame code?Are there any restrictions or limitations (in terms of performance or governance) when using UDFs in Databricks? Regards,Janga
- 2471 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello some of the things you need to take in consideration is that:UDFs might introduce significant processing bottlenecks into code execution. Databricks uses a number of different optimizers automatically for code written with included Apache Spark...
- 0 kudos
- 44567 Views
- 2 replies
- 8 kudos
What's the difference between a Global view and a Temp view?
The difference between Global and Temp is how the lifetime of the view is tied to the application:http://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.createOrReplaceTempView.html?highlight=createorreplacetempview#pyspar...
- 44567 Views
- 2 replies
- 8 kudos
- 8 kudos
Correct A Temp View is scoped to the SparkSession and dropped when that session closes. Each notebook runs in its own SparkSession. The Global Temp View is scoped to the cluster and dropped when the cluster re-starts or you drop it. ---------------...
- 8 kudos
- 17727 Views
- 5 replies
- 4 kudos
Resolved! Pass date value as parameter in Databricks SQL notebook
I want to pass yesterday date (In the example 20230115*.csv) in the csv file. Don't know how to create parameter and use it here.CREATE OR REPLACE TEMPORARY VIEW abc_delivery_logUSING CSVOPTIONS ( header="true", delimiter=",", inferSchema="true", pat...
- 17727 Views
- 5 replies
- 4 kudos
- 4 kudos
@Retired_mod @sp1 @Chaitanya_Raju @daniel_sahal Hi Everyone,I need the same scenario on SQL code, because my DBR cluster not allowed me to run python codeError: Unsupported cell during execution. SQL warehouses only support executing SQL cells.I appr...
- 4 kudos
- 978 Views
- 0 replies
- 0 kudos
On the fly transformations on DLT tables
Hi, I am loading data from a kinesis data stream using DLT. CREATE STREAMING TABLE Consumers_kinesis_2 ( ..., unbase64(data) String, ... ) AS SELECT * FROM STREAM read_kinesis (...) Is it possible to directly cast, unbase64, and/or transform the resu...
- 978 Views
- 0 replies
- 0 kudos
- 19137 Views
- 9 replies
- 6 kudos
Databricks now supports event-driven workloads, especially for loading cloud files from external locations. This means you can save costs and resource...
Databricks now supports event-driven workloads, especially for loading cloud files from external locations. This means you can save costs and resources by triggering your Databricks jobs only when new files arrive in your cloud storage instead of mou...
- 19137 Views
- 9 replies
- 6 kudos
- 6 kudos
@daniel_sahal I get your point, but if for a scheduled trigger you can get all kind of attributes on the trigger time (arguably, this is available for all the triggers), then why wouldn't the most important attribute of a file event not be available ...
- 6 kudos
- 5900 Views
- 0 replies
- 0 kudos
Connecting to MuleSoft from Databricks
Hi, Is there any connectivity pipeline established already to access MuleSoft or AnyPoint exchange data using Databricks. I have seen many options to access databricks data in mulesoft but can we read the data from Mulesoft into databricks. Please gi...
- 5900 Views
- 0 replies
- 0 kudos
- 1083 Views
- 0 replies
- 0 kudos
Using AAD Spn on AWS Databricks
I use AWS Databricks which has an SSO&Scim integration with AAD. I generated an SPN in AAD, synced it to Databricks, and want to use this SPN with using AAD client secrets to use Databricks SDK. But it doesnt work. I dont want to generate another tok...
- 1083 Views
- 0 replies
- 0 kudos
- 1280 Views
- 1 replies
- 0 kudos
An error occurred when loading Jobs and Workflows App.
Hi,I was trying to open the Workflows but there is an error "An error occurred when loading Jobs and Workflows App." we need help to know why it happened and how we can resolve it please.
- 1280 Views
- 1 replies
- 0 kudos
- 0 kudos
Same...and the weirdest is that all of the services looks healthy in https://status.databricks.com/Region: eu-central-1Provider: AWSCould anyone provide some info here?
- 0 kudos
- 1401 Views
- 0 replies
- 0 kudos
Migrate a notebook that reside in workspace using Databricks Asset Bundle
Hello Community Folks -Did anyone implemented migration of notebooks that is in workspace to production databricks workspace using Databricks Asset Bundle? If so can you please help me with any documentation which I can refer? Thanks!!RegardsNiruban ...
- 1401 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
