- 4890 Views
- 4 replies
- 2 kudos
Resolved! Executing Bash Scripts or Binaries Directly in Databricks Jobs on Single Node Cluster
Hi,Is it possible to directly execute a Bash script or a binary executable from the operating system of a Databricks job compute node using a single node cluster?I’m using databricks asset bundels for job initialization and execution. When the job s...
- 4890 Views
- 4 replies
- 2 kudos
- 2 kudos
Hello @jorperort , I did some research internally and have some tips/suggestions for you to consider: Based on the research and available documentation, it is not possible to directly execute a Bash script or binary executable from the operating sy...
- 2 kudos
- 4885 Views
- 5 replies
- 0 kudos
Enabling enableChangeDataFeed on Streaming Table created in DLT
Hello, Can I enable Change Data Feed on Streaming Tables? How should I do this? I couldn't find this in the existing documentation https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed .
- 4885 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @Vsleg i think you cannot enable cdf like this for streaming tables. it is not natively supported for DLT streaming tables , please have a look here = Propagating Deletes: Managing Data Removal using D... - Databricks Community - 90978
- 0 kudos
- 1604 Views
- 3 replies
- 1 kudos
Unable to configure clustering on DLT tables
Hi TeamI have a DLT pipeline with `cluster_by` property configured for all my tables. The code looks something like below:@Dlt.table( name="flows", cluster_by=["from"] ) def flows(): <LOGIC>It was all working fine and in couple of days, the queries w...
- 1604 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Chris_N , You have mentioned - "I couldn't find any cluster properties configured." If they existed and were changed, you can use the delta history command to check if someone changed on the clustering information. It is possible there were ch...
- 1 kudos
- 2778 Views
- 1 replies
- 2 kudos
Best approach for writing/updating delta tables from python?
Hi,We are migrating a local dash app to the Databricks infrastructure (using databricks apps and our delta lake).The local app does the following (among others):- takes Excel files form the end-user- read in-memory and transforms to pandas dataframe-...
- 2778 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @georgemichael40 My suggestion wud be to try using MERGE INTO for delta tables which works with connector then using delete/insert statements. This will also keep your code in SQL as you wanted. your tables are not large so this shud be sufficient...
- 2 kudos
- 2705 Views
- 2 replies
- 0 kudos
Salesforce Bulk API 2.0 not getting all rows from large table
Has anyone run into an incomplete data extraction issue with the Salesforce Bulk API 2.0 where very large source object tables with more than 260k rows (s/b approx 13M) - result in only extracting approx 250k on attempt?
- 2705 Views
- 2 replies
- 0 kudos
- 0 kudos
@AlanDanque I am working on a similar use case and will share screen shots shortlyBut to reach the root cause can you share the below detailsChecks at SalesforceDescriptionHeader used?Was Sforce-Enable-PKChunking: chunkSize=250000 explicitly included...
- 0 kudos
- 3124 Views
- 4 replies
- 4 kudos
Resolved! Node type not available in Central India (Student Subscription)
Hi Community,I have deployed an Azure Databricks workspace in the Central India region using a student subscription. While trying to create a compute resource, I encountered an error stating that the selected node type is not available in Central Ind...
- 3124 Views
- 4 replies
- 4 kudos
- 4 kudos
@Abrarali8708 As discussed can you trymanaging the Azure Policy definition:Locate the policy definition ID /providers/Microsoft.Authorization/policyDefinitions/b86dabb9-b578-4d7b-b842-3b45e95769a1.Modify the parameter listOfAllowedLocations to inclu...
- 4 kudos
- 3030 Views
- 3 replies
- 3 kudos
Resolved! To change ownership of a materialized view
working in a Unity Catalog-enabled Databricks workspace, and we have several materialized views (MVs) that were created through a Delta Live Tables (DLT) / Lakeflow pipeline.Currently, the original owner of the pipeline has moved out of the project,...
- 3030 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Data_NXT ,You can change the owner of a materialized view if you are a both a metastore admin and a workspace admin.Use the following steps to change a materialized views owner:Open the materialized view in Catalog Explorer, then on the Overview ...
- 3 kudos
- 1692 Views
- 2 replies
- 2 kudos
Resolved! Save as Delta file in catalog
Hello, I have created data frame on csv file when I try to write it as:df_op_clean.write.format("delta").save("/Volumes/optimisation/trial")I get this error :Cannot access the UC Volume path from this location. Path was /Volumes/optimisation/trial/_d...
- 1692 Views
- 2 replies
- 2 kudos
- 2 kudos
Also to add on this:avoid overlap between tables and Volumes.Create a separate folder for tables and files.Unity catalog does this too if you use managed tables/volumes.
- 2 kudos
- 1558 Views
- 2 replies
- 1 kudos
Most suitable Data Promotion orchestration for multi-tenant data lake in Databricks
Hi there !!! I would like to find the most suitable orchestration process to promote data between medallion layers I need to solve the following key architectural decision for scaling my multi-tenant data lake in Databricks.My setup:Independent medal...
- 1558 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey mbanxp! The most scalable and maintainable orchestration pattern for multi-tenant medallion architectures in Databricks is to build independent pipelines per table for all clients, with each pipeline parameterized by client/tenant. Why this appro...
- 1 kudos
- 2549 Views
- 6 replies
- 1 kudos
How to reference a workflow to use multiple GIT sources?
Hi community,Is it possible for a workflow to reference multiple Git sources? Specifically, can different tasks within the same workflow point to different Git repositories or types of Git sources?Ty
- 2549 Views
- 6 replies
- 1 kudos
- 1 kudos
A workflow can reference multiple Git sources. You can specify the git information for each task. However, I am not sure you can have multiple GitProvider for the same workspace....
- 1 kudos
- 2322 Views
- 8 replies
- 10 kudos
ResultSet metadata does not return correct type for TIMESTAMP_NTZ
Hello, using the JDBC driver, when I retrieve the metadata of a ResultSet, the type for a TIMESTAMP_NTZ is not correct (it's a TIMESTAMP one).My SQL is a simple SELECT * on a table where you have a TIMESTAMP_NTZ columnThis works when retrieving metad...
- 2322 Views
- 8 replies
- 10 kudos
- 10 kudos
Hello @EricCournarie! Just to confirm, were you initially using the JDBC driver v2.7.3? According to the release notes, this version adds support for the TIMESTAMP_NTZ data type.
- 10 kudos
- 2594 Views
- 4 replies
- 5 kudos
Resolved! Getting an warning message in Declarative Pipelines.
Hi Team,While creating a Declarative ETL pipeline in Databricks, I tried to configure a notebook using the "Add existing assets" option by providing the notebook path. However, I received a warning message:"Legacy configuration detected. Use files in...
- 2594 Views
- 4 replies
- 5 kudos
- 5 kudos
Thank you @szymon_dybczak, Now I have a good clarification from my end.
- 5 kudos
- 3396 Views
- 8 replies
- 7 kudos
Resolved! Streamlining Custom Job Notifications with a Centralized Email List
Hi Everyone,I am working on setting up success/failure notifications for a large number of jobs in our Databricks environment. The manual process of configuring email notification using UI for each job individually is not scalable and is becoming ver...
- 3396 Views
- 8 replies
- 7 kudos
- 7 kudos
@Raj_DB Databricks sends notifications via its internal email service, which often requires the address to be a valid individual mailbox or a distribution list that accepts external mail. If your group email is a Microsoft 365, Please check if “Allow...
- 7 kudos
- 985 Views
- 2 replies
- 0 kudos
Filling a STRUCT field with a PreparedStatement in JDBC
Hello, I'm trying to fill a STRUCT field with a PreparedStatement in Java by giving a JSON string in the PreparedStatement.But it complains Cannot resolve "infos" due to data type mismatch: cannot cast "STRING" to "STRUCT<AGE: BIGINT, NAME: STRING>"....
- 985 Views
- 2 replies
- 0 kudos
- 0 kudos
Could you provide a sample of JSON string along with a code you're using? Otherwise it will be hard for us to help you.
- 0 kudos
- 3531 Views
- 2 replies
- 3 kudos
Resolved! Difference between libraries dlt and dp
In all Databricks documentation, the examples use import dlt to create streaming tables and views. But, when generating sample Python code in ETL pipeline, the import in the sample is:import pyspark import pipelines as dpWhich one is the correct libr...
- 3531 Views
- 2 replies
- 3 kudos
- 3 kudos
@yit Functionally, they are equivalent concepts (declarative definitions for streaming tables, materialized views, expectations, CDC, etc.). The differences you’ll notice are mostly naming/ergonomics:Module name:Databricks docs & most existing notebo...
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
