- 1015 Views
- 3 replies
- 0 kudos
Talend ETL code to Databricks
Hi Team,What is the best way to transfer Talend ETL code to Databricks and what are the best methods/practices for migrating Talend ETL's to Databricks (notebook, code conversion/migration strategy, workflow's etc)?Regards,Janga
- 1015 Views
- 3 replies
- 0 kudos
- 0 kudos
if you're planning to migrate from your current technology to Databricks, Travinto Technologies' Code Converter Tool is here to make the process seamless. This powerful tool enables you to migrate data, ETL workflows, and reports across platforms eff...
- 0 kudos
- 1035 Views
- 3 replies
- 0 kudos
Databricks CI/CD
Hi Team,How can we implement CI/CD in Databricks and its best practices, and tools to consider?Regards,Janga
- 1035 Views
- 3 replies
- 0 kudos
- 211 Views
- 1 replies
- 0 kudos
code vulnerabilities, code smells, and bugs
Hi Team, is there a way in Databricks to check for code vulnerabilities, code smells, and bugs?Note :Databricks native functionality only
- 211 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,As far as I am aware, for security scanning/monitoring at Databricks account level, we have belowSAT - https://github.com/databricks-industry-solutions/security-analysis-toolhttps://www.databricks.com/trust/trusthttps://learn.microsoft.com/en-us/a...
- 0 kudos
- 446 Views
- 1 replies
- 0 kudos
incremental loads without date column
Hi All,We are facing a situation where our data source is Snowflake, and the data is saved in a storage location(adls) in parquet format. However, the tables or data lack a date column or any incremental column for performing incremental loads to Dat...
- 446 Views
- 1 replies
- 0 kudos
- 0 kudos
Ideally you would have some change tracking system (cdc f.e.) on the source tables (Streams in the case of Snowflake, Introduction to Streams | Snowflake Documentation).But that is not the case.So I think you approach is ok. You cannot track what is...
- 0 kudos
- 293 Views
- 1 replies
- 0 kudos
Oracle -> Oracle Golden Gate ->Databricks Delta lake
Hi All,We have a situation where we are collecting data from different Oracle instances.The customer is using Oracle GoldenGate to replicate this data into a storage location.From there, we can use Auto Loader or Delta Live Tables to read Avro files ...
- 293 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Phani1 ,In my opinion this is really good setup. You have push scenario where Oracle GoldenGate is responsible for delivering data into storage, so you don't have to bother about extraction part. And autoloader is the best choice when it comes t...
- 0 kudos
- 264 Views
- 0 replies
- 0 kudos
Denodo Connection Parameters.
Hi All,We are establishing a connection from Denodo to Databricks. During the development phase, we utilized a personal access token associated with developer account. However, this approach is not considered a best practice for production environm...
- 264 Views
- 0 replies
- 0 kudos
- 466 Views
- 1 replies
- 0 kudos
Databricks serverless Vs snowflake
Hi All,We want to switch from Snowflake to Databricks SQL Warehouse/serverless to simplify our data layers and reduce data copies before the reporting layer. Please share the benefits of using serverless over Snowflake and any limitations you see. We...
- 466 Views
- 1 replies
- 0 kudos
- 0 kudos
one big pro is that you do not need to copy data to the dwh. also your transformations and analytics queries reside on the same platform (databricks).If databricks can cover all the requirementsm compared to snowflakem is hard to tell. Probably ther...
- 0 kudos
- 684 Views
- 1 replies
- 0 kudos
best practices for implementing early arriving fact handling
Hi All,Can you please share us the best practices for implementing early arriving fact handling in databricks for streaming data processed in near real time using structured streaming.There are many ways to handle this use case in batch/mini batch. ...
- 684 Views
- 1 replies
- 0 kudos
- 0 kudos
Greetings Team, I would like to inquire if any of you have suggestions regarding the query.
- 0 kudos
- 502 Views
- 1 replies
- 0 kudos
autoloader data processing
Hi Team,Can you share the best practices for designing the autoloader data processing?We have data from 30 countries data coming in various files. Currently, we are thinking of using a root folder i.e country, and with subfolders for the individual ...
- 502 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Phani1 ,Structure of folders that you are going to use make sense to me. Since you've mentioned that there will be thousands of files, the best practice will be to use autoloader with file notification mode. Also, you can read about databricks r...
- 0 kudos
- 1013 Views
- 2 replies
- 1 kudos
classic cluster vs serverless cost
Hi Team,Can you help me the cost comparison between classic cluster and serverless?
- 1013 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Phani1 , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback...
- 1 kudos
- 2664 Views
- 2 replies
- 1 kudos
Azure Synapse vs Databricks
Hi team,Could you kindly provide your perspective on the cost and performance comparison between Azure Synapse and Databricks SQL Warehouse/serverless, as well as their respective use cases? Thank you.
- 2664 Views
- 2 replies
- 1 kudos
- 1 kudos
Agree with @mhiltner, it doesn't make sense to compare it with Synapse, as it's literally dead. You most likely want to compare it to Fabric instead. Fabric is highly under development, but IMHO it still lacks behind other Data/AI solutions. No catal...
- 1 kudos
- 2583 Views
- 0 replies
- 1 kudos
- 2583 Views
- 0 replies
- 1 kudos