Databricks Community

MichTalebzadeh · 08-03-2024

For a UK Government Agency, I made a Comprehensive presentation titled " Feature Engineering for Data Engineers: Building Blocks for ML Success". I made an article of it in Linkedlin together with the relevant GitHub code. In summary the code delve...

MichTalebzadeh · 07-24-2024

Summary"The Big Book of Generative AI from Databricks" provides a comprehensive guide to these critical questions and more, offering valuable insights into the tools and skills required for successful GenAI implementation. By addressing these questio...

MichTalebzadeh · 07-24-2024

"The Big Book of Generative AI from Databricks"https://lnkd.in/dR4VuEyQprovides a comprehensive guide to understanding and implementing Generative AI (GenAI) effectively within an enterprise. Here are some critical questions that businesses should co...

MichTalebzadeh · 05-21-2024

I just wanted to share a tool I built called spark-column-analyzer. It's a Python package that helps you dig into your Spark DataFrames with ease.Ever spend ages figuring out what's going on in your columns? Like, how many null values are there, or h...

MichTalebzadeh · 04-27-2024

For those interested in Data Mesh and Data Lakes for FinCrime detection:Data mesh is a relatively new architectural concept for data management that emphasizes domain-driven data ownership and self-service data availability. It promotes the decentral...

MichTalebzadeh · 09-17-2024

Many thanks for your kind words Anushree. Much appreciated.

MichTalebzadeh · 05-21-2024

An example added to README in GitHubDoing analysis for column PostcodeJson formatted output{"Postcode": {"exists": true,"num_rows": 93348,"data_type": "string","null_count": 21921,"null_percentage": 23.48,"distinct_count": 38726,"distinct_percentage"...

MichTalebzadeh · 05-14-2024

Hi,The error indicates that the Unity Catalog does not support Spark higher-order functions, such as those used in pandas_udf. This limitation likely comes from architectural or compatibility constraints. To resolve the issue, consider alternative ap...

MichTalebzadeh · 03-31-2024

Your aim is to access external S3 tables from a Unity Catalog workspace without data duplication and keeping data updates synchronized. Configure external location permissions. This ensure that both your Unity Catalog and Hive metastore workspaces h...

MichTalebzadeh · 03-20-2024

short answer is noDatabricks uses a single catalog -> Unity Catalog for managing tables created with Hive or Spark SQL, simplifying things. In spark sql you can dospark-sql (default)> show catalogs; spark_catalogHTH

Databricks Community

User Stats

User Activity

Feature Engineering for Data Engineers: Building Blocks for ML Success

Review of the First 20 Pages of "The Big Book of Generative AI from Databricks"

Navigating the Future: Key Questions for Implementing Production-Quality GenAI

A handy tool called spark-column-analyser

Financial Crime detection with the help of Apache Spark, Data Mesh and Data Lake

Re: Feature Engineering for Data Engineers: Building Blocks for ML Success

Re: A handy tool called spark-column-analyser

Re: Iterating over a pyspark.pandas.groupby.DataFrameGroupBy

Re: Accessing data from a legacy hive metastore workspace on a new Unity Catalog workspace

Re: databricks-jdbc lists `spark_catalog` among catalogs for Standard tier Azure workspace