Databricks

MichTalebzadeh

For those interested in Data Mesh and Data Lakes for FinCrime detection:Data mesh is a relatively new architectural concept for data management that emphasizes domain-driven data ownership and self-service data availability. It promotes the decentral...

MichTalebzadeh · 03-22-2024

I created this article in Linkedlin to allow both this community and Apache Spark user community to have access to it.It is particularly useful for data engineers who want to have a basic understanding of what Generative AI with Spark can do.Leverag...

MichTalebzadeh · 03-19-2024

IntroductionFinancial fraud is a significant concern for businesses and consumers alike. I have written about this concern a few times in Linkedlin articles. Machine learning offers powerful tools to combat this issue by automatically identifying sus...

MichTalebzadeh · 03-07-2024

I have downloaded Am azon reviews for sentiment analysis from here. The file is not particularly large (just over 500MB) but comes in the following formattest.ft.txt.bz2.zipSo it is a text file that is compressed by bz2 followed by zip. Now I like t...

MichTalebzadeh · 03-06-2024

I recently saw an article from Databricks titled "Scalable Spark Structured Streaming for REST API Destinations". A great article focusing on continuous Spark Structured Streaming (SSS). About a year old. I then decided, given customer demands to wo...

MichTalebzadeh · 03-31-2024

Your aim is to access external S3 tables from a Unity Catalog workspace without data duplication and keeping data updates synchronized. Configure external location permissions. This ensure that both your Unity Catalog and Hive metastore workspaces h...

MichTalebzadeh · 03-20-2024

short answer is noDatabricks uses a single catalog -> Unity Catalog for managing tables created with Hive or Spark SQL, simplifying things. In spark sql you can dospark-sql (default)> show catalogs; spark_catalogHTH

MichTalebzadeh · 03-20-2024

This is my guessYour code (LOCATION "s3://<bucket-name>/<folder-name>/parquet_file/) is likely pointing to a single Parquet file instead of the entire folder containing partitioned data.try thisCREATE TABLE table_name ( col1 type1, col2 type2, ...

MichTalebzadeh · 03-20-2024

Thanks, for feedbackReading through your comment, I believe this is what is meant for further clarity."To build a machine learning model for detecting fraudulent transactions using PySpark's MLlib, synthetic transaction data can be generated. This pr...

MichTalebzadeh · 03-16-2024

Short answer is no. A single Spark AutoLoader typically cannot handle JSON files in a container with two different schemas by default.. AutoLoader relies on schema inference to determine the data structure. It analyses a sample of data from files ass...

Databricks

User Stats

User Activity

Financial Crime detection with the help of Apache Spark, Data Mesh and Data Lake

Feature article: Leveraging Generative AI with Apache Spark: Transforming Data Engineering

Build a machine learning model to detect fraudulent transactions using PySpark's MLlib library

Working with a text file that is both compressed by bz2 followed by zip in PySpark

Building Event-Driven Real-Time Data Processor with Spark Structured Streaming and API Integration

Re: Accessing data from a legacy hive metastore workspace on a new Unity Catalog workspace

Re: databricks-jdbc lists `spark_catalog` among catalogs for Standard tier Azure workspace

Re: Creating External Table from partitioned parquet table

Re: Build a machine learning model to detect fraudulent transactions using PySpark's MLlib library

Re: issue in loading the json files in same container with different schemas