Hi Toby,

Managing diverse, unstructured data can be challenging. At Know2Ledge (ShareArchiver), we specialize in unstructured data management to streamline this process.

To handle your scenario efficiently:

1️⃣Pre-Process Before Ingestion – Use AI-powered classification and indexing to structure data before storing it in the cloud.
2️⃣Optimize Storage & Indexing – Leverage automated metadata extraction to make data Databricks-ready without excessive scripting.
3️⃣Scale Smartly – Instead of running Python scripts on a single node, use containerized microservices for preprocessing and tiered storage for cost optimization.

This approach reduces overhead while ensuring scalability. More details here: ShareArchiver Unstructured Data Solutions

Would love to hear your thoughts!

[mark] |