Hi @harisrinivasay,

Your feedback is valid and aligned with areas the product team is still actively evolving. Since this is a Public Preview connector, Databricks explicitly expects scalability and UX feedback like yours to shape GA behaviour.

Concretely, you’ll get the most traction by opening a support ticket and referencing the need for table discovery to work cleanly when serverless is selected in the UI and also the latency you’re seeing, enumerating ~1k tables and the requirement to add tables without repeated long waits. Also, if you have a Databricks account team for your company, you can loop them in and point them at that ticket and your environment scale.

Check this page for full details on the other limitations. You can see it says the below.

Databricks recommends ingesting 250 or fewer tables per pipeline. However, there is no limit on the number of rows or columns that are supported within these tables.

In a big environment like yours, the UIs "browse and select tables" pattern becomes painful because it tries to enumerate a very large metadata surface. You may want to consider API/Declarative Automation Bundles instead of the UI for table selection as a workaround.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.