Hi @RutujaKadam,

The error you are seeing, "Error starting gateway compute resources" with a message about VM quota exhaustion, is related to your Azure subscription's vCPU quota rather than a misconfiguration in Databricks itself. Here is what is happening and how to resolve it.

UNDERSTANDING THE ERROR

Lakeflow Connect for SQL Server uses two components:

1. An ingestion gateway that runs on classic compute (a Job Compute cluster) to extract snapshots and change data from your SQL Server source.
2. A serverless ingestion pipeline that loads the staged data into your destination streaming tables.

The gateway cluster requires a minimum of 8 cores for efficient data extraction. When Databricks tries to launch this gateway cluster, it requests VMs from Azure in your workspace region (Central Canada). The error "the VM size you are specifying is not available" means your Azure subscription does not have enough vCPU quota in that region to provision the required VMs.

WHY THIS HAPPENS ON FREE TRIAL SUBSCRIPTIONS

Azure free trial subscriptions have limited default vCPU quotas per region. These quotas are often too low for workloads that require dedicated compute clusters. Additionally, free trial subscriptions are not eligible for quota increases through the Azure portal. You would need to upgrade to a Pay-As-You-Go subscription to request higher quotas.

HOW TO RESOLVE THIS

Option 1: Upgrade your Azure subscription and request a quota increase

1. Upgrade from the free trial to a Pay-As-You-Go subscription in the Azure portal.
2. Go to the Azure portal, search for "Quotas", and select "Compute".
3. Find the VM family used in your region (Central Canada). Look for the VM family that shows insufficient quota.
4. Select the quota and click "New Quota Request" to request an increase. The gateway needs at least 8 vCPUs for the driver node.
5. After the quota increase is approved, retry the Lakeflow Connect pipeline deployment.

Microsoft documentation on requesting quota increases:
https://learn.microsoft.com/en-us/azure/quotas/per-vm-quota-requests

Option 2: Free up existing quota

If you have other running clusters or VMs consuming your vCPU quota in Central Canada, consider stopping them temporarily to free up capacity for the Lakeflow Connect gateway.

Option 3: Try a different Azure region

If your Databricks workspace allows it, you could consider creating a workspace in a different Azure region where you may have more available VM quota. Note that this would require a new workspace since the region is set at workspace creation time.

VERIFYING YOUR CURRENT QUOTA

To check your current Azure vCPU quota and usage:

1. Sign in to the Azure portal.
2. Search for "Quotas" and select the Quotas service.
3. Select "Compute" on the overview page.
4. Filter by your region (Central Canada) and look for the VM families with high utilization or zero availability.

This will show you exactly which VM family is exhausted and by how much you need to increase it.

ADDITIONAL PREREQUISITES TO VERIFY

Once you resolve the quota issue, make sure you also have the following in place for Lakeflow Connect SQL Server ingestion:

1. Unity Catalog enabled on your workspace (you confirmed this).
2. Serverless compute available (you confirmed this).
3. The required privileges: CREATE CONNECTION, USE CONNECTION, USE CATALOG, CREATE SCHEMA, CREATE TABLE, and CREATE VOLUME.
4. SQL Server configured with either Change Tracking or CDC enabled on the tables you want to ingest.
5. A dedicated database user in SQL Server for Databricks ingestion with the required privileges.
6. Network connectivity between your Databricks workspace and the Azure SQL Server (firewall rules allowing access).

Lakeflow Connect SQL Server documentation:
https://docs.databricks.com/en/ingestion/lakeflow-connect/sql-server.html

Lakeflow Connect SQL Server pipeline setup:
https://docs.databricks.com/en/ingestion/lakeflow-connect/sql-server-pipeline.html

Lakeflow Connect troubleshooting:
https://docs.databricks.com/en/ingestion/lakeflow-connect/sql-server-troubleshoot.html

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.