Hi @amanjethani ,

Thanks for laying out the setup and symptoms so clearly. The hang likely occurs because LightGBM’s distributed network either doesn’t fully form between executors or because the expected task count doesn’t match actual tasks, leading to a deadlock immediately after the driver closes its coordination sockets. Pinning a single listen port (e.g., 12400) on executors, matching numTasks to your partitions, and toggling barrier execution mode are the most reliable fixes. Set a time_out to fail fast instead of hanging so you can capture the underlying error in logs.

To answer you questions specifically:

• Why does training hang even after all workers successfully establish socket communication?
  Because the LightGBM workers still need to complete peer-to-peer connections across all executors after the driver shares topology. Any failure to connect to a peer (firewall, port not open, wrong IP/interface) or a mismatch in the number of tasks expected vs. actually started will cause the network init to block without error in Spark. (source)

• Is this a known issue with certain versions of synapseml or LightGBM?
  Yes—there are reports of indefinite hangs on SynapseML 0.11.1 in distributed fits (non-Databricks) and prior versions where barrier mode and networking interplay caused hangs; turning off barrier mode has helped in some cases. (source)

• How can I restrict or fix the port range LightGBM uses?
  You can pin the ports with SynapseML parameters defaultListenPort (executors) and driverListenPort (driver), and/or pass LightGBM’s native local_listen_port via passThroughArgs. The default listen port used by LightGBM is 12400, and you can reliably pin to a single port instead of a wide range. (source)

• Any workaround or logs I should enable to debug deeper?
  Increase SynapseML verbosity to 2 for more detailed logs, enable barrier mode selectively, and set LightGBM’s time_out (minutes) to cause a timeout instead of an infinite wait. Use Spark’s logs and explicit connectivity tests between workers on the pinned port to validate end-to-end reachability.

• Is it possible that a missing barrier or stage finalization in Spark is causing this silent hang?
  Yes—SynapseML explains that deadlocks can occur during initialization if the driver’s expected task count differs from actual, and barrier mode is available as a mitigation (with caveats). Toggling it can resolve hangs depending on the cluster.