syntect-server freeport detection failing

[slimsag]

A customer https://app.hubspot.com/contacts/2762526/company/693777200 / https://sourcegraph.slack.com/archives/C011899KB6U/p1597363626348700 reports syntect-server is failing in their EKS cluster with nothing other than:

2020/08/14 00:06:27 worker command: env ROCKET_PORT={{.Port}} /syntect_server
2020/08/14 00:06:27 failed to find free port
2020/08/14 00:06:27 failed to find free port
2020/08/14 00:06:27 failed to find free port
2020/08/14 00:06:27 failed to find free port
2020/08/14 00:06:28 failed to find free port

This behavior happened only recently with no prior changes, indicating that freeport detection was working before in the same deployment but no longer is.

Most likely, this is caused by syntect-server's workers dying and allocating new ports - leading to ephemeral port exhaustion on the EKS host node. However, the logs which indicate which files are problematic appear to not be present (most likely they were paged out due to the number of "failed to find free port" messages.)

https://github.com/sourcegraph/syntect_server makes use of the freeport Go package to spawn worker processes. That package has a number of reported failures 1 2 3 about its localhost-based resolution.

We should adopt a different package and/or phayes/freeport#8 for port allocation and see if that resolves the issue.

[pecigonzalo]

The customer managed to start the service by removing http-server-stabilizer as a workaround.

[slimsag]

• Fixed in http-server-stabilizer via slimsag/http-server-stabilizer@5f85eec
• Fixed in syntect_server via sourcegraph/syntect_server@e7ad1ef

You can disable the new freeport library and restore old functionality by setting USE_OLD_FREEPORT=true on the syntax highlighter container if needed, but eventually this will be removed (just an escape hatch in case this change introduces any new issues).

[slimsag]

Fix will be included in 3.19, waiting to confirm with customer that it actually worked.

[slimsag]

Fix confirmed: https://sourcegraph.slack.com/archives/C011899KB6U/p1600720189004800?thread_ts=1597363626.348700&cid=C011899KB6U