You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@alexcrichton: No, looks like one of our components was leaking VMs, which is probably why you were seeing slower builds as well, as the hosts were filling up with extra VMs. We're normally running an extra component to detect this and remove stale VMs, but it looks like that hasn't been enabled for your cluster yet.Sorry for the lack of monitoring of this on our end, we're in the progress of spinning up some new infrastructure that allows us to create these dedicated clusters but have the monitoring be integrated with the rest of our monitoring so we can catch stuff like this earlier. I don't have an ETA for when that'd be ready right now, but I'll see if we could get the cleanup component running on your cluster. For now I'll run a manual cleanup to get your cluster back to normal, I'll let you know when it's fully back again.
No idea why, but they're taking > 3hrs which is killing them.
The text was updated successfully, but these errors were encountered: