Spawns and terminate VM for builder process.
Checks that number of VM in any state excluding Terminating plus number of running spawn processes is less than threshold defined by BackendConfig.build_group[group][“max_vm_total”]
Checks that time elapsed since latest VM spawn attempt is greater than threshold defined by BackendConfig.build_group[group][“vm_spawn_min_interval”]
Check that number of running spawn processes is less than threshold defined by BackendConfig.build_group[][“max_spawn_processes”]
Check that number of running spawn processes is less than threshold defined by BackendConfig.build_group[][“max_spawn_processes”]
After server crash it’s possible that some VM’s will remain in check_health state Here we are looking for such records and mark them with check_health_failed state
If we failed to terminate instance request termination once more. Non-terminated instance detected as vm in the terminating state with
time.time() - terminating since > Threshold
It’s possible, that VM was terminated but termination process doesn’t receive confirmation from VM provider, but we have already got a new VM with the same IP => it’s safe to remove old vm from pool