Task: with internal failure on first try, single worker, high expiration timeout get stuck

[maruel]

This is a degenerate case:

• Only one bot can accept and run a task
• An internal failure happens

In that case, when the bot polls, it will see the task but won't be allowed to run it again, so the task will stay pending until it hits its expiration timeout. This is wasteful and hard to detect; swarming doesn't know that there isn't another bot that could come up any time to service the task, yet the internal failure situation is not told to the user so this is nearly impossible to diagnose for them.

AI:
On internal failure, Swarming looks if there is any other known worker that could service the task. If not, immediately mark as internal_failure, does not retry.

Swarming is designed to retry on internal failure to another worker. There are use cases where there will be only one bot to service one kind of task, for example with exotic/prototype hardware. It's better to handle this case explicitly.