Feature request: allow accepting/ignoring timeout errors

[yegor256]

Problem

Currently, the --accept flag only supports HTTP status codes. There is no way to ignore connection-level errors like timeouts.

When checking links across a large site, some external URLs may intermittently timeout due to network conditions or slow servers. These transient failures cause CI to fail even though the links are valid.

Proposed Solution

Extend the accept configuration to support error types in addition to HTTP status codes. For example:

accept = [200, 429, "timeout"]

Or add a separate option:

skip_errors = ["timeout", "dns"]

Use Case

I run lychee in CI to check links in a blog with hundreds of posts. Some external sites occasionally timeout, causing the entire CI run to fail. I would like to:

1. Report timeouts as warnings (not errors)
2. Or ignore them entirely

Currently, the only workarounds are:

• Increasing timeout and retries (doesn't fully solve the problem)
• Excluding specific URLs (requires maintenance as new timeout-prone URLs appear)
• Post-processing lychee output in CI scripts

Alternatives Considered

• Excluding URLs via .lycheeignore — requires knowing which URLs will timeout in advance
• Increasing timeout and max_retries — helps but doesn't eliminate the issue
• Filtering output with grep in CI — fragile workaround

[cristiklein]

In case it makes implementation easier, I could also imagine mapping various errors to HTTP Status Codes >=600. For example, TIMEOUT could be 601. Rust documentation seems to allow this:

Values above 599 are unclassified but allowed for legacy compatibility, though their use is discouraged. Applications may interpret such values as protocol errors.

In that case, the user would simply set:

accept = [200, 429, 601]

[thomas-zahner]

For example, TIMEOUT could be 601.

We should really try to avoid such magic sentinel values. In fact lychee is intended to work with with non-standard/non-conformant status codes such as 601. A prominent example and use case is linkedin which returns the status code 999 (depending on the user-agent if I remember correctly). In this case the user can allow lychee to accept the code with e.g. --accept 200..300,999.

Only because it's neither recommended by the spec nor by the http crate doesn't mean it's invalid. But this is more of a HTTP spec problem.. So we specifically must be able to work with 100-999.

I have to say that I never encountered the use-case of wanting to accept timeouts. As you noticed we also have a retry mechanism. So a server has to be really unreliable or rather purposefully block the user's requests, which seems like the more likely scenario. But in both cases users would generally exclude the affected URLs. Could you maybe elaborate more on your use case?

Also with lychee 0.23.0 encountering rate-limiting and timeouts should be much rarer due to the new per-host handling mechanisms. I really recommend to test the newer version if you haven't already.

[cristiklein]

Thanks for your feedback @thomas-zahner !

For example, TIMEOUT could be 601.

We should really try to avoid such magic sentinel values. In fact lychee is intended to work with with non-standard/non-conformant status codes such as 601. A prominent example and use case is linkedin which returns the status code 999 (depending on the user-agent if I remember correctly). In this case the user can allow lychee to accept the code with e.g. --accept 200..300,999.

Only because it's neither recommended by the spec nor by the http crate doesn't mean it's invalid. But this is more of a HTTP spec problem.. So we specifically must be able to work with 100-999.

I was not aware that non-standard HTTP responses are a thing. 😄 Thanks for highlighting that. In light of this, I agree that sentinel values must be avoided.

I have to say that I never encountered the use-case of wanting to accept timeouts. As you noticed we also have a retry mechanism. So a server has to be really unreliable or rather purposefully block the user's requests, which seems like the more likely scenario. But in both cases users would generally exclude the affected URLs. Could you maybe elaborate more on your use case?

I'm running lychee against a rather big website with a lot of links. We quickly realized that we ended up being someone else's uptime monitor. For example, yesterday lwm.net seems to have been down. (Unsure if it was really down or if it was down only from the perspective of the GitHub Action runner.) 2 weeks ago, it was freedesktop.org. Excluding these URLs is not really what we want, as we do want to check our links against them, when they eventually come up.

Also with lychee 0.23.0 encountering rate-limiting and timeouts should be much rarer due to the new per-host handling mechanisms. I really recommend to test the newer version if you haven't already.

I'm not sure this is a lychee problem or a "websites do fail" problem. Before lychee, we used linkchecker and that was bumping into frequent timeouts too. But we'll give lychee 0.23.0 a try.