Send a default user agent

[craigreyenga]

While attempting to run Muffet against a WikiJS instance, I found that I was getting a 500 error. Certainly that is WikiJS's fault. However, it seems reasonable that Muffet should send up a non-empty user agent by default.

[craigreyenga]

Workaround:

muffet --header="User-Agent: Muffet/$(muffet --version)"

[raviqqe]

We decided to remove a default agent by default a while ago because we couldn't think of a default value that works for everyone. #142

[craigreyenga]

We decided to remove a default agent by default a while ago because we couldn't think of a default value that works for everyone. #142

That would make Muffet the only somewhat mainstream HTTP client that does not send up a user-agent string.

AWS and Cloudflare both offer documentation specifically about how to block clients that do not send up this header.