Sudden increase in load on a few machines with HTTPX #138
Description
We are experimenting with httpx. We do a lot of short 1-2-3s requests in batches on dozens of machines. All the machines are load balanced so they do not get any specific traffic from any specific client. This is the main thing this cluster is doing. Sending requests and processing their responses.
Yesterday out of nowhere a few of the machines with HTTPX went up 80-100% CPU. No increase in traffic and the amount of requests they should send. It also happened only with a few of the machines. They stayed like this for 10-15 minutes even after traffic to them was cut off and they were not processing any requests. The requests are run inside a Rails app run a multithreaded Passenger. Had to hard restart passenger.
It happened with a few of the machines. All of them have HTTPX installed for the last 2-3 days and that's the only change - httpx.
Version: 1.7.5
I know this is not enough for an issue, so my question is what would be enough here. I've read that there were some issues with previous versions of httpx with CPU going to 100%, but could not find a specific issue. Is this true? How was it investigate? What are some tools and measurements I could use in the next few days on the machines that would then be useful to understand where this is coming from?
