findNLDI throttling

[hparker21]

What is your question?
I recently attempted to use the findNLDI function to find COMIDs downstream from multiple initial COMIDs. The function seemed to throttle my request in a way not previously seen. I received messages saying, "Waiting x seconds for retry backoff". It slowed down the overall process quite a bit. I have not seen this message previously when using findNLDI. Is there a way to avoid the retry backoff message and speed up the process? It now takes much longer to run a set of COMIDs through the function compared to prior attempts.

Expected behavior
I have used findNLDI extensively before and it has been very fast, even when inputting lists of numerous COMIDs. I've had it process multiple entries per second in prior iterations.

[dblodgett-usgs]

Apologies @hparker21 -- the NLDI has become increasingly popular and we've experienced situations where unthrottled access has caused the database to be overwhelmed and to slow down/become unresponsive. To avoid this, we've had to put rate limits in place. The client in dataRetrieval could probably be improved to avoid rate limit errors that you are experiencing, but that's about all we could do to make it better. We currently accept 400 navigation requests per hour per user (the limit you are hitting I think) and 3600 feature lookups per hour per user.

There is a header in the request to the API that indicates if the limit is close / says that the limit has been hit... perhaps we could implement some error handling or more direct rate limiting in dataRetrieval to help out?

[hparker21]

Hi David, thank you for the reply!

Is there currently a way to view that header in the API request? If not, I think the addition of a short message notifying users if the limit has been reached or how many submissions remain before the limit is reached would be a really helpful addition. It would help us space out our requests accordingly to prevent overwhelming the API.

[dblodgett-usgs]

I'll work up a PR that supports the header in dataRetrieval. Stay tuned.

[dblodgett-usgs]

@hparker21 -- with what's in #762 -- you will get messages about the rate limit as you approach it and an enforced sleep will be applied if you are close to the rate limit.

The enforced sleep is not as bad as the "retry-after" behavior but also a bit conservative. You will be able to play around with the timing of requests to get a bit better performance. There's probably some documentation work we could do to help people optimize throughput without overwhelming the NLDI. In general, about navigation and basin requests can be very expensive, so 400/hr is the limit right now. You can burst up to 400 per hour then you need to slow them down to about one per 10s. All other requests are capped at 3600/hr and you generally won't have any issues with them. However, feature lookups can add up fast if you are looking up network locations for lots of features.

[jarq6c]

@dblodgett-usgs Are there alternative sources or endpoints we should be using for large data requests? I'm currently trying to retrieve ~8500 basin geometries at gauges using getBasin for a NWM evaluation. I'm hitting the 400 request/hr limit pretty quickly. Is there an alternative to bogging down the service for 20+ hours with individual basin requests?

Note: I got here from Google. I'm not using dataRetrieval. I'm using aiohttp to query NLDI directly from Python.

[dblodgett-usgs]

There is a dedicated nldi issue tracker over here for the record https://github.com/internetofwater/nldi-services/issues

Short of implementing your own basin aggregation tool, this is what we can offer right now. The NLDI is designed for use cases like showing maps on monitoring location pages or working through a few basins at a time. We don't have the resources to support intermittent high throughput use.

If you wanted to implement something that would allow you to use your own compute to scale up, you might want to look at some of the capabilities of hyRiver. e.g. https://docs.hyriver.io/examples/notebooks/nhd_navigation.html