Skip to content

update throttle handling with nldi for #759#762

Merged
ldecicco-USGS merged 1 commit intoDOI-USGS:mainfrom
dblodgett-usgs:main
Apr 7, 2025
Merged

update throttle handling with nldi for #759#762
ldecicco-USGS merged 1 commit intoDOI-USGS:mainfrom
dblodgett-usgs:main

Conversation

@dblodgett-usgs
Copy link
Copy Markdown
Contributor

I'm not totally satisfied with this, but I think it ticks the box for #759

I now see:

> sapply(1:500, function(x) findNLDI(comid = 101, nav = "DM", find = c("nwis", "wqp")))
Approaching NLDI rate limit. 28 of 400 remaining
Approaching NLDI rate limit. 26 of 400 remaining
Approaching NLDI rate limit. 25 of 400 remaining
Approaching NLDI rate limit. 23 of 400 remaining
Approaching NLDI rate limit. 22 of 400 remaining
Approaching NLDI rate limit. 20 of 400 remaining
Approaching NLDI rate limit. 19 of 400 remaining
Sleeping to try to avoid going over rate limit.
Approaching NLDI rate limit. 19 of 400 remaining
Sleeping to try to avoid going over rate limit.
Approaching NLDI rate limit. 20 of 400 remaining
Approaching NLDI rate limit. 19 of 400 remaining
Sleeping to try to avoid going over rate limit.

This won't send messages unless a user gets to within 90% of the overall rate limit. It won't sleep until the user is under 20 requests remaining in their allocation. The rate limit calculations are a black box so I was quite conservative with waiting 3x what should be technically possible. It seems like the rate limits accrue in some way differently from 1:1.

The way the rate limit header is listed, it shows the absolute limit and how many is left in that allocation. It does not say anything about the time base for the rate limit. For now, it's all based on a per hour allocation. We will try to stick to the hour basis going forward.

@dblodgett-usgs
Copy link
Copy Markdown
Contributor Author

@ldecicco-USGS -- this is my first time touching the httr2 implementation in here -- hope I didn't step in anything.

@ldecicco-USGS
Copy link
Copy Markdown
Collaborator

Looks good!

@ldecicco-USGS ldecicco-USGS merged commit 5fcfc00 into DOI-USGS:main Apr 7, 2025
1 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants