Possible Issue with logging ResponseSize caused by insufficient storage type

[nathanlcarlson]

This function (https://github.com/Unidata/thredds/blob/v5.0.0-beta6/tds/src/main/java/thredds/servlet/filter/TdsServletResponseWrapper.java#L76) uses an int to set the content size of the HTTP response which later gets logged in "Request Completed" log entries in threddsServlet.log. My understanding is that for this reason the logs will only be able to report response sizes up to 2 GB (2^31). Let me know if I am misunderstanding something here.

[lesserwhirls]

Great question! I believe this particular call is only used in a few places where the size will be much smaller than 2 GB (such as serving a catalog response in html or xml). There are other places in the code, like here that explicitly handle larger transfers, such as HTTPServer requests. In those cases, I can confirm the correct bytes transferred size will show up in threddsServlet.log. That said, because this isn't handled uniformly in the codebase, I would not be too surprised if what you describe above happens for some protocols/cases. Do you have a case where you see this behavior happening?

[nathanlcarlson]

I have found it by parsing logs from the past few years looking for primarily transactions that match /thredds/fileServer/.....nc.

That line you linked will set the 'Content-Length' header, but it leaves the httpResponseBodyLength member in TdsServletResponseWrapper untouched. So, any time contentLength > Integer.MAX_VALUE the log will report -1 for the size, as I understand it.

The member/class that is used to report to "Request Completed" entries in the threddsServlet.log.
https://github.com/Unidata/thredds/blob/v5.0.0-beta5/tds/src/main/java/thredds/servlet/filter/TdsServletResponseWrapper.java#L23

Where the httpResponseBodyLength member is used.

thredds/tds/src/main/java/thredds/servlet/filter/RequestBracketingLogMessageFilter.java

Line 55 in 11b9d63

log.info(UsageLog.closingMessageForRequestContext(response.getHttpStatusCode(), response.getHttpResponseBodyLength()));

[lesserwhirls]

Ah, thank you! I think I see now. We normally parse the Apache logs to get stats on how much data our TDS instances serve, but we should try to accurately capture that in the threddsServlet logs as well. I'll be out of the office for the next few weeks, but I can take a look again once I get back (unless @ethanrd wants to give this a look).

[nathanlcarlson]

That is what we have arrived at as well, using the Apache logs.