samples_per_second_per_gpu or tokens_per_second_per_gpu?

[Muennighoff]

I'm probably missing something but isn't this tokens per second per GPU:

open_lm/open_lm/train.py

Line 282 in 083fa31

samples_per_second_per_gpu = inputs.numel() / batch_time_m.val

inputs.numel() gives all tokens; for samples it would be inputs.shape[0], no?

[achalddave]

Yeah it’s poorly named; it is in fact tokens per second. Happy to accept a PR for a fix!

…

On Mon, Apr 29, 2024 at 1:29 PM Niklas Muennighoff ***@***.***> wrote: I'm probably missing something but isn't this tokens per second per GPU: https://github.com/mlfoundations/open_lm/blob/083fa31449c3456e889269e44913578acfced67a/open_lm/train.py#L282 inputs.numel() gives all tokens; for samples it would be inputs.shape[0], no? — Reply to this email directly, view it on GitHub <#262>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI5L4NIJOCZJVC74SMAD5TY72URNAVCNFSM6AAAAABG7AA4DCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3DSOJRGUYDKMY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>