feat: report absolute eval time in nanoseconds instead of relative to reference#1383
feat: report absolute eval time in nanoseconds instead of relative to reference#1383ArpanC6 wants to merge 3 commits intoTuringLang:mainfrom
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1383 +/- ##
=======================================
Coverage 82.35% 82.35%
=======================================
Files 50 50
Lines 3531 3531
=======================================
Hits 2908 2908
Misses 623 623 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Hlw @yebai, this PR partially addresses #1374. I've gone through the issue carefully and made the following changes
I also noted @penelopeysm's point that introducing an external baseline doesn't simplify comparative benchmarks you still need two checkouts regardless. So I've kept this PR focused on the absolute time reporting for now. Note: The Introducing Stan as an external baseline would be a separate more involved task happy to work on that as a follow up PR. |
Closes #1374 (partially)
What was the problem?
The benchmark reported
t(eval)as a ratio relative to an arbitraryreference function (
simple_assume_observe_non_model), which maderesults hard to interpret and added unnecessary noise.
What did I fix?
t(eval)in absolute nanosecondst(grad)/t(eval)ratio is kept as-isNote
Introducing Stan as an external baseline is a separate more involved
task left for a future PR.