feat (future): RIFE neural frame interpolation as a fourth client-side mode

[geldoronie]

Status: future / wishlist — recording the idea so it isn't lost. No milestone, no timeline. Will be considered when there's bandwidth and a clear demand signal beyond the original Reddit suggestion.

Suggested by a Reddit user:

It would be really neat if you could in the future incorporate RIFE motion interpolation models, similar to what Vint and SVP do. RIFE is much better than LSFG at interpolating 30 fps → 60 fps with appreciably fewer artifacts.

The primary use case described is the direct local capture path — someone using RetroCapture as a standalone viewer on the same machine where the capture card is plugged in, watching the live source on their monitor. When the source runs at 30 fps (PC games via DirectShow, certain emulator pipelines, some HDMI sources) and the monitor is at 60 / 120 / 144 Hz, the result today is judder: every captured frame is displayed for multiple refreshes. RIFE addresses that by synthesising the missing in-between frames.

We do already have basic interpolation logic in the codebase — but as an extra that came with the remote-stream feature (#47): a per-refresh linear blend in the client viewer to deal with display-rate mismatches when watching a remote stream. That work isn't the focus of this issue. The Reddit ask is about the local capture path, which today has no interpolation at all. The same RIFE backend would naturally also be usable from the remote client (replacing / augmenting the existing linear blend), but that's a bonus call site, not the headline.

Background — what RIFE is and why it matters here

Today's interpolation story in RetroCapture:

• Local capture path (host using RetroCapture directly with a capture card): no interpolation. Each captured frame is held on screen until the next one arrives. If capture rate < display refresh, the held frames produce judder. The shader pass and the display are decoupled in time only through vsync — there is no concept of "in-between frame".
• Remote-stream client path (feat: shader-preserving distributed playback — "Connect to Remote Stream" client mode #47): three modes shipped — off, nearest, linear blend. Linear blend smooths display-rate mismatches but produces ghosting on fast motion (both positions of a moving object visible simultaneously).

RIFE (Real-Time Intermediate Flow Estimation, arxiv:2011.06294, hzwer/Practical-RIFE, MIT-licensed source) is a small neural network that, given two consecutive frames, estimates per-pixel optical flow and synthesises an intermediate frame with every object placed where it should be at the in-between moment. No ghosting. The "real-time" claim is justified — on a mid-range GPU (RTX 3060 class) RIFE produces 1080p intermediates well above 60 Hz, fast enough for live capture.

Slotted into RetroCapture, RIFE becomes a fourth interpolation mode available wherever it's useful: a new dropdown on the local capture path (today there is none), and a new entry alongside off / nearest / linear in the remote-client dropdown.

Glossary of tools mentioned

• SVP — SmoothVideo Project. A long-running plugin family for MPV / MPC-HC that interpolates video playback to the display refresh rate. Modern SVP supports multiple backends including RIFE. Closest analogue to the local-playback use case the Reddit user described.
• Vint — newer tool focused on live sources (DirectShow capture devices, screen capture). Same idea as SVP but built for low-latency game capture rather than file playback. Closest analogue to what this issue is proposing for RetroCapture's local capture path.
• LSFG (Lossless Scaling Frame Generation) — paid Windows utility that does frame generation on top of any window. Not RIFE-based; uses its own (simpler) algorithm. Worse than RIFE for 30 → 60 according to the user feedback that prompted this issue.
• rife-ncnn-vulkan — the reference C++ implementation: nihui/rife-ncnn-vulkan. Uses Tencent's NCNN inference library with a Vulkan backend, so it runs on NVIDIA / AMD / Intel GPUs without requiring CUDA. This is the variant that's feasible to embed in a desktop application.

What RIFE would unlock

Primary (the Reddit ask — local capture path)

1. 30 fps capture sources at 60 / 120 / 144 Hz display: PC games captured via DirectShow that run at 30, certain emulator pipelines, some HDMI capture devices that hand frames in at fixed 30 fps. Today these judder visibly; RIFE gives smooth motion.
2. PAL 50 Hz → 60 Hz display: removes telecine pulldown judder on PAL retro consoles displayed on standard 60 Hz monitors.
3. Any sub-display-refresh capture rate: same principle — fill in the missing temporal samples.

Secondary (extension of #47 client path)

4. Bandwidth-saving remote stream: host encodes at 30 fps, client RIFEs to 60 / 120 fps locally. Combined bandwidth + visual-quality tradeoff can beat encoding 60 fps at the same bitrate, especially for high-motion content.
5. Replacing linear-blend on the client: substantially fewer artifacts at 60 → 120 / 144 Hz on high-refresh client displays. Linear blend stays as the no-GPU fallback.

Where RIFE would NOT help

• Native-refresh-matched sources viewed at their native refresh (60 Hz console on 60 Hz monitor). Already 1:1.
• Encoding-side use on a host streaming for remote clients. The information gain would be illusory — the consumer already has access to the original signal, and RIFE artefacts get baked into the encoded stream forever.
• Reducing visible compression artifacts. RIFE addresses temporal smoothness only; spatial artifacts are an encoder bitrate / shader question.

High-level architecture sketch

                                       monitor refresh rate
                                              │
   ┌─────────────────────────────────┐        │
   │  Local capture path (HOST)      │        │
   │                                 │        ▼
   │  Capture ─► [interp gate*] ─► Shader pipeline ─► Swap
   │             │                                       
   │             ├─ off (today's behaviour)              
   │             ├─ linear blend (NEW small step)        
   │             └─ RIFE (this issue)                    
   │                                                     
   └─────────────────────────────────┘                   

   ┌─────────────────────────────────┐
   │  Remote-stream client path      │
   │  (this exists today, #47)       │
   │                                 │
   │  /raw decode ─► [interp gate] ─► Shader pipeline ─► Swap
   │                  │
   │                  ├─ off
   │                  ├─ nearest
   │                  ├─ linear blend
   │                  └─ RIFE (this issue, second call site)
   │
   └─────────────────────────────────┘

The local-path interpolation gate doesn't exist today; it would be added by this issue alongside the RIFE option. Worth noting that even adding linear blend locally would already be an improvement over today's no-interpolation, and it's much cheaper to implement — could ship as a precursor step.

Per-frame data flow (same on both paths)

1. Two adjacent source frames are kept in memory (already true on both paths — local capture has the current frame, remote-client keeps a small queue).
2. The renderer asks: "I need a frame for display time t that falls between N–1 and N."
3. The RIFE backend receives (frame[N-1], frame[N], t) and returns an RGB image (t ∈ [0, 1]).
4. That image is uploaded to the texture the shader pipeline already consumes.

Latency budget: one full frame interval. RIFE inference on a modern GPU at 1080p takes ~5–15 ms — fits comfortably inside a 16.6 ms (60 Hz) or 8.3 ms (120 Hz) frame budget on capable hardware.

Implementation options for the backend

Three realistic paths, ordered by integration cost (cheapest first):

1. Statically link rife-ncnn-vulkan as a library: vendor the NCNN dependency (already CMake-friendly), include the small RIFE inference wrapper, expose a RIFEInterpolator class that both pipelines call. Cleanest from a UX perspective — no external installs. Cost: ~30 MB of NCNN binaries linked in, ~50–100 MB of model weights shipped (or downloaded on first use, see Open Questions).
2. Spawn rife-ncnn-vulkan as a subprocess: simpler to bring up (no build-system integration with NCNN), but pipes a raw frame stream between processes — extra memcopies, extra failure mode. Probably not worth it for a real-time path.
3. ONNX Runtime: more general (could host other interpolation models down the line) but bigger binary footprint and more complex build. Not recommended unless we add multiple NN-based features.

Recommendation when this gets picked up: option 1, with a build-time flag (-DRETROCAPTURE_ENABLE_RIFE=ON) so the platforms that don't support it (currently the ARM32 Pi 3 build) just don't include the feature and the existing interpolation choices remain.

Cross-platform reality

• Linux x86_64: NCNN-Vulkan works out of the box. Build feasible.
• Windows x86_64: same, via MXE toolchain. Vulkan SDK already a build dependency for parts of the renderer.
• Linux ARM64 (Raspberry Pi 4/5): NCNN-Vulkan supports it. Inference performance is borderline — RIFE at 480p on a Pi 4 is roughly real-time, 1080p is not. Acceptable as "best effort" with a UI warning.
• Linux ARM32 (Raspberry Pi 3): not viable. Build keeps the existing modes only; RIFE option hidden in the dropdown.

UI surface

• Local capture path (new): add an Interpolation dropdown under the Source / Image tab (placement TBD during implementation). Initial entries: Off (default, preserves today's behaviour exactly) and RIFE. Linear blend could optionally be added as a middle option if it's deemed useful as a low-cost fallback for systems without RIFE.
• Remote-stream client path (existing): append RIFE to the existing dropdown in the Remote → Connect window.
• Shared sub-setting when RIFE is selected: model variant (RIFE v4.6 lite / v4.6 full / v4.13 — quality vs speed tradeoffs).
• Performance indicator: per-frame inference time visible in the Info tab so users can tell whether their hardware is keeping up.
• Graceful fallback: if RIFE fails to initialise (no Vulkan, insufficient VRAM, model files missing), the dropdown shows the option as disabled with a tooltip explaining why, and the path falls back to whatever it was doing before.

Open questions

• Should linear blend land locally as a precursor? Cheap to implement, gives a partial improvement over today's "frame held until next" behaviour, and the shader pipeline doesn't fundamentally care which of two frames it samples. Could be a separate small issue.
• Model weights distribution: ship them in the binary (~50–100 MB heavier installer), download on first enable (one-time UX hiccup), or both (small default model bundled, larger variants downloadable)? Probably bundle one default + offer downloads via GitHub Releases.
• Model variant selection: RIFE has a handful of pretrained checkpoints. Pick one default, expose the rest under an advanced setting. Need to evaluate which one balances quality and inference cost on common hardware.
• Apple Silicon / macOS: NCNN-Vulkan on Apple via MoltenVK works in principle; whether it works well needs testing once macOS becomes a build target (currently not in scope).
• Licensing fine print: RIFE source is MIT, but the pretrained models from hzwer/Practical-RIFE have their own non-commercial-friendly license terms last time we checked. Verify before distribution. If terms are incompatible, fall back to the strictly-open weights from the original RIFE paper.
• Latency on the local path: RIFE inserts a one-frame delay (we need to see frame N+1 to interpolate between N and N+1). For passive viewing this is invisible; for input-loop critical use (someone playing on the captured console while watching their own monitor through RetroCapture), it adds ~16 ms of latency. Worth a UI warning when enabling RIFE in the local capture path specifically.
• /meta advertising: should the host's /meta indicate "I'm running RIFE locally at 60 fps" so remote clients know not to double-interpolate? Probably yes; small protocol bump.
• Interaction with HDR / colour space: RIFE was trained on SDR Rec.709. Applying it to HDR content would need an out-of-band tone-map → interpolate → tone-back-up pass. Not a current concern (we don't carry HDR through the pipeline yet) but worth flagging.

Non-goals

• Replacing the existing interpolation modes on the client path. RIFE coexists as an opt-in; linear / nearest / off stay.
• Adding RIFE on the encoding side of the host (the streaming output). Information gain illusory; artifacts baked into the stream.
• Other NN-based image enhancements (super-resolution, denoising). Tempting once an NN runtime is in the binary, but each adds its own model-distribution + UX surface. Open separate issues if those become desirable.
• Real-time RIFE on ARM32 / Pi 3. Hardware not capable.

Acceptance criteria

When/if this lands, it should achieve:

• RIFE selectable as an interpolation mode on the local capture path (primary ask).
• RIFE selectable as a mode on the remote-stream client path (secondary).
• Static / vendored NCNN-Vulkan integration; no external installs required for end users on supported platforms.
• Build-time flag to disable the feature on unsupported platforms (ARM32) without affecting the rest of the build.
• At least one bundled / first-run-downloaded model variant.
• Visible per-frame inference time in the Info tab.
• Graceful fallback / disabled-with-tooltip on init failure.
• Latency warning surfaced in the UI when enabling RIFE on the local capture path (one-frame delay).
• Documented in docs/ including which sources / refresh rates benefit most, and how to read the inference-time indicator.
• No regression on existing interpolation behaviour when RIFE is disabled or unsupported.

Notes

This issue is not on any milestone. It records the idea, the tradeoffs, and the integration cost so future-us (or a contributor) doesn't have to re-do the research before deciding whether to act on it.

Original suggestion attribution: Reddit user, via post discussion thread on the RetroCapture remote-stream feature. The original wording explicitly compared RIFE to Vint and SVP, both of which target the local-realtime viewing case — that's the primary framing here.