HF writes incur heavy context switching

[greged93]

I'm monitoring 3 log files on a Linux host machine, using the RecommendedWatcher. Writes on the log files happen at HF (around 10k-20k per second). This leads to a very high amount of context switches and interrupts, degrading the performance of my host machine.
I realize there probably isn't a solution for me here (and that I should have expected this before trying to use inotify), but might be nice to document this somewhere as a gotcha.

Below is a screenshot of the context switches, interrupts and CPU schedule timeslices. The high value occur when running the below, low ranges when the watcher is shutdown.

        let (fs_event_tx, watcher_rx) = tokio::sync::mpsc::unbounded_channel();
        let mut watcher = recommended_watcher(move |res| {
            let fs_event_tx = fs_event_tx.clone();
            if let Err(err) = fs_event_tx.send(res) {
                error!(target: "feed::file_watcher", ?err, "Failed to send event from recommended watcher");
            }
        })?;

        for source in EventSource::iter() {
            let dir = source.event_source_dir(&dir, batch).canonicalize()?;
            info!(target: "feed::file_watcher", ?dir, "watching file");
            watcher.watch(&dir, RecursiveMode::Recursive)?;
        }

 loop {
            tokio::select! {
                event = watcher_rx.recv() => match event {
                    Some(Ok(event)) => (),
                    Some(Err(err)) => {
                        log_and_panic(&format!("Notify error: {err}"));
                    }
                    None => {
                        log_and_panic("Watcher channel closed");
                    }
                },
                () = tokio::time::sleep(Duration::from_secs(15)) => {
                    log_and_panic("Stream has fallen behind (node failed?)");
                }

            }
        }

[JohnTitor]

I think some parts of the issue are on backends and we cannot address, but maybe there could be a place to improve our implementation.
One stopgap would be filtering events by EventKindMask.