feat: add pluggable persistence backends with storage API (#93)

* ci: add C extension code coverage with lcov

Add gcov/lcov-based coverage for the native C extension. Coverage is
compiled with -O0 --coverage flags and generates HTML reports showing
line and function coverage for vector.c, matrix.c, svd.c, and
classifier_ext.c.

New rake tasks:
- coverage:clean - remove gcov data files
- coverage:compile - build with coverage instrumentation
- coverage:report - generate HTML report via lcov
- coverage:c - full cycle (clean, compile, test, report)

CI uploads coverage reports as artifacts on each run.

* feat: add pluggable persistence backends with storage API

Implement ActiveRecord-style storage abstraction allowing classifiers
to persist to any backend. Users can configure storage at setup time
and use simple save/reload methods without path arguments.

Key additions:
- Storage::Base abstract class defining write/read/delete/exists? protocol
- Storage::Memory for testing and ephemeral use
- Storage::File for file-based persistence
- Dirty tracking with dirty? method on Bayes and LSI
- reload raises UnsavedChangesError, reload! forces discard
- load(storage:) class method sets storage on returned instance

Legacy API preserved:
- save_to_file(path) replaces old save(path)
- load_from_file(path) replaces old load(path)

Closes #90

* fix: add super calls to storage class initializers

* style: fix RuboCop offenses in storage tests

* fix: add auto_rebuild to remove_item and fix Rakefile lint

Author: cardmagic

.github/workflows/ruby.yml

@@ -52,3 +52,23 @@ jobs:
     - name: Type check with Steep
       if: matrix.ruby-version != '4.0'
       run: bundle exec steep check
+
+  coverage:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Ruby
+      uses: ruby/setup-ruby@v1
+      with:
+        ruby-version: '3.4'
+        bundler-cache: true
+    - name: Install lcov
+      run: sudo apt-get update && sudo apt-get install -y lcov
+    - name: Run C coverage
+      run: bundle exec rake coverage:c
+    - name: Upload C coverage report
+      uses: actions/upload-artifact@v4
+      with:
+        name: c-coverage-report
+        path: coverage/c/html/
+        retention-days: 30

.gitignore

@@ -9,6 +9,8 @@ pkg/
 *.bundle
 *.so
 *.o
+*.gcda
+*.gcno
 tmp/
 lib/classifier/classifier_ext.*
 Makefile

Rakefile

@@ -65,3 +65,88 @@ task :publish do
   `ssh rufy update-classifier-doc`
   Rake::RubyForgePublisher.new('classifier', 'cardmagic').upload
 end
+
+# C Code Coverage tasks
+namespace :coverage do # rubocop:disable Metrics/BlockLength
+  desc 'Clean C coverage data files'
+  task :clean do
+    FileUtils.rm_f(Dir.glob('ext/classifier/**/*.gcda'))
+    FileUtils.rm_f(Dir.glob('ext/classifier/**/*.gcno'))
+    FileUtils.rm_f(Dir.glob('tmp/**/classifier/**/*.gcda'))
+    FileUtils.rm_f(Dir.glob('tmp/**/classifier/**/*.gcno'))
+    FileUtils.rm_rf('coverage/c')
+  end
+
+  desc 'Compile C extension with coverage instrumentation'
+  task :compile do
+    ENV['COVERAGE'] = '1'
+    Rake::Task['clobber'].invoke if Rake::Task.task_defined?('clobber')
+    Rake::Task['compile'].reenable
+    Rake::Task['compile'].invoke
+  end
+
+  desc 'Generate C coverage report using lcov'
+  task :report do # rubocop:disable Metrics/BlockLength
+    project_root = File.expand_path(__dir__)
+    ext_dir = File.join(project_root, 'ext/classifier')
+    # Find the directory containing .gcda files (build directory varies by platform/Ruby version)
+    tmp_ext_dir = Dir.glob('tmp/**/classifier_ext/**/*.gcda').first&.then { |f| File.dirname(f) }
+    coverage_dir = 'coverage/c'
+
+    unless tmp_ext_dir
+      puts 'No coverage data found. Run tests with coverage first.'
+      next
+    end
+
+    FileUtils.mkdir_p(coverage_dir)
+
+    # Run gcov manually in the build directory to generate .gcov files
+    Dir.chdir(tmp_ext_dir) do
+      # Find all source files and run gcov on them
+      gcda_files = Dir.glob('*.gcda')
+      gcda_files.each do |gcda|
+        # Source file is in ext/classifier, referenced via relative path in the gcno
+        sh "gcov -o . #{gcda} 2>/dev/null || true"
+      end
+    end
+
+    # Capture coverage data with base directory for source resolution
+    sh "lcov --capture --directory #{tmp_ext_dir} --base-directory #{ext_dir} " \
+       "--output-file #{coverage_dir}/coverage.info " \
+       '--ignore-errors inconsistent,gcov,source 2>&1 || true'
+
+    if File.exist?("#{coverage_dir}/coverage.info") && File.size("#{coverage_dir}/coverage.info").positive?
+      # Filter out system headers
+      sh "lcov --remove #{coverage_dir}/coverage.info '/usr/*' '*/ruby/*' " \
+         "--output-file #{coverage_dir}/coverage.info --ignore-errors unused 2>/dev/null || true"
+
+      # Fix source paths: the gcov relative paths resolve incorrectly
+      # Substitute wrong paths with correct absolute paths
+      info_content = File.read("#{coverage_dir}/coverage.info")
+      info_content.gsub!(%r{SF:.*/ext/classifier/}, "SF:#{ext_dir}/")
+      File.write("#{coverage_dir}/coverage.info", info_content)
+
+      # Generate HTML report
+      sh "genhtml #{coverage_dir}/coverage.info --output-directory #{coverage_dir}/html " \
+         "--prefix #{project_root} --ignore-errors unmapped,source 2>&1 || true"
+
+      puts "\nC coverage report generated at: #{coverage_dir}/html/index.html"
+
+      # Print summary
+      sh "lcov --summary #{coverage_dir}/coverage.info 2>/dev/null || true"
+    else
+      puts 'Coverage data capture failed. Check that tests exercise the C extension.'
+    end
+  end
+
+  desc 'Run tests and generate C coverage report'
+  task :run do
+    Rake::Task['coverage:clean'].invoke
+    Rake::Task['coverage:compile'].invoke
+    Rake::Task['test'].invoke
+    Rake::Task['coverage:report'].invoke
+  end
+end
+
+desc 'Run C code coverage (alias for coverage:run)'
+task 'coverage:c' => 'coverage:run'

ext/classifier/extconf.rb

@@ -1,7 +1,15 @@
 require 'mkmf'
 
-# Optimization flags for performance
-$CFLAGS << ' -O3 -ffast-math -Wall' # rubocop:disable Style/GlobalVars
+# rubocop:disable Style/GlobalVars
+if ENV['COVERAGE']
+  # Coverage flags: disable optimization for accurate line coverage
+  $CFLAGS << ' -O0 -g --coverage -Wall'
+  $LDFLAGS << ' --coverage'
+else
+  # Optimization flags for performance
+  $CFLAGS << ' -O3 -ffast-math -Wall'
+end
+# rubocop:enable Style/GlobalVars
 
 # Create the Makefile
 create_makefile('classifier/classifier_ext')

lib/classifier.rb

@@ -25,6 +25,8 @@
 # License::   LGPL
 
 require 'rubygems'
+require 'classifier/errors'
+require 'classifier/storage'
 require 'classifier/extensions/string'
 require 'classifier/extensions/vector'
 require 'classifier/bayes'

lib/classifier/bayes.rb

@@ -17,6 +17,10 @@ class Bayes
     # @rbs @category_word_count: Hash[Symbol, Integer]
     # @rbs @cached_training_count: Float?
     # @rbs @cached_vocab_size: Integer?
+    # @rbs @dirty: bool
+    # @rbs @storage: Storage::Base?
+
+    attr_accessor :storage
 
     # The class can be created with one or more categories, each of which will be
     # initialized and given a training method. E.g.,
@@ -31,6 +35,8 @@ def initialize(*categories)
       @category_word_count = Hash.new(0)
       @cached_training_count = nil
       @cached_vocab_size = nil
+      @dirty = false
+      @storage = nil
     end
 
     # Provides a general training method for all categories specified in Bayes#new
@@ -46,6 +52,7 @@ def train(category, text)
       word_hash = text.word_hash
       synchronize do
         invalidate_caches
+        @dirty = true
         @category_counts[category] += 1
         word_hash.each do |word, count|
           @categories[category][word] ||= 0
@@ -70,6 +77,7 @@ def untrain(category, text)
       word_hash = text.word_hash
       synchronize do
         invalidate_caches
+        @dirty = true
         @category_counts[category] -= 1
         word_hash.each do |word, count|
           next unless @total_words >= 0
@@ -159,17 +167,81 @@ def self.from_json(json)
       instance
     end
 
-    # Saves the classifier state to a file.
+    # Saves the classifier to the configured storage.
+    # Raises ArgumentError if no storage is configured.
+    #
+    # @rbs () -> void
+    def save
+      raise ArgumentError, 'No storage configured. Use save_to_file(path) or set storage=' unless storage
+
+      storage.write(to_json)
+      @dirty = false
+    end
+
+    # Saves the classifier state to a file (legacy API).
     #
     # @rbs (String) -> Integer
-    def save(path)
-      File.write(path, to_json)
+    def save_to_file(path)
+      result = File.write(path, to_json)
+      @dirty = false
+      result
+    end
+
+    # Reloads the classifier from the configured storage.
+    # Raises UnsavedChangesError if there are unsaved changes.
+    # Use reload! to force reload and discard changes.
+    #
+    # @rbs () -> self
+    def reload
+      raise ArgumentError, 'No storage configured' unless storage
+      raise UnsavedChangesError, 'Unsaved changes would be lost. Call save first or use reload!' if @dirty
+
+      data = storage.read
+      raise StorageError, 'No saved state found' unless data
+
+      restore_from_json(data)
+      @dirty = false
+      self
+    end
+
+    # Force reloads the classifier from storage, discarding any unsaved changes.
+    #
+    # @rbs () -> self
+    def reload!
+      raise ArgumentError, 'No storage configured' unless storage
+
+      data = storage.read
+      raise StorageError, 'No saved state found' unless data
+
+      restore_from_json(data)
+      @dirty = false
+      self
+    end
+
+    # Returns true if there are unsaved changes.
+    #
+    # @rbs () -> bool
+    def dirty?
+      @dirty
     end
 
-    # Loads a classifier from a file saved with #save.
+    # Loads a classifier from the configured storage.
+    # The storage is set on the returned instance.
+    #
+    # @rbs (storage: Storage::Base) -> Bayes
+    def self.load(storage:)
+      data = storage.read
+      raise StorageError, 'No saved state found' unless data
+
+      instance = from_json(data)
+      instance.storage = storage
+      instance
+    end
+
+    # Loads a classifier from a file (legacy API).
     #
     # @rbs (String) -> Bayes
-    def self.load(path)
+    def self.load_from_file(path)
       from_json(File.read(path))
     end
 
@@ -219,6 +291,7 @@ def categories
     def add_category(category)
       synchronize do
         invalidate_caches
+        @dirty = true
         @categories[category.prepare_category_name] = {}
       end
     end
@@ -228,16 +301,17 @@ def add_category(category)
     # Custom marshal serialization to exclude mutex state
     # @rbs () -> Array[untyped]
     def marshal_dump
-      [@categories, @total_words, @category_counts, @category_word_count]
+      [@categories, @total_words, @category_counts, @category_word_count, @dirty]
     end
 
     # Custom marshal deserialization to recreate mutex
     # @rbs (Array[untyped]) -> void
     def marshal_load(data)
       mu_initialize
-      @categories, @total_words, @category_counts, @category_word_count = data
+      @categories, @total_words, @category_counts, @category_word_count, @dirty = data
       @cached_training_count = nil
       @cached_vocab_size = nil
+      @storage = nil
     end
 
     # Allows you to remove categories from the classifier.
@@ -255,6 +329,7 @@ def remove_category(category)
         raise StandardError, "No such category: #{category}" unless @categories.key?(category)
 
         invalidate_caches
+        @dirty = true
         @total_words -= @category_word_count[category].to_i
 
         @categories.delete(category)
@@ -265,6 +340,17 @@ def remove_category(category)
 
     private
 
+    # Restores classifier state from a JSON string (used by reload)
+    # @rbs (String) -> void
+    def restore_from_json(json)
+      data = JSON.parse(json)
+      raise ArgumentError, "Invalid classifier type: #{data['type']}" unless data['type'] == 'bayes'
+
+      synchronize do
+        restore_state(data)
+      end
+    end
+
     # Restores classifier state from a hash (used by from_json)
     # @rbs (Hash[String, untyped]) -> void
     def restore_state(data)
@@ -275,6 +361,8 @@ def restore_state(data)
       @category_word_count = Hash.new(0) #: Hash[Symbol, Integer]
       @cached_training_count = nil
       @cached_vocab_size = nil
+      @dirty = false
+      @storage = nil
 
       data['categories'].each do |cat_name, words|
         @categories[cat_name.to_sym] = words.transform_keys(&:to_sym)

lib/classifier/errors.rb

@@ -0,0 +1,16 @@
+# rbs_inline: enabled
+
+# Author::    Lucas Carlson  (mailto:[email protected])
+# Copyright:: Copyright (c) 2005 Lucas Carlson
+# License::   LGPL
+
+module Classifier
+  # Base error class for all Classifier errors
+  class Error < StandardError; end
+
+  # Raised when reload would discard unsaved changes
+  class UnsavedChangesError < Error; end
+
+  # Raised when a storage operation fails
+  class StorageError < Error; end
+end