feat: vibes branch consolidation (48 commits)

CLAUDE.md

@@ -16,34 +16,43 @@ This is a Python-based Reddit moderation log publisher that automatically scrape
 
 ## Development Commands
 
+**IMPORTANT**: Always use `/opt/.venv/redditbot/bin/python` for all Python commands in this project.
+
 ### Setup and Dependencies
 ```bash
-# Install dependencies
-pip install praw
-
+# Dependencies are pre-installed in the venv
 # Copy template config (required for first run)
 cp config_template.json config.json
 ```
 
 ### Running the Application
 ```bash
 # Test connection and configuration
-python modlog_wiki_publisher.py --test
+/opt/.venv/redditbot/bin/python modlog_wiki_publisher.py --test
 
 # Single run
-python modlog_wiki_publisher.py --source-subreddit SUBREDDIT_NAME
+/opt/.venv/redditbot/bin/python modlog_wiki_publisher.py --source-subreddit SUBREDDIT_NAME
 
 # Continuous daemon mode
-python modlog_wiki_publisher.py --source-subreddit SUBREDDIT_NAME --continuous
+/opt/.venv/redditbot/bin/python modlog_wiki_publisher.py --source-subreddit SUBREDDIT_NAME --continuous
+
+# Force wiki update only (using existing database data)
+/opt/.venv/redditbot/bin/python modlog_wiki_publisher.py --source-subreddit SUBREDDIT_NAME --force-wiki
 
 # Debug authentication issues
-python debug_auth.py
+/opt/.venv/redditbot/bin/python debug_auth.py
 ```
 
 ### Database Operations
 ```bash
-# View recent processed actions
-sqlite3 modlog.db "SELECT * FROM processed_actions ORDER BY created_at DESC LIMIT 10;"
+# View recent processed actions with removal reasons
+sqlite3 modlog.db "SELECT action_id, action_type, moderator, removal_reason, subreddit, created_at FROM processed_actions ORDER BY created_at DESC LIMIT 10;"
+
+# View actions by subreddit
+sqlite3 modlog.db "SELECT action_type, moderator, target_author, removal_reason FROM processed_actions WHERE subreddit = 'usenet' ORDER BY created_at DESC LIMIT 5;"
+
+# Track content lifecycle by target ID  
+sqlite3 modlog.db "SELECT target_id, action_type, moderator, removal_reason, datetime(created_at, 'unixepoch') FROM processed_actions WHERE target_id LIKE '%1mkz4jm%' ORDER BY created_at;"
 
 # Manual cleanup of old entries
 sqlite3 modlog.db "DELETE FROM processed_actions WHERE created_at < date('now', '-30 days');"
@@ -53,13 +62,26 @@ sqlite3 modlog.db "DELETE FROM processed_actions WHERE created_at < date('now',
 
 The application supports both JSON config files and CLI arguments (CLI overrides JSON):
 
+### Core Options
 - `--source-subreddit`: Target subreddit for reading/writing logs
 - `--wiki-page`: Wiki page name (default: "modlog")
 - `--retention-days`: Database cleanup period (default: 30)
 - `--batch-size`: Entries fetched per run (default: 100)
 - `--interval`: Seconds between updates in daemon mode (default: 300)
 - `--debug`: Enable verbose logging
 
+### Display Options
+- `anonymize_moderators`: Whether to show "HumanModerator" for human mods (default: true)
+  - `true` (default): Shows "AutoMod", "Reddit", or "HumanModerator"
+  - `false`: Shows actual moderator usernames
+
+### Database Features
+- **Multi-subreddit support**: Single database handles multiple subreddits safely
+- **Removal reason storage**: Full text/number handling from Reddit API
+- **Target author tracking**: Actual usernames stored and displayed
+- **Content ID extraction**: Unique IDs from permalinks for precise tracking
+- **Data separation**: Subreddit column prevents cross-contamination
+
 ## Authentication Requirements
 
 The bot account needs:
@@ -80,8 +102,70 @@ The bot account needs:
 
 Use `--test` flag to verify configuration and Reddit API connectivity without making changes.
 
+## Content Link Guidelines
+
+**CRITICAL**: Content links in the modlog should NEVER point to user profiles (`/u/username`). Links should only point to:
+- Actual removed posts (`/comments/postid/`)  
+- Actual removed comments (`/comments/postid/_/commentid/`)
+- No link at all if no actual content is available
+
+User profile links are a privacy concern and not useful for modlog purposes.
+
+## Recent Improvements (v2.1)
+
+### Multi-Subreddit Database Support
+- ✅ Fixed critical error that prevented multi-subreddit databases from working
+- ✅ Single database now safely handles multiple subreddits with proper data separation
+- ✅ Per-subreddit wiki updates without cross-contamination
+- ✅ Subreddit-specific logging and error handling
+
+### Removal Reason Transparency
+- ✅ Fixed "Removal reason applied" showing instead of actual text
+- ✅ Full transparency - shows ALL available removal reason data including template numbers
+- ✅ Consistent handling between storage and display logic using correct Reddit API fields
+- ✅ Displays actual removal reasons like "Invites - No asking", "This comment has been filtered due to crowd control"
+
+### Unique Content ID Tracking
+- ✅ Fixed duplicate IDs in markdown tables where all comments showed same post ID
+- ✅ Comments now show unique comment IDs (e.g., "n7ravg2") for precise tracking
+- ✅ Posts show post IDs for clear content identification
+- ✅ Each modlog entry has a unique identifier for easy reference
+
+### Content Linking and Display
+- ✅ Content links point to actual Reddit posts/comments, never user profiles for privacy
+- ✅ Fixed target authors showing as [deleted] - now displays actual usernames  
+- ✅ Proper content titles extracted from Reddit API data
+- ✅ AutoModerator displays as "AutoModerator" (not anonymized)
+- ✅ Configurable anonymization for human moderators
+
+### Data Integrity
+- ✅ Pipe character escaping for markdown table compatibility
+- ✅ Robust error handling for mixed subreddit scenarios  
+- ✅ Database schema at version 5 with all required columns
+- ✅ Consistent Reddit API field usage (action.details vs action.description)
+
+## Development Guidelines
+
+### Git Workflow
+- If branch is not main, you may commit and push if a PR is draft or not open
+- Use conventional commits for all changes
+- Use multiple commits if needed, or patch if easier
+- Always update CLAUDE.md and README.md when making changes
+
+### Code Standards
+- Always escape markdown table values like removal reasons for pipes
+- Store pipe-free data in database to prevent markdown issues
+- Confirm cache file of wiki page and warn if same, interactively ask to force refresh
+- Always use the specified virtual environment path
+
+### Documentation
+- Always update commands and flags in documentation
+- Remove CHANGELOG from CLAUDE.md (keep separate)
+- Create and update changelog based on git tags (should be scripted)
+
 ## Common Issues
 
-- 401 errors: Check app type is "script" and verify client_id/client_secret
-- Wiki permission denied: Ensure bot has moderator or wiki contributor access
-- Rate limiting: Increase `--interval` and/or reduce `--batch-size`
+- **401 errors**: Check app type is "script" and verify client_id/client_secret
+- **Wiki permission denied**: Ensure bot has moderator or wiki contributor access
+- **Rate limiting**: Increase `--interval` and/or reduce `--batch-size`
+- **Module not found**: Always use `/opt/.venv/redditbot/bin/python` instead of system python

README.md

@@ -4,14 +4,19 @@ Automatically publishes Reddit moderation logs to a subreddit wiki page with mod
 
 ## Features
 
-* 📊 Publishes modlogs as organized markdown tables
-* 📧 Pre-populated modmail links for removal inquiries
-* 🗄️ SQLite database for deduplication and retention
-* ⏰ Configurable update intervals
-* 🔒 Automatic cleanup of old entries
-* ⚡ Handles Reddit's 524KB wiki size limit
+* 📊 Publishes modlogs as organized markdown tables with unique content tracking IDs
+* 📧 Pre-populated modmail links for removal inquiries (formatted as clickable markdown links)
+* 🗄️ SQLite database for deduplication and retention with **multi-subreddit support**
+* ⏰ Configurable update intervals with continuous daemon mode
+* 🔒 Automatic cleanup of old entries with configurable retention
+* ⚡ Handles Reddit's 524KB wiki size limit automatically  
 * 🧩 Fully CLI-configurable (no need to edit `config.json`)
-* 📁 Per-subreddit log files for debugging
+* 📁 Per-subreddit log files for debugging and monitoring
+* 🔒 Configurable moderator anonymization (AutoModerator/HumanModerator)
+* 📝 **Complete removal reason transparency** - AutoModerator rule text, addremovalreason descriptions, all actual removal text (never generic messages or template numbers)
+* 🔗 Links directly to actual content (posts/comments), never user profiles for privacy
+* 🆔 **Unique content IDs** - comments show comment IDs, posts show post IDs for precise tracking
+* ✅ **Multi-subreddit database support** - single database handles multiple subreddits safely
 
 ## Quick Start
 
@@ -67,33 +72,52 @@ Create `config.json`:
   "ignored_moderators": ["AutoModerator"],
   "update_interval": 300,
   "batch_size": 100,
-  "retention_days": 30
+  "retention_days": 30,
+  "anonymize_moderators": true
 }
 ```
 
 ### Configurable via CLI
 
-| CLI Option           | JSON Key           | Description                            | Default       |
-| -------------------- | ------------------ | -------------------------------------- | ------------- |
-| `--source-subreddit` | `source_subreddit` | Subreddit to read and write logs       | required      |
-| `--wiki-page`        | `wiki_page`        | Wiki page name                         | `modlog`      |
-| `--retention-days`   | `retention_days`   | Keep entries this many days            | `30`          |
-| `--batch-size`       | `batch_size`       | Entries to fetch per run               | `100`         |
-| `--interval`         | `update_interval`  | Seconds between updates in daemon mode | `300`         |
-| `--config`           | –                  | Path to config file                    | `config.json` |
+| CLI Option | JSON Key | Description | Default | Min | Max |
+|------------|----------|-------------|---------|-----|-----|
+| `--source-subreddit` | `source_subreddit` | Subreddit to read and write logs | required | - | - |
+| `--wiki-page` | `wiki_page` | Wiki page name | modlog | - | - |
+| `--retention-days` | `retention_days` | Keep entries this many days | 90 | 1 | 365 |
+| `--batch-size` | `batch_size` | Entries to fetch per run | 50 | 10 | 500 |
+| `--interval` | `update_interval` | Seconds between updates in daemon mode | 600 | 60 | 3600 |
+| `--config` | – | Path to config file | config.json | - | - |
+| `--debug` | – | Enable verbose output | false | - | - |
+| `--show-config-limits` | – | Show configuration limits and defaults | false | - | - |
+| `--force-migrate` | – | Force database migration | false | - | - |
+| `--no-auto-update-config` | – | Disable automatic config file updates | false | - | - |
 
 CLI values override config file values.
 
+## Configuration Limits
+
+All configuration values are automatically validated and enforced within safe limits. Use `--show-config-limits` to see current limits and defaults.
+
+## Automatic Config Updates
+
+The application automatically updates your config file when new configuration options are added, while preserving your existing settings. A backup is created before any changes. Use `--no-auto-update-config` to disable this behavior.
+
+## Database Migration
+
+The database will automatically migrate to the latest schema version on startup. Use `--force-migrate` to manually trigger migration.
+
 ## Wiki Output
 
 Sample wiki table output:
 
 ```markdown
-## 2025-01-15
+## 2025-08-09
 
-| Time | Action | Moderator | Content | Reason | Inquire |
-|------|--------|-----------|---------|--------|---------|
-| 14:25:33 UTC | removepost | ModName | [Post Title](url) | spam | [Contact Mods](modmail_url) |
+| Time | Action | ID | Moderator | Content | Reason | Inquire |
+|------|--------|----|-----------|---------|--------|---------|
+| 08:15:42 UTC | removecomment | n7ravg2 | AutoModerator | [Comment by u/user123](https://www.reddit.com/r/opensignups/comments/1ab2cd3/title/n7ravg2/) | Possibly requesting an invite - [invited] Offers must be [O] 3x Invites to MyAwesomeTracker | [Contact Mods](https://www.reddit.com/message/compose?to=/r/opensignups&subject=Comment%20Removal%20Inquiry...) |
+| 07:45:18 UTC | addremovalreason | 1ab2cd3 | Bakerboy448 | [Post title here](https://www.reddit.com/r/opensignups/comments/1ab2cd3/title/) | Invites - No asking | [Contact Mods](https://www.reddit.com/message/compose?to=/r/opensignups&subject=Removal%20Reason%20Inquiry...) |
+| 06:32:15 UTC | removelink | 1xy9def | AutoModerator | [Another post](https://www.reddit.com/r/opensignups/comments/1xy9def/another/) | No standalone URL in post body | [Contact Mods](https://www.reddit.com/message/compose?to=/r/opensignups&subject=Post%20Removal%20Inquiry...) |
 ```
 
 ## Logging
@@ -122,20 +146,76 @@ Options:
   --debug                  Enable debug logging
   --test                   Run a test and exit
   --continuous             Run continuously
+  --force-modlog           Fetch ALL actions from Reddit API and rebuild wiki
+  --force-wiki             Update wiki even if content appears unchanged
+  --force-all              Do both --force-modlog and --force-wiki
+```
+
+### Force Commands Explained
+
+**--force-modlog**: Complete rebuild from Reddit
+- Fetches ALL recent modlog actions from Reddit API 
+- Stores them in database
+- Rebuilds entire wiki page from database
+- Use when: Starting fresh, major updates, or troubleshooting
+
+**--force-wiki**: Force wiki update only
+- Uses existing database data
+- Forces wiki update even if content hash matches
+- Use when: Format changes, modmail updates, or cache issues
+
+**--force-all**: Complete refresh (replaces old --force)
+- Combines both --force-modlog and --force-wiki
+- Fetches from Reddit AND forces wiki update
+- Use when: Major changes, troubleshooting, or unsure which force to use
+
+```bash
+# Complete rebuild from Reddit API
+python modlog_wiki_publisher.py --source-subreddit usenet --force-modlog
+
+# Update wiki with current database data (bypass cache)  
+python modlog_wiki_publisher.py --source-subreddit usenet --force-wiki
+
+# Do both (equivalent to old --force)
+python modlog_wiki_publisher.py --source-subreddit usenet --force-all
 ```
 
 ## Database
 
 Uses `modlog.db` (SQLite) for deduplication and history:
 
 ```bash
-# View recent actions
+# View recent actions with removal reasons
+sqlite3 modlog.db "SELECT action_id, action_type, moderator, removal_reason, subreddit, created_at FROM processed_actions ORDER BY created_at DESC LIMIT 10;"
+
+# View all columns including removal reasons and target author
 sqlite3 modlog.db "SELECT * FROM processed_actions ORDER BY created_at DESC LIMIT 10;"
 
+# View actions by subreddit
+sqlite3 modlog.db "SELECT action_type, moderator, target_author, removal_reason FROM processed_actions WHERE subreddit = 'usenet' ORDER BY created_at DESC LIMIT 5;"
+
+# Track content lifecycle by target ID
+sqlite3 modlog.db "SELECT target_id, action_type, moderator, removal_reason, datetime(created_at, 'unixepoch') FROM processed_actions WHERE target_id LIKE '%1mkz4jm%' ORDER BY created_at;"
+
+# View removal reasons that are text (not numbers)
+sqlite3 modlog.db "SELECT action_type, removal_reason FROM processed_actions WHERE removal_reason NOT LIKE '%[0-9]%' AND removal_reason != 'remove' LIMIT 5;"
+
 # Clean manually
 sqlite3 modlog.db "DELETE FROM processed_actions WHERE created_at < date('now', '-30 days');"
 ```
 
+### Database Schema
+
+The database includes comprehensive moderation data with full transparency:
+
+- **`removal_reason` column**: Stores actual removal reason text from Reddit's API
+  - AutoModerator actions: Full rule text (e.g., "Possibly requesting an invite - [invited] Offers must be [O]")
+  - addremovalreason actions: Readable removal reason (e.g., "Invites - No asking") instead of template numbers
+  - Manual removals: Moderator-provided text or rule details
+- **`target_author` column**: Actual usernames of content authors (never shows [deleted])
+- **`subreddit` column**: Multi-subreddit support with proper data separation
+- **Unique content IDs**: Comments show comment IDs (e.g., n7ravg2), posts show post IDs
+
 ## Systemd Service (Optional)
 
 ```ini

config_template.json

@@ -6,10 +6,18 @@
     "password": "YOUR_BOT_PASSWORD"
   },
   "source_subreddit": "YourSubreddit",
-  "target_subreddit": "YourSubreddit",
   "wiki_page": "modlog",
-  "ignored_moderators": ["AutoModerator", "BotDefense"],
-  "update_interval": 300,
-  "batch_size": 100,
-  "retention_days": 30
+  "retention_days": 90,
+  "batch_size": 50,
+  "update_interval": 600,
+  "max_wiki_entries_per_page": 1000,
+  "max_continuous_errors": 5,
+  "rate_limit_buffer": 60,
+  "max_batch_retries": 3,
+  "archive_threshold_days": 7,
+  "ignored_moderators": ["AutoModerator"],
+  "display_format": {
+    "show_full_ids": false,
+    "id_format": "prefixed"
+  }
 }