feat(server): Add GitHub repository size check before processing - #1412
feat(server): Add GitHub repository size check before processing#1412yamadashy wants to merge 3 commits into
Conversation
Check repository size via GitHub API before downloading and processing. Repositories exceeding 500MB are rejected with a 422 error to prevent resource exhaustion on the server. The check runs after cache lookup (to avoid unnecessary API calls) and fails open — if the GitHub API is unreachable or the repo is non-GitHub, processing continues normally. Also exports `parseGitHubRepoInfo` and `GitHubRepoInfo` from the repomix package to support server-side repository URL parsing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
⚡ Performance Benchmark
Details
Historyce37173 feat(server): Add GITHUB_TOKEN support for repo size check API
b2257d5 feat(server): Add GitHub repository size check before processing
|
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe PR extends the module's public API by exporting Git repository parsing utilities and introduces GitHub repository size validation during remote repository processing to prevent large repositories from being packed. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a size check for GitHub repositories before they are processed, setting a maximum limit of 500MB. The implementation includes a new utility to fetch repository metadata via the GitHub API. Feedback highlights that the specific 422 error thrown for oversized repositories may be swallowed by existing generic error handling in the calling function, and suggests using authenticated API requests to avoid strict rate limits on public-facing servers.
| } | ||
|
|
||
| // Check repository size before processing | ||
| await checkGitHubRepoSize(repoUrl); |
There was a problem hiding this comment.
The AppError thrown by checkGitHubRepoSize (with a 422 status) will be caught by the generic catch block at the end of this function (line 109). That block wraps all errors in a new AppError with a 500 status and a generic message about the repository not being public. This defeats the purpose of providing a specific 422 error for oversized repositories. You should update the catch block at line 109 to check if the error is already an AppError and re-throw it if so.
| const response = await fetch(url, { | ||
| headers: { | ||
| Accept: 'application/vnd.github.v3+json', | ||
| 'User-Agent': 'Repomix', | ||
| }, | ||
| signal: AbortSignal.timeout(GITHUB_API_TIMEOUT_MS), | ||
| }); |
There was a problem hiding this comment.
Unauthenticated requests to the GitHub API are subject to a strict rate limit (60 requests per hour per IP). On a public-facing server, this limit can be exhausted quickly, causing the size check to "fail open" and return null for most requests. Consider using an authenticated request if a GitHub token is available in the environment variables (e.g., by adding an Authorization: token <TOKEN> header).
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1412 +/- ##
==========================================
- Coverage 87.42% 87.26% -0.17%
==========================================
Files 116 117 +1
Lines 4397 4420 +23
Branches 1020 1021 +1
==========================================
+ Hits 3844 3857 +13
- Misses 553 563 +10 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Code Review: feat(server): Add GitHub repository size check before processingOverall: Clean, well-scoped PR that adds a sensible safeguard against processing oversized GitHub repos. The fail-open design and placement after cache lookup are good architectural choices. A few items worth considering below. Noteworthy Findings1. Unauthenticated GitHub API rate limit (60 req/hour) could be exhausted under moderate load The size check uses unauthenticated GitHub API requests, which are limited to 60/hour per IP. Under sustained traffic, most requests will hit rate limits and fall through to fail-open — effectively disabling the size check. Consider supporting an optional 2. No tests included The PR adds a new utility file with non-trivial logic (API calls, error handling, size comparison) but includes no tests. Key test cases to consider:
The codebase has established patterns for mocking 3. Missing dependency injection (project convention)Per CLAUDE.md guidelines, dependencies should be injected through a export const checkGitHubRepoSize = async (
repoUrl: string,
deps = { parseGitHubRepoInfo, fetchGitHubRepoSize },
): Promise<void> => {
// ...
};This follows the pattern used in 4. Silent error swallowing — consider trace-level loggingThe catch (error) {
logger.trace(`Failed to fetch GitHub repo size: ${error}`);
return null;
}5. GitHub API `size` field accuracy caveatGitHub's 6. Timeout could be reducedThe 10-second timeout is generous — GitHub API typically responds in <500ms. A 3-5 second timeout would release resources faster during API outages while still providing ample margin for normal responses. What looks good
🤖 Generated with Claude Code |
Use GITHUB_TOKEN environment variable when calling GitHub API to increase rate limit from 60 to 5,000 requests per hour. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Deploying repomix with
|
| Latest commit: |
ae43543
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://4558d743.repomix.pages.dev |
| Branch Preview URL: | https://feat-server-github-repo-size.repomix.pages.dev |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@website/server/src/domains/pack/utils/gitHubRepoSize.ts`:
- Line 1: Remove the named type import GitHubRepoInfo from the repomix import
and instead derive the type locally from parseGitHubRepoInfo; replace the import
line to only import parseGitHubRepoInfo and add a local type alias such as type
GitHubRepoInfo = NonNullable<ReturnType<typeof parseGitHubRepoInfo>> (and update
any usages in this file to use that local GitHubRepoInfo type) so the code no
longer relies on a non-exported declaration.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 4a196284-7c03-4f77-8e0c-72afaf5cd3a7
📒 Files selected for processing (3)
src/index.tswebsite/server/src/domains/pack/remoteRepo.tswebsite/server/src/domains/pack/utils/gitHubRepoSize.ts
| @@ -0,0 +1,70 @@ | |||
| import { type GitHubRepoInfo, parseGitHubRepoInfo } from 'repomix'; | |||
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "1) Verify failing import site"
rg -n "import \\{\\s*type\\s+GitHubRepoInfo\\s*,\\s*parseGitHubRepoInfo\\s*\\} from 'repomix'" --type ts
echo "2) Verify root package source re-export"
rg -n "export type \\{\\s*GitHubRepoInfo\\s*\\}" src/index.ts
echo "3) Verify source declaration/export exists"
rg -n "export (interface|type) GitHubRepoInfo" src/core/git/gitRemoteParse.ts
echo "4) If declaration files are present, verify public d.ts actually contains it"
fd -i "index.d.ts" | xargs -r rg -n "GitHubRepoInfo"Repository: yamadashy/repomix
Length of output: 483
🏁 Script executed:
# Check if there are any TypeScript errors in the actual file
cat -n website/server/src/domains/pack/utils/gitHubRepoSize.ts
# Check if the file actually uses GitHubRepoInfo to confirm it's needed
rg -A 5 "GitHubRepoInfo" website/server/src/domains/pack/utils/gitHubRepoSize.ts
# Verify the package.json exports/types configuration
cat package.json | head -50Repository: yamadashy/repomix
Length of output: 5778
🏁 Script executed:
# Check if lib directory and declaration file exist
ls -la lib/ 2>/dev/null | head -20 || echo "lib/ directory does not exist"
# Check if lib/index.d.ts has GitHubRepoInfo export
if [ -f lib/index.d.ts ]; then
echo "=== Checking lib/index.d.ts for GitHubRepoInfo ==="
rg "GitHubRepoInfo" lib/index.d.ts
else
echo "lib/index.d.ts does not exist"
fi
# Verify tsconfig.build.json to understand what gets compiled
echo "=== tsconfig.build.json ==="
cat tsconfig.build.json 2>/dev/null | head -30Repository: yamadashy/repomix
Length of output: 314
🏁 Script executed:
# Check website package.json to see how it imports repomix
cat website/server/package.json | grep -A 5 -B 5 "repomix" || echo "No repomix reference in website package.json"
# Check if there's a monorepo config (pnpm-workspace.yaml, lerna.json, etc.)
ls -la | grep -E "(pnpm-workspace|lerna|workspaces)"
# Check the root package.json workspaces config
cat package.json | grep -A 10 "workspaces"Repository: yamadashy/repomix
Length of output: 513
🏁 Script executed:
# Check .gitignore to confirm lib/ is excluded
cat .gitignore | grep -E "^lib|^/lib"
# Verify: src/index.ts DOES export GitHubRepoInfo
echo "=== Verifying src/index.ts export ==="
grep -n "GitHubRepoInfo" src/index.ts
# Check src/core/git/gitRemoteParse.ts to see the actual type definition
echo "=== Type definition in source ==="
sed -n '5,20p' src/core/git/gitRemoteParse.tsRepository: yamadashy/repomix
Length of output: 765
🏁 Script executed:
# Verify that parseGitHubRepoInfo return type allows null (to confirm NonNullable is needed)
echo "=== Check parseGitHubRepoInfo signature ==="
rg -A 5 "export.*parseGitHubRepoInfo" src/core/git/gitRemoteParse.ts | head -20
# Check if lib/index.d.ts gets regenerated during build
echo "=== Check if build script creates lib/ ==="
cat package.json | grep -A 2 '"build"'
# Verify the actual return type by checking function definition
echo "=== Full parseGitHubRepoInfo function ==="
rg -B 2 "export const parseGitHubRepoInfo" src/core/git/gitRemoteParse.tsRepository: yamadashy/repomix
Length of output: 806
Remove GitHubRepoInfo from the repomix import and derive the type locally from parseGitHubRepoInfo.
The type GitHubRepoInfo is not exposed in the compiled declaration files (lib/index.d.ts), causing TS2305. The suggested approach using NonNullable<ReturnType<typeof parseGitHubRepoInfo>> is more robust as it derives the type directly from the function signature rather than relying on the re-export.
Suggested fix
-import { type GitHubRepoInfo, parseGitHubRepoInfo } from 'repomix';
+import { parseGitHubRepoInfo } from 'repomix';
+
+type GitHubRepoInfo = NonNullable<ReturnType<typeof parseGitHubRepoInfo>>;🧰 Tools
🪛 GitHub Actions: autofix.ci
[error] 1-1: TypeScript (tsgo) failed with TS2305: Module 'repomix' has no exported member 'GitHubRepoInfo'.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@website/server/src/domains/pack/utils/gitHubRepoSize.ts` at line 1, Remove
the named type import GitHubRepoInfo from the repomix import and instead derive
the type locally from parseGitHubRepoInfo; replace the import line to only
import parseGitHubRepoInfo and add a local type alias such as type
GitHubRepoInfo = NonNullable<ReturnType<typeof parseGitHubRepoInfo>> (and update
any usages in this file to use that local GitHubRepoInfo type) so the code no
longer relies on a non-exported declaration.
Use a more descriptive environment variable name to clarify its purpose. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Code Review (Follow-up): feat(server): Add GitHub repository size check before processingThe previous review covered the major points well (missing tests, DI, logging, timeout). Here are additional findings worth considering. New Findings1. No runtime validation of In const data = (await response.json()) as GitHubRepoResponse;
return data.size;If the GitHub API ever returns a response without a Suggested fix: const data = (await response.json()) as Record<string, unknown>;
if (typeof data.size !== 'number') return null;
return data.size;2. Consider caching the size check result separately Every unique A lightweight size cache keyed on 3. headers.Authorization = `token ${token}`;GitHub has deprecated the headers.Authorization = `Bearer ${token}`;4. GitHub API The 5. No observability on API failures — silent degradationWhen the GitHub API returns a non-200 response (rate-limited, server error), the function silently returns Items from bot reviews evaluated
SummaryThe previous review's top items (missing tests, missing DI) remain the most important blockers. The runtime 🤖 Generated with Claude Code |
1 participant
Add a pre-download size check for GitHub repositories to prevent processing oversized repos that would exhaust server resources.
/repos/{owner}/{repo}) to get repository size before downloadingparseGitHubRepoInfoandGitHubRepoInfofrom the repomix package for server-side URL parsingChecklist
npm run testnpm run lint