Implement GitHub Commit Comments Downloader Function - #381
Conversation
- Adds `github_api_project_commit_comments` to fetch commit comments from GET /repos/{owner}/{repo}/comments
- Adds `github_parse_project_commit_comments` to convert commit comment API responses into a parsed data.table
- Captures key fields for analysis (comment_id, commit_id, author info, body text, path/line context, and timestamps)
- Enables commit-comment workflows that join external sentiment labels by comment_id
- Adds `get_github_commit_comment_path()` to `config.R` to read the `commit_comments` save path from project .yml files - Adds commit comments download and parse sections to `download_github_events.Rmd` - Adds `page` and `per_page` parameters to `github_api_project_commit_comments` and `github_api_project_pull_request_inline_comments` to allow explicit page requests - Fixes a bug in `github_api_iterate_pages` where `gh_next()` returns the same page repeatedly. Adds duplicate-page detection and an explicit page=N fallback for both the commit comments and PR inline comments endpoints - Fixes missing `save_path_pr_comments` argument in `download_github_pull_request_comments.Rmd` - Adds `commit_comments` path to `kaiaulu.yml`
- Added `dir.create` function to `download_github_events.Rmd` and `download_github_pull_request_comments.Rmd` to automatically create directories to store extracted PR and commit comments
- Uses `gh` version 1.2.0 to resolve the bug of the same page saving - Removes previously implemented duplicate-page detection fallback functions created in [commit d93ad20](sailuh@d93ad20)
…kflows
- Add generator script (exec/conf.R) that pulls all projects from the GHTorrent database to produce conf YAMLs for each project with sentiment labels
- Generate YAMLs for main project repos and fork repos with comments that are not accessible from the main repo. Verify via API checks
- Generate GitHub paths for commit/PR comments used in sentiment labeling (e.g., rawdata/github/{owner}/{repo}/commit_comments/ and rawdata/github/{owner}/{repo}/pr_comments/)
- Standardize naming and headers for reproducible config files
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #381 +/- ##
==========================================
- Coverage 39.79% 32.11% -7.69%
==========================================
Files 20 21 +1
Lines 3091 4151 +1060
==========================================
+ Hits 1230 1333 +103
- Misses 1861 2818 +957 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
- Add 82 generated YAML configuration files for main project repositories with sentiment-labeled data from the GHTorrent database - Include GitHub rawdata paths for sentiment workflow (commit comments and PR inline comments)
- R/github.R: removed NULL columns for cleaner CSV conversion - vignettes/download_github_events.Rmt: Added ability to save data table for commit comments as a CSV
phuong808
left a comment
There was a problem hiding this comment.
I ran the notebook locally and verified that everything executes without issues. The outputs match the results shown in the PR, and I was able to reproduce them successfully. Looks good to me.
|
@phuong808 don't forget to update the kanban on this PR! You can do from the right pane in this PR. |
Some columns in inline_comments contain NULL or list-type values that fwrite() can't handle directly. The added code chunk coerces those columns to character before writing.
jseto808
left a comment
There was a problem hiding this comment.
The R function additions follow existing naming conventions: github_api_project_commit_comments and github_parse_project_commit_comments follow the github_api_project_* and github_parse_project_* patterns respectively, and get_github_commit_comment_path follows the get_github_*_path pattern. The 82 new project YAML configs are consistent with existing conf files. The notebooks ran without issue. No issues found.
Added name to `DESCRIPTION` as a contributor. Added `github_api_project_commit_comments()`, `github_parse_project_commit_comments()`, and `get_github_commit_comment_path()` under NEW FEATURES in `NEWS.md`.
Purpose
Kaiaulu currently has the function
github_api_project_commitsto download commits from the GitHub API, but no equivalent function for downloading commit comments.This new function is needed to transfer sentiment labels attached to commit comments from the GHTorrent database. To map these labels to Kaiaulu-downloaded data, the comment text must be retrieved through Kaiaulu's GitHub API pipeline rather than sourced directly from the static GHTorrent database. The bridge between the sentiment label and the downloaded comment is the commit comment ID.
This was identified and discussed in sailuh/sentiment_github_dataset#1.
Process
Implement a new functionc called
github_api_project_commit_commentsinR/github.Rby:github_api_project_commitsfunction as a templateGET /repos/{owner}/{repo}/commentsper the GitHub REST API documentationowner,repo, andtokencomment_idvalues match those in GHTorrent'scommit_commentstableTask List
github_api_project_commit_commentsfunction withowner,repo, andtokenparametersgithub_api_project_commitsfunction as a templateGET /repos/{owner}/{repo}/commentscomment_idvalues match GHTorrent'scommit_commentstable