Transform and Graph Standardization - #375
Conversation
Introduce reusable graph constructors (model_directed_graph_v2, model_undirected_graph, model_bipartite_graph, model_heterogeneous_graph) and add transform_gitlog_to_heterogeneous_network to build heterogeneous author/commit/file networks. Enhance transform_dependencies_to_network with input validation, correct handling of weight_types (including NA as 'use all'), use of copy() to avoid mutating parsed data, explicit node type/color assignment, and returning a directed graph model rather than a raw list. Also adjust parse_dependencies output filename suffix from ".json" to "-file.json". These changes improve safety, clarity, and reuse of graph construction logic. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
…ection Introduce transform_gitlog_to_heterogeneous_network (with node_cols) to build heterogeneous graphs and refactor transform_gitlog_to_entity_bipartite_network to validate input and use model_bipartite_graph. Add bipartite_graph_projection_v2 S3 generic with methods for bipartite_graph and heterogeneous_graph to support projections through edge types and reuse weight schemes. Add documentation for new model constructors (bipartite/heterogeneous/undirected/directed v2) and the new projection, update many Rd cross-references, and export the new functions in NAMESPACE. Also bump RoxygenNote to 7.3.3. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Introduce S3 generic APIs for graph export and projection: graph_to_dsmj, bipartite_graph_projection and temporal_graph_projection now dispatch by graph class (directed/undirected/bipartite/heterogeneous) and delegate to shared implementations. Rework model constructors and projections: model_directed_graph signature standardized (replacing _v2), model_bipartite_graph used to build graphs in git transforms, and temporal/bipartite projection methods now return appropriately typed graph classes. Update transform functions in git.R and dv8.R to build node/edgelist tables explicitly (with input validation and NSE-safe variable declarations), add transform_gitlog_to_bipartite_network, and adjust calls to graph_to_dsmj/model_directed_graph. Add timestamp propagation and weight aggregation improvements for heterogeneous/temporal transforms and simplify commit-message-id and bipartite network construction. Call sites in src.R updated to new constructor. These changes improve type-safe dispatch, clarify responsibilities between builders and exporters, and centralize DSMJ export logic. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Update NAMESPACE after function renames
Introduce an edge_type argument (default NULL) to temporal_graph_projection to allow slicing heterogeneous graphs by edge type before projection. Updated the function signature in R/graph.R and added corresponding documentation in man/temporal_graph_projection.Rd. The new parameter is ignored for bipartite graphs and preserves existing UseMethod dispatch and default behavior. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
a2bfddc to
8b2a00f
Compare
|
PR checks fail because of the changes made to |
|
@jseto808 any reason why you holding on changing them? Too many changes / waiting confirmation/ something else ? |
Add a source parameter to reply transform and restructure how reply and dependency graphs are built. transform_reply_to_bipartite_network now accepts source, constructs explicit node and edgelist tables (including a source column), aggregates reply weights and uses model_bipartite_graph. transform_understand_dependencies_to_network disambiguates node labels with IDs, filters dependency kinds, aggregates edge weights, builds a nodes/edgelist and uses model_directed_graph (and slightly reworded the empty-edge error). transform_r_dependencies_to_network was simplified to pick columns by dependency_type, aggregate weights, build nodes/edgelist and call model_directed_graph. Updated Rd docs and vignettes to match the new "nodes"/"edgelist" output keys and the new source argument. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Add an optional `edge_type` argument to temporal_graph_projection.bipartite_graph (defaults to NULL). The function signature is updated to accept `edge_type` before `lag`, allowing callers to filter or specify edge types when projecting temporal bipartite graphs. No other logic changes in this diff. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Include the new 'edge_type = NULL' argument in the temporal_graph_projection Rd file so the documentation matches the updated function signature. Documents the optional edge_type parameter (default NULL) for users to control edge classification in projections. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Hi @carlosparadis, apologies for the confusion. I had listed that as a note to myself on what still needed to be fixed since I didn’t have time to address it at the moment. |
Add defensive checks to heterogeneous projection functions to ensure an edge_type connects exactly two node types and error otherwise. Fix a broken nrow(...) check in temporal_graph_projection.bipartite_graph. In git utilities, avoid mutating the input by copying project_git, rename the target column from 'entity_definition_name' to 'entity', remove an unused NSE variable, and replace a vectorized '&' with scalar '&&' to correct the conditional logic. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Replace the Unicode em dash (—) with an ASCII hyphen (-) in stop() messages within bipartite_graph_projection.heterogeneous_graph and temporal_graph_projection.heterogeneous_graph in R/graph.R to ensure consistent, ASCII-safe error text and avoid potential encoding/display issues. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Clean up and simplify transform_cve_cwe_file_to_network: remove unused NSE variable declarations, standardize data.table syntax/spacing, and extract CWE edges by merging nvd_feed. Rename cwe_id to to, add an integer weight column for CWE edges, and combine node/edge sets into a directed graph via model_directed_graph (returning the graph instead of a list). These changes simplify the edgelist handling and produce a consistent graph structure for downstream use. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Rename heterogeneous_graph -> multimodal_graph and undirected/ directed graph types to unimodal_graph. Update constructors (model_multimodal_graph, model_unimodal_graph), S3 methods (graph_to_dsmj.*, bipartite_graph_projection.*, temporal_graph_projection.*), and transform functions to use the new model_unimodal_graph where appropriate. Refresh NAMESPACE entries and all related Rd docs (including renames/deletions) and replace references to transform_gitlog_to_heterogeneous_network with transform_gitlog_to_multimodal_network. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Introduce superset_bipartite_from_multimodal to R/graph.R: extracts a bipartite graph from a multimodal graph by selecting edges of a given edge_type and returning a model_bipartite_graph. The function checks for no-matches and ensures the edge type connects exactly two node types, mapping the from-side to type = TRUE and to-side to type = FALSE. Also add corresponding Rd documentation (man/superset_bipartite_from_multimodal.Rd) and export the new function in NAMESPACE to enable usage with bipartite/temporal projections. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Rename the 'parsed' parameter to 'understand_parsed' in transform_understand_dependencies_to_network and update its man page accordingly to improve clarity. Standardize local return variable names (e.g. graph -> understand_graph, depends_graph, r_dependencies_graph, cve_cwe_graph, git_graph) across R/src.R, R/git.R, and R/vulnerabilities.R to avoid ambiguous 'graph' identifiers. These are naming/refactor-only changes and do not alter function behavior. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Compute is_directed from the edges' direction column and pass it to igraph::graph_from_data_frame instead of hardcoding TRUE. This makes the vignette plot respect directed vs. undirected edges (vignettes/custom_graph_weight_showcase.Rmd). --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Depends may emit either <project>-file.json (newer) or <project>.json (older). Previously the code assumed the -file.json name; this change checks for both filenames (output_path_file and output_path_plain), sets output_path accordingly, and raises an informative error if neither is found. Keeps downstream parsing behavior unchanged. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Update custom_graph_weight_showcase.Rmd to load the CakePHP commit comments CSV from a remote URL and append a new "Emotion Graph" section that visualizes binary emotion labels (love, joy, sadness, anger, surprise, fear) as weighted bipartite author-issue graphs. Add new vignette dv8_author_communication_showcase.Rmd demonstrating an end-to-end pipeline: parse mbox to reply table, build bipartite author-subject graph, project to author-author network, export DSM, convert to DV8 binary, run DR Space hierarchical clustering, and parse cluster assignments. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Normalize and fix how command arguments are constructed for system2 in dv8_mdsmb_to_hierclsxb: expand hierclsxb_path, initialize args as character vectors, remove trailing-space flags, and correctly build the -maxDepth flag from max_depth. Pass the -outputFile and path as separate arguments and stop redirecting stdout to a file (stdout=FALSE). These changes avoid malformed command strings and incorrect conditional checks when invoking the dv8 binary. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Fix DV8 tool path in tools.yml and update the DV8 author showcase to use project-specific names: switch config to openssl.yml, use get_pipermail_path, derive project_name from dv8_out, and generate DV8 JSON/binary/cluster filenames with paste0(project_name, ...). Also add a new vignette (radio_silence_showcase.Rmd) demonstrating the radio silence analysis pipeline using pipermail data, OSLOM community detection, and the smell_radio_silence routine. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Rewrite parse_dv8_clusters to recursively traverse DV8 hierarchy: collect leaf nodes, emit (file_path, module, layer) rows, assign "Isolated" to leaf children, and handle nested module names robustly. Update vignettes to use project_path/project_name variables, pass recursive = TRUE to dv8_mdsmb_to_hierclsxb, and show/merge only top-level cluster rows for summaries. Add a DV8-based radio_silence example that exports DSMs, runs DR Space clustering, and computes radio silence per layer. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Replace mbox/perceval-based examples with GitHub issue/PR reply parsing across three vignettes. Switch example datasets to Kaiaulu prediction CSVs, remap polarity integers to signed edge weights in the custom-graph vignette, and update config keys to use kaiaulu.yml and GitHub-specific path helpers. Add parsing via parse_github_replies, change transform inputs to project_github_replies, and add a boundary-spanner 1-hop neighborhood visualization in the radio-silence vignette. Miscellaneous wording and node/subject semantics updated to reflect issues/PR threads instead of mail threads. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
|
Interesting, I just realized this was made as a branch out of a fork, rather than a branch within Kaiaulu (the write permission was so you could branch off Kaiaulu!). But not a big deal 👍 |
Without dv8 yml path, the code will break on radio_silence.Rmd as project_path will be null from get_dv8_folder_path. Signed-off-by: Carlos Paradis <carlosviansi@gmail.com>
hdsm stands for historical DSM, and sdsm for structural dsm. This notebook introduces a new type of dsm, author. As such, for now I will call it adsm. The -clsx currently does not differentiate the type of dsm since it is cluster information. Maybe in the future I will extend it to account for all types. There is the merit on whether author-committer or author-commit will be adsm or something else. Will defer this to the future. Signed-off-by: Carlos Paradis <carlosviansi@gmail.com>
Kaiaulu convention of analysis folder hierarchy was modified awhile back and this seems to have been missed. Previously: analysis/dv8/<project>. Currently: analysis/project/dv8 In this sense, the project name is no longer found on the last position of the string split by "/" but rather the 2nd to last (hence length - 1). Signed-off-by: Carlos Paradis <carlosviansi@gmail.com>
carlosparadis
left a comment
carlosparadis
left a comment
There was a problem hiding this comment.
In order for me to do the final test, which runs all notebooks in Kaiaulu using your new api, I need to list the functions you added and deleted in kaiaulu/_pkgdown.yml. This is where I define the modules you see in the itm0 docs.
Could you version that file in your PR, add the functions you added and remove those from there you deleted for me?
I am done making commits here for now.
Remember that you need to git checkout to this branch on your local machine, then git pull my commits before making changes. If you try to edit before doing that, you will enter a merge conflict which is way more annoying to deal with.
Please ping me here when you can pass the torch so we don't enter commit conflict.
This is really, really good. I could run all notebooks very easily by literally git pulling and pressing run all.
I see you also respected the relative paths, folder conventions, and did not commit the tools.yml with your local config (a very common mistake).
I will take a closer look on the transforms after I can use the pkgdown to run all notebooks, but I imagine any major change here would be too late to patch at this point. Fingers crossed it works across all notebooks!
| hierclsxj_table <- parse_dv8_clusters(hierclsxj_path) | ||
| kable(head(hierclsxj_table)) | ||
| top_hierclsxj_table <- hierclsxj_table[!grepl("/", layer)] | ||
| kable(head(top_hierclsxj_table)) |
There was a problem hiding this comment.
@jseto808 why was this modified? I feel I am missing something.
There was a problem hiding this comment.
@carlosparadis parse_dv8_clusters now returns one row per file per layer. Since the flaws map only needs one cluster assignment per file, the fix keeps only the top-level layer rows before merging to avoid duplicates.
Expose multimodal and bidirected graph features in the pkgdown config. Adds transform_gitlog_to_multimodal_network to the reference list and expands the Graph contents to include model_unimodal_graph, model_multimodal_graph, subset_bipartite_from_multimodal, and get_bidirected_edges so these topics appear in the generated documentation. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Swap knitr::kable usage for gt tables across three vignettes (custom_graph_weight_showcase.Rmd, dv8_author_communication_showcase.Rmd, radio_silence_showcase.Rmd). Updated require(knitr) to require(gt) and replaced kable(...) calls with head(... ) %>% gt(auto_align = FALSE) (or data.table(...) %>% gt(auto_align = FALSE)) to standardize table rendering; also adjusted a few conditional prints to use gt(... ) %>% print(). This modernizes table output and ensures consistent formatting in the examples. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
|
@carlosparadis I updated the |
|
Acknowledged. Will review after I take a pass on Phuong's code. Can you issue a code review request for me on the PR? |
|
@jseto808 I was able to finish running the code through all notebooks. I have not inspected the outputs yet, but here's what I would say breaks the code and prevents this PR from merge. FYI you can test it yourself by doing vignettes/understand_showcase.RmdBoth understand transforms are breaking. kaiaulu/vignettes/understand_showcase.Rmd Lines 108 to 136 in 5bbdbb2 vignettes/community_detection_showcase.Rmdkaiaulu/vignettes/community_detection_showcase.Rmd Lines 221 to 228 in 5bbdbb2 This recolor function now breaks. vignettes/causal_flaws.Rmdkaiaulu/vignettes/causal_flaws.Rmd Lines 483 to 489 in 5bbdbb2 The chunk 35 keeps breaking when I use pkgdown, but can run and compile on knitr. Not sure what is going on. That's it.Surprisingly that's all that crashed across ALL Kaiaulu notebooks when compiling itm0 docs. I also sent you via email the drive url of the generated docs. You should need to open the index.html file and you can browse locally. As I said before, there is little to no time. If it is an easy fix, go for it. If not, I'd appreciate if you could add some pointers here for Fall'26 to help me patch this up. I also need to of course look through the output notebooks to see if the code didn't run and had NAs where it used to be results. Note you will not see causal_flaws.Rmd there since I had to remove it from compiling altogether. The rest will show up but the code block that breaks i modified to eval = false so there will be no output for them. Signing off for now, just wanted to handle this over to you the earliest I could. |
Replace the deprecated parsed parameter with understand_parsed in transform_understand_dependencies_to_network calls within vignettes/understand_showcase.Rmd (file and class dependency examples). Keeps the vignette examples consistent with the updated function signature. --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
Populate the edgelist with a direction column set to "directed". This ensures the edgelist carries explicit direction metadata for downstream network processing or visualization (added edgelist[, direction := "directed"]). --------- Signed-off-by: Jared Seto <jaredws@hawaii.edu>
|
Hi @carlosparadis, Here is a report of the three breaking notebooks, what caused each failure, and what was fixed. understand_showcase.RmdWhat broke: Both Fix: Changed both call sites in the vignette: # Before
transform_understand_dependencies_to_network(parsed = file_dependencies, ...)
transform_understand_dependencies_to_network(parsed = class_dependencies, ...)
# After
transform_understand_dependencies_to_network(understand_parsed = file_dependencies, ...)
transform_understand_dependencies_to_network(understand_parsed = class_dependencies, ...)community_detection_showcase.RmdWhat broke: Reported as the Fix: No code change. causal_flaws.RmdWhat broke: Column mismatch in Fix: Added edgelist <- file_network[["edgelist"]][,.(from=src_filepath, to=dest_filepath)]
edgelist$weight <- rowSums(...)
edgelist[, direction := "directed"] # added
file_network[["edgelist"]] <- edgelist |
|
Acknowledged. Will leave it running on the fixes and let you know how it goes, thanks! |
community_detection_showcase.RmdThere is something going wrong with project_commit_network <- transform_gitlog_to_bipartite_network(project_git_slice,
mode="commit-file")I don't think it is happening to you because this may be tied to the dataset. The error is: I sent you the repo of APR I am using. You can use the apr config to run. Could you try to run with this to see if it blows up on your side? |
|
Disregard comment above. Problem was a slice that was emptying the table: #project_git_slice <- project_git[author_datetimetz >= as.POSIXct("2015-01-01", format = "%Y-%m-%d",tz = "UTC") & author_datetimetz < as.POSIXct("2019-12-31", format = "%Y-%m-%d",tz = "UTC")] project_git_slice <- project_git[author_datetimetz >= as.POSIXct("1999-01-01", format = "%Y-%m-%d",tz = "UTC") & author_datetimetz < as.POSIXct("2000-12-31", format = "%Y-%m-%d",tz = "UTC")] That was causing memory errors. Commented led to a empty git slice that would then blow up on the recolor. Testing causal flaws next. Understand now works. |
|
Individually, I can now knit all notebooks! I am doing a run all to see how it goes. |
The existing git slice does not match to the local dataset where the docs is generated. This led to code chunk 20 to result in empty tables, and ultimately causing some strange memory error. Should use the commit prior to add some checks before merging in the future. This commit generates *ALL* notebooks, however I did not have a chance to inspect all the outputs to see if there is any empty tables or graphs. Signed-off-by: Carlos Paradis <carlosviansi@gmail.com>
carlosparadis
left a comment
carlosparadis
left a comment
There was a problem hiding this comment.
The last commit compiles all itm0 docs, meaning all notebooks currently execute start to end without error with the new added changes!
I still need to look at the docs of itm0 to sanity check the actual output and the logic before merging but this is as good as it gets with what time is left.
4 participants
#361 Transform and Graph Standardization
Summary
This PR standardizes the graph construction and transform APIs across all network types in Kaiaulu. It replaces the monolithic
model_directed_graph()constructor with two purpose-built constructors (model_unimodal_graph(),model_multimodal_graph()), introduces S3 dispatch for projection and export functions, refactors all transform functions to explicitly build node and edgelist tables before passing them to constructors, and extends DV8 integration to support hierarchical clustering of author communication networks.Changes
New graph model constructors (
R/graph.R)model_unimodal_graph()— replacesmodel_directed_graph()for single-node-type graphsmodel_multimodal_graph()— replacesmodel_directed_graph()for bipartite and multimodal graphs; supportsis_bipartiteflag and multiple edge types via anedge_typecolumnget_bidirected_edges()— utility to extract edges where the reverse also existssubset_bipartite_from_multimodal()— extracts a bipartite subgraph from a multimodal graph by edge typeS3 dispatch for projections and export (
R/graph.R)bipartite_graph_projection()— now dispatches onis_bipartiteandmultimodal; bipartite graphs embedded in a multimodal class are correctly routedtemporal_graph_projection()— same dispatch pattern; validatestimestamp_columnpresence; supportsedge_typeparameter to slice non-bipartite multimodal graphs before projectiongraph_to_dsmj()— now dispatches onunimodalandmultimodal; infers direction from graph S3 class instead of an explicitis_directedparameterRefactored transform functions
R/git.R:transform_gitlog_to_temporal_network,transform_gitlog_to_bipartite_network,transform_gitlog_to_entity_bipartite_network,transform_gitlog_to_entity_temporal_network,transform_commit_message_id_to_network— all migrated to new constructors with explicit node/edgelist construction; weight semantics preserved from master (weight = .Nfor bipartite,weight = 1Lper event for temporal,n_lines_changedfor entity temporal)R/git.R(new):transform_gitlog_to_multimodal_network— constructs a three-edge-type multimodal graph (authored / changed / modified) from a parsed git logR/src.R:transform_dependencies_to_network,transform_understand_dependencies_to_network,transform_r_dependencies_to_network— migrated tomodel_unimodal_graph()R/reply.R:transform_reply_to_bipartite_network— migrated tomodel_multimodal_graph()R/vulnerabilities.R:transform_cve_cwe_file_to_network— migrated tomodel_unimodal_graph()R/dv8.R:transform_dependencies_to_sdsmj,transform_gitlog_to_hdsmj,transform_temporal_gitlog_to_adsmj— updated to use new constructors and S3-dispatchedgraph_to_dsmj()DV8 hierarchical clustering extensions (
R/dv8.R)parse_dv8_clusters()— rewritten to traverse the full cluster hierarchy recursively, emitting one row per leaf per layer; supports recursive DR Space outputdv8_mdsmb_to_hierclsxb()— argument handling fixed;recursiveparameter addedNew vignettes
dv8_author_communication_showcase.Rmd— end-to-end pipeline from mbox data to DV8 DR Space clustering of an author communication networkradio_silence_showcase.Rmd— radio silence social smell with both OSLOM and DV8 DR Space per-layer community detectionBug fix
transform_gitlog_to_entity_bipartite_networkcommitter-entity mode incorrectly usedfrom=authorinstead offrom=committerTest plan
Knit notebooks in the
masterbranch and compare to knitted notebooks in this PR:gitlog_showcase.Rmdgitlog_entity_showcase.Rmdgitlog_vulnerabilities_showcase.Rmddepends_showcase.Rmdunderstand_showcase.Rmddv8_showcase.Rmdcustom_graph_weight_showcase.Rmddv8_author_communication_showcase.Rmdradio_silence_showcase.Rmdcommunity_detection_showcase.Rmdreply_communication_showcase.Rmdsocial_smell_showcase.Rmdissue_social_smell_showcase.Rmdbug_count.Rmdcausal_flaws.Rmdmotif_analysis.Rmdkaiaulu_architecture.Rmd