Refactor Mail and Jira Notebook Config

[Michelle4929]

Purpose

Splitting Configuration files for Mail and JIRA notebooks, similar to issue #359 .

Currently, configuration files are mixed with both config data types, immutable and mutable data. Will be making it so that the notebooks will take two configurations when running analyses, one for project and another for analysis.

Will be working with the two notebooks download_jira_issue.Rmd and download_mail.Rmd. Associated notebook runs with geronimo.yml config file so that configuration will be split between immutable and analysis specific data.

Task List

• Split geronimo into geronimo_project.yml and see if it works with analysisE.yml
• Split openssl.yml into openssl_project.yml and analysisI.yml
• Reconfigure download_jira_issue.Rmd to take the split configs
• Reconfigure download_mail.Rmd to take the split configs
• Reconfigure reply_communication_showcase.Rmd to take the split configs

[Michelle4929]

After the meeting today, we agreed that I will make a pull request for the depends/understands issue. I will also use the new proposed changes to keep the timelines/issue in the proper kanban board and correct milestone. After that, I will be then looking and changing how configs are parsed in the jira and mail notebooks. This includes taking another config file, this week, I'll be using geronimo.yml and changing it to an project yml file and seeing if it works with different analyses. Another thing I will be focusing on this week is to think more deeply on M2 and M3 ideas and see which ideas stick closer to my interests.

[Michelle4929]

Hello @carlosparadis ,

After our meeting on 3/16, we agreed that I will present my proposed configuration split into an easily digestable table, similar to the one I did in #359 (comment). I totally agree with that as I should've been coming to an agreement with you for each notebook. We also privately discussed my two choices between M2.

Next Steps:

• Make a table for configuration split (old vs new) and post on each github issue (e.g. scitools/depends, mail/Jira)

• github

• scitools/depends

• mail/Jira

• Rework each notebook to take two configuration (immutable and analysis)

• download_jira_issues.Rmd

• download_mail.Rmd

• reply_communication_showcase.Rmd

• Rework each associated config yml file into project vs analysis(e.g. geronimo_immutable.yml + analysis1.yml)

• geronimo_immutable.yml +analysis.yml

• helix_immutable.yml + analysis.yml

• Decide between the two options of M2 then start researching/looking into them (e.g. fake data generator --> start looking at github.R and associated data)

1. Config/Notebook Analysis Time
2. Fake Data Generator

[Michelle4929]

Table

Here is my current split that I made for the notebooks download_jira_issues.Rmd, download_mail.Rmd, and reply_communication_showcase.Rmd as well as associated notebooks geronimo.yml, helix.yml, and openssl.yml. Table will show which notebook is using which yml files + old configuration + new split sections of configurations.

Notebook Name	Project yml used	What sections of the yml used (old)	Immutable project sections of the yml used (new)	Analysis sections of the yml used (new)
download_jira_issues.Rmd	geronimo.yml	issue_tracker(jira)	project, version_control, mailing_list, issue_tracker, commit_message_id_regex	filter, tool, analysis
download_mail.Rmd	Tools.yml, helix.yml, openssl.yml	tools, mailing_list	project, version_control, mailing_list, issue_tracker, vulnerabilties, commit_message_id_regex	filter, tool, analysis
reply_communication_showcase.Rmd	Tools.yml, helix.yml	Tools, mailing_list, issue_tracker(jira)	project, version_control, mailing_list, issue_tracker, commit_message_id_regex	filter, tool, analysis

The new project yml files are geronimo_immutable.yml and openssl_immutable.yml. The configuration files that are used to run these notebooks only need the project data (immutable) as these notebooks only used the downloaded data from jira and mail.

[carlosparadis]

@Michelle4929 For the M2 part of this issue, could you paste as a new Discussion so we can iterate there and edit this out of this issue? Just to make easier for people in the future needing to reference here! Thank you for starting on this.

[Michelle4929]

@carlosparadis
Just wanted to update you on what I currently worked on this week and if I could get feedback on any improvements or changes, I am totally willing to fix things if there is.

Diagrams

The diagrams and notebook changes are on all 3 of my issues:

• gitlog: Refactor Git Log Notebook Config #359 (comment)
• scitools/depends: Refactor Scitools/Depends/Line Metrics/Kaiaulu Architecture Notebook Config #366 (comment)
• mail/Jira: Refactor Mail and Jira Notebook Config #376 (comment)

Jira/Mail Notebook Config Changes

These are the changes I've made to the notebooks in jira and mail. Before it would parse through one config file, now the configuration is split between immutable vs mutable data.

download_jira_issues.Rmd

Previous parsing of config in the notebook:

kaiaulu/vignettes/download_jira_issues.Rmd

Lines 48 to 67 in 5bbdbb2

	conf <- parse_config("../conf/kaiaulu.yml")
	# Project domain
	# Specify project_key_index in get_jira_domain() (e.g. "project_key_1")
	issue_tracker_domain <- get_jira_domain(conf, "project_key_1")
	# Project key
	# Specify project_key_index in get_jira_project_key_name() (e.g. "project_key_1")
	issue_tracker_project_key <- get_jira_project_key_name(conf, "project_key_1")
	# Altered save paths. Important for naming conventions
	# Specify project_key_index in get_jira_issues_path() (e.g. "project_key_1")
	save_path_issue_tracker_issues <- get_jira_issues_path(conf, "project_key_1")
	# Specify project_key_index in get_jira_issues_comments_path() (e.g. "project_key_1")
	save_path_issue_tracker_issue_comments <- get_jira_issues_comments_path(conf, "project_key_1")
	# Unaltered save paths from config file for use with refresh function
	# Specify project_key_index in get_jira_issues_path() (e.g. "project_key_1")
	refresh_issues <- get_jira_issues_path(conf, "project_key_1")

Changes I've Made

Now it parses through split configuration (project + analysis):
https://github.com/Michelle4929/kaiaulu/blob/db3d3ca5162a2aeb5a563f645b542c15d5e51692/vignettes/download_jira_issues.Rmd#L51-L60

Here is geronimo_immutable.yml I made for jira notebook, it's the entire file but the short condensed version of what is in this file is in the diagram above!
https://github.com/Michelle4929/kaiaulu/blob/db3d3ca5162a2aeb5a563f645b542c15d5e51692/conf/geronimo_immutable.yml#L1-L121

download_mail.Rmd

Previous parsing of config in the notebook (parses 2 different mailing list systems so that means it uses 2 different config files):

kaiaulu/vignettes/download_mail.Rmd

Lines 85 to 91 in 5bbdbb2

	# Load tools configuration
	tools <- parse_config("../tools.yml")
	parse_perceval_path <- get_tool_project("perceval", tools)
	# Load project configuration
	conf <- parse_config("../conf/helix.yml")
	mbox_file_path <- get_mbox_path(conf, "project_key_1")

kaiaulu/vignettes/download_mail.Rmd

Lines 107 to 113 in 5bbdbb2

	conf <- parse_config("../conf/openssl.yml")
	pipermail_mailing_list <- get_pipermail_domain(conf, "project_key_1")
	pipermail_save_folder_path <- get_pipermail_path(conf, "project_key_1")
	# Define the date range
	pipermail_start_year_month <- 202310
	pipermail_end_year_month <- 202405

Changes I've Made

Now it parses through split configuration (project + analysis):
For Helix: https://github.com/Michelle4929/kaiaulu/blob/db3d3ca5162a2aeb5a563f645b542c15d5e51692/vignettes/download_mail.Rmd#L93-L96
For Openssl: https://github.com/Michelle4929/kaiaulu/blob/db3d3ca5162a2aeb5a563f645b542c15d5e51692/vignettes/download_mail.Rmd#L116-L120

Here is openssl_immutable.yml I made for mail notebook, it's the entire file but the short condensed version of what is in this file is in the diagram above!
https://github.com/Michelle4929/kaiaulu/blob/db3d3ca5162a2aeb5a563f645b542c15d5e51692/conf/openssl_immutable.yml#L1-L103
I already previously made the helix_immutable.yml file but here it is as well:
https://github.com/Michelle4929/kaiaulu/blob/db3d3ca5162a2aeb5a563f645b542c15d5e51692/conf/helix_immutable.yml#L1-L84

reply_communication_showcase.Rmd

Previous parsing of config in the notebook:

kaiaulu/vignettes/reply_communication_showcase.Rmd

Lines 32 to 36 in 5bbdbb2

	tool <- parse_config("../tools.yml")
	conf <- parse_config("../conf/helix.yml")
	perceval_path <- get_tool_project("perceval", tool)
	mbox_path <- get_mbox_path(conf, "project_key_1")
	jira_issue_comments_path <- get_jira_issues_comments_path(conf, "project_key_1")

Changes I've Made

Now it parses through split configuration (project + analysis):
https://github.com/Michelle4929/kaiaulu/blob/db3d3ca5162a2aeb5a563f645b542c15d5e51692/vignettes/reply_communication_showcase.Rmd#L38-L42

These notebooks only use the project data in the configuration, as mail's and jira's notebooks uses mailing_list and issue_tracker.

[Michelle4929]

After 3/20 meeting, I will be working on the next notebooks: bugcount and source (kaiaulu architecture and line metrics). This will include making a diagram, splitting the associated config files. I will also be working on M2, fleshing out more details and understanding of my task in the discussion post.