Improve report sections and avoid regenerating existing files #261

oree-xx · 2026-01-07T04:32:53Z

Fixes

Description

Added prefix to determine sections and entry better.
Improved the update_readme function to insert report sections in alphabetical order.
Updated the check_for_data_files function to check if processed files have been generated previously to avoid regeneration by listing all file paths under constants and calling function in main().
Defined the section_order function to list all report path.
Added the parser argument --force to enable regenerating of processed files during testing.

Checklist

I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
My pull request doesn't include code or content generated with AI.
My pull request has a descriptive title (not a vague title like Update index.md).
My pull request targets the default branch of the repository (main or master).
My commit messages follow best practices.
My code follows the established code style of the repository.
I added or updated unit tests and/or test scripts for the changes I made (if applicable).
I added or updated documentation (if applicable).
I tried running the project locally and verified that there are no
visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

TimidRobot

I think you can leave the files with their original name, except notes.py which can be renamed zzz-notes.py to ensure it is last.

scripts/3-report/02-github_report.py

scripts/3-report/03-openverse_report.py

scripts/3-report/04-wikipedia_report.py

scripts/3-report/100-notes.py

oree-xx · 2026-01-08T17:10:32Z

@TimidRobot I worked on the updatereadme function, it inserts the sections in an order. I just have a little issue with what I pushed to this PR. I am trying to resolve that.

TimidRobot

Good work here! Please also update the PR description and title to better represent all of the work now included.

scripts/2-process/gcs_process.py

scripts/shared.py

scripts/2-process/openverse_process.py

scripts/3-report/openverse_report.py

scripts/2-process/gcs_process.py

scripts/2-process/github_process.py

scripts/2-process/wikipedia_process.py

TimidRobot · 2026-01-12T16:54:44Z

scripts/2-process/wikipedia_process.py

-        raise shared.QuantifyingException(
-            f"Processed data already exists for {QUARTER}", 0
-        )
+def check_for_data_files(args, file_paths):


I would exit function early if args.force (avoids doing filesystem lookups):

if args.force: return

We already do a similar thing in shared.py:

quantifying/scripts/shared.py

Lines 153 to 155 in 925e721

def git_push_changes(args, repo_path):

if not args.enable_git:

return

Also, maybe it's worth moving this function to shared.py since it's the same in multiple files

TimidRobot · 2026-01-12T17:04:39Z

@oree-xx Please ensure all of the report scripts are executable (-rwxr-xr-x or 0755).

See also: File permissions — Foundational technologies — Creative Commons Open Source

TimidRobot · 2026-01-12T17:08:07Z

scripts/shared.py

+    logger.info("ordered_sections:", ordered_sections)
+    logger.info("section_title:", repr(section_title))


I suspect you converted these from print() statements. As they are now, I get a stack trace. Please convert to f-strings. For example:

logger.info(f"ordered_sections: {ordered_sections}")

For more information on the arguments accepted by logger.info(), see Logger.debug(): https://docs.python.org/3.11/library/logging.html#logging.Logger.debug

TimidRobot · 2026-01-12T17:14:24Z

scripts/shared.py

+def section_order():
+    report_dir = os.path.join(os.path.dirname(__file__), "3-report")
+    report_files = os.listdir(report_dir)
+    return report_files


this function doesn't establish section order. please update to use .sort() or sorted()

TimidRobot · 2026-01-12T17:17:07Z

scripts/3-report/gcs_report.py

 # Constants
 QUARTER = os.path.basename(PATHS["data_quarter"])
-SECTION = "Google Custom Search (GCS)"
+SECTION = Path(__file__).name


oops, I didn't think about the displayed section title.

I think we need both SECTION_FILE and SECTION_TITLE or something similar. One for comments and one for heading/display

Ok, so the update readme function would have SECTION_FILE and SECTION_TITLE as argument.

report_automation

7ff5a69

oree-xx requested review from a team as code owners January 7, 2026 04:32

oree-xx requested review from TimidRobot and possumbilities and removed request for a team January 7, 2026 04:32

cc-open-source-bot added this to TimidRobot Jan 7, 2026

cc-open-source-bot moved this to In review in TimidRobot Jan 7, 2026

Add function for listing out section order

a7645a5

TimidRobot requested changes Jan 7, 2026

View reviewed changes

scripts/3-report/02-github_report.py Outdated Show resolved Hide resolved

scripts/3-report/03-openverse_report.py Outdated Show resolved Hide resolved

scripts/3-report/04-wikipedia_report.py Outdated Show resolved Hide resolved

scripts/3-report/100-notes.py Outdated Show resolved Hide resolved

oree-xx added 3 commits January 7, 2026 21:56

process_script

297dc81

made review changes

9f8ffd8

made new changes

16cb9c9

Added check_for_data_file function for gcs_process.py

5ae1584

TimidRobot requested changes Jan 9, 2026

View reviewed changes

scripts/2-process/gcs_process.py Show resolved Hide resolved

scripts/shared.py Show resolved Hide resolved

scripts/shared.py Outdated Show resolved Hide resolved

TimidRobot reviewed Jan 9, 2026

View reviewed changes

scripts/2-process/openverse_process.py Outdated Show resolved Hide resolved

TimidRobot reviewed Jan 9, 2026

View reviewed changes

scripts/3-report/openverse_report.py Outdated Show resolved Hide resolved

oree-xx added 2 commits January 9, 2026 13:53

Remove openverse process and report

80b6475

Made review changes

7eba804

oree-xx changed the title ~~Prefix for section order~~ Improve report sections and avoid regenerating existing files Jan 9, 2026

TimidRobot mentioned this pull request Jan 9, 2026

Add openverse process and report script #260

Open

9 tasks

TimidRobot reviewed Jan 9, 2026

View reviewed changes

scripts/2-process/gcs_process.py Outdated Show resolved Hide resolved

TimidRobot reviewed Jan 9, 2026

View reviewed changes

scripts/2-process/github_process.py Outdated Show resolved Hide resolved

TimidRobot reviewed Jan 9, 2026

View reviewed changes

scripts/2-process/wikipedia_process.py Outdated Show resolved Hide resolved

Made review changes

241634a

TimidRobot self-assigned this Jan 9, 2026

Added the parser argument --force for regenerating processed files

8f4f079

TimidRobot reviewed Jan 12, 2026

View reviewed changes

oree-xx added 2 commits January 13, 2026 09:15

Made review changes

8a470e4

Made review changes

b2ad661

	def git_push_changes(args, repo_path):
	if not args.enable_git:
	return

		logger.info("ordered_sections:", ordered_sections)
		logger.info("section_title:", repr(section_title))

Uh oh!

Improve report sections and avoid regenerating existing files #261

Are you sure you want to change the base?

Improve report sections and avoid regenerating existing files #261

Uh oh!

Conversation

oree-xx commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fixes

Description

Checklist

Developer Certificate of Origin

Uh oh!

TimidRobot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

oree-xx commented Jan 8, 2026

Uh oh!

TimidRobot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimidRobot commented Jan 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

oree-xx commented Jan 7, 2026 •

edited

Loading