0.2.0: Fixes and improvements (#33)

Fixes #27, #29, #13

Author: ethteck

.github/workflows/ci.yml

@@ -5,14 +5,14 @@ on:
       - main
   pull_request:
 jobs:
-  full_test_and_build:
-    name: unit tests
+  tests:
+    name: tests
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
-      - uses: actions/setup-python@v1
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
         with:
-          python-version: 3.9
+          python-version: '3.10'
       - uses: snok/install-poetry@v1
       - run: poetry install
       - name: Run backend tests
@@ -22,10 +22,10 @@ jobs:
     name: mypy
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
-      - uses: actions/setup-python@v1
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
         with:
-          python-version: 3.9
+          python-version: '3.10'
       - uses: snok/install-poetry@v1
       - run: |-
           poetry install
@@ -34,7 +34,11 @@ jobs:
     name: black
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
-      - uses: psf/black@stable
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
         with:
-          src: "."
+          python-version: '3.10'
+      - uses: snok/install-poetry@v1
+      - run: |-
+          poetry install
+          poetry run black . --check

CONTRIBUTING.md

@@ -1,7 +1,7 @@
 # Contributing
 
 ## Dependencies
-- Python >=3.9
+- Python >=3.10
 - [Poetry](https://python-poetry.org/docs/master/#installing-with-the-official-installer)
 
 ## Setup

GUIDE.md

@@ -1,8 +1,8 @@
-# Guide on how to monitor decomp progress using Frogress
+# Guide on how to monitor decomp progress using frogress
 
 ## Overview
 
-It will guide you to onboard your decomp project to Frogress.
+This guide will provide a flow for how to use frogress for your project.
 
 ```mermaid
 sequenceDiagram
@@ -12,19 +12,19 @@ sequenceDiagram
     Contributor->>+CI: Trigger
     CI->>CI: Build
     CI->>CI: Calculate progress
-    CI->>-Frogress: Upload progress
-    Note over CI,Frogress: POST /data/project:/version:/
-    Monitor->>Frogress: Fetch progress
-    Note over Monitor,Frogress: GET /data/project:/version:/?mode=all
-    Frogress->>Monitor: Return progress
+    CI->>-frogress: Upload progress
+    Note over CI,frogress: POST /data/project:/version:/
+    Monitor->>frogress: Fetch progress
+    Note over Monitor,frogress: GET /data/project:/version:/?mode=all
+    frogress->>Monitor: Return progress
     Monitor->>Monitor: Render progress
 ```
 
 *CI: Continuous Integration services including GitHub Actions, Gitlab pipelines, Travis CI and Jenkins.*
 
 ## Steps
 
-1. Contact Frogress admin to create project and assign api_key
+1. Contact frogress admin (Ethan) to add your project to the database and obtain an api_key
 
 2. Create schema with `cli.py`
 
@@ -45,9 +45,7 @@ sequenceDiagram
     ./cli.py create version fireemblem8 us
     ```
 
-    2.3 Create category (optional)
-    
-    Default category: `default`
+    2.3 Create category
 
     ```bash
     # Usage
@@ -56,14 +54,14 @@ sequenceDiagram
     ./cli.py create category fireemblem8 us default
     ```
 
-3. Upload progress in CI
+3. Configure CI to upload data on build
 
     3.1 API
 
     ```
     POST https://progress.deco.mp/data/project:/version:/
     ```
-    
+
     ```python
     {
         "api_key": "",
@@ -80,39 +78,47 @@ sequenceDiagram
     ```
 
     3.2 Example
-    
+
     https://github.com/FireEmblemUniverse/fireemblem8u/pull/307
 
-4. Supplement historical data (optional)
+4. Supplement historical data (optional, one-time)
 
-    Calculate progress for historical commits and upload it to Frogress if you would like to draw historical curve.
+    Calculate progress for historical commits and upload it to frogress for the purpose of visualizing and tracking historical data
 
     [Example](https://github.com/laqieer/fireemblem8u/blob/master/.github/workflows/supplement-progress.yml)
 
-5. Prune duplicated data (optional)
-
-    Background: https://github.com/decompals/frogress/issues/27
-  
-    ```
-    # Usage
-    ./cli.py prune -h
-    # Example
-    ./cli.py prune fireemblem8 us
-    ```
+5. Fetch project data
 
-6. Fetch project data
+    5.1 API
 
-    6.1 API
+    There are 3 "modes" for returning results: `all`, `latest`, and `shield`. The first two are pretty self explanatory, and `shield` can be used to generate badges for your repo's README.md (shields.io).
 
     ```
     GET https://progress.deco.mp/data/project:/version:/?mode=all
     ```
+    ```
+    GET https://progress.deco.mp/data/project:/version:/?mode=latest
+    ```
+    Note that the category and measure are required for mode `shield`.
+    ```
+    GET https://progress.deco.mp/data/project:/version:/category:/?mode=shield&measure=MEASURE
+    ```
+
+    **Note:** Specifying a category in the request URL is optional for `all` and `latest` modes. If the category is not specified, results from all categories will be returned. However, for `shield`, it is necessary to specify the category and measure in the request url, as shown above.
+
+    `all` example:
+
+    `https://progress.deco.mp/data/fireemblem8/us/?mode=all`
+
+    `shield` example:
+
+    `https://progress.deco.mp/data/dukezh/us/default/?mode=shield&measure=bytes`
 
-    https://progress.deco.mp/data/fireemblem8/us/?mode=all
+    5.2 Build a website
 
-    6.2 Build a website
+    Build a website to display your progress!
 
-    Build a website to render progress graph using a library such as [uPlot](https://github.com/leeoniya/uPlot) and [Chart.js](https://www.chartjs.org).
+    You can use the "latest" mode to retrieve just the latest datapoint or render full graphs using a library such as [uPlot](https://github.com/leeoniya/uPlot) or [Chart.js](https://www.chartjs.org)
 
       - https://pikmin.dev
       - https://axiodl.com

cli.py

@@ -83,93 +83,6 @@ def delete_category(args: argparse.Namespace) -> None:
     print(response.text)
 
 
-def prune_entries(args: argparse.Namespace) -> None:
-    # Get entries
-
-    url = f"{domain}/data/{args.project}/{args.version}?mode=all"
-
-    debug("GET " + url)
-
-    response = requests.get(url)
-    print(response.text)
-
-    categories = response.json().get(args.project, {}).get(args.version, {})
-
-    if len(categories) == 0:
-        return
-
-    # Filter entries
-
-    filtered = {}
-
-    for category, entries in categories.items():
-        if len(entries) == 0:
-            continue
-        for entry in entries:
-            if entry["git_hash"] not in filtered:
-                filtered[entry["git_hash"]] = {
-                    "git_hash": entry["git_hash"],
-                    "timestamp": entry["timestamp"],
-                    "categories": {},
-                }
-            if category not in filtered[entry["git_hash"]]["categories"]:
-                filtered[entry["git_hash"]]["categories"][category] = entry["measures"]
-            else:
-                for measure, value in entry["measures"].items():
-                    if (
-                        measure
-                        not in filtered[entry["git_hash"]]["categories"][category]
-                    ):
-                        filtered[entry["git_hash"]]["categories"][category][
-                            measure
-                        ] = value
-
-    entries = list(filtered.values())
-
-    # Clear entries
-
-    for category in categories.keys():
-        # Delete categories
-
-        url = f"{domain}/projects/{args.project}/{args.version}/{category}/"
-
-        data = {"api_key": api_key}
-
-        debug("DELETE " + url)
-
-        response = requests.delete(url, json=data)
-        print(response.text)
-
-        # Recreate categories
-
-        data["name"] = category
-
-        debug("POST " + url)
-
-        response = requests.post(url, json=data)
-        print(response.text)
-
-    # Upload entries
-
-    url = f"{domain}/data/{args.project}/{args.version}/"
-
-    data = {"api_key": api_key, "entries": entries}
-
-    debug("POST " + url)
-
-    response = requests.post(url, json=data)
-    print(response.status_code, response.text)
-
-    # Check entries
-
-    url = f"{domain}/data/{args.project}/{args.version}?mode=all"
-
-    debug("GET " + url)
-
-    response = requests.get(url)
-    print(response.text)
-
-
 def main() -> None:
     parser = argparse.ArgumentParser()
 
@@ -235,17 +148,6 @@ def main() -> None:
     delete_category_parser.add_argument("slug", help="the slug for the category")
     delete_category_parser.set_defaults(func=delete_category)
 
-    # Prune entries
-    prune_parser = subparsers.add_parser(
-        "prune",
-        help="prune entries to remove duplicates",
-    )
-    prune_parser.add_argument(
-        "project", help="the project for which to prune duplicated entries"
-    )
-    prune_parser.add_argument("version", help="the slug for the version")
-    prune_parser.set_defaults(func=prune_entries)
-
     args = parser.parse_args()
     args.func(args)
 

frog_api/migrations/0010_alter_entry_category_alter_project_discord_and_more.py

@@ -0,0 +1,45 @@
+# Generated by Django 4.2.1 on 2023-05-25 08:56
+
+from django.db import migrations, models
+import django.db.models.deletion
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        (
+            "frog_api",
+            "0001_squashed_0009_alter_project_discord_alter_project_repository_and_more",
+        ),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name="entry",
+            name="category",
+            field=models.ForeignKey(
+                on_delete=django.db.models.deletion.CASCADE, to="frog_api.category"
+            ),
+        ),
+        migrations.AlterField(
+            model_name="project",
+            name="discord",
+            field=models.URLField(blank=True),
+        ),
+        migrations.AlterField(
+            model_name="project",
+            name="repository",
+            field=models.URLField(blank=True),
+        ),
+        migrations.AlterField(
+            model_name="project",
+            name="website",
+            field=models.URLField(blank=True),
+        ),
+        migrations.AddConstraint(
+            model_name="entry",
+            constraint=models.UniqueConstraint(
+                fields=("timestamp", "git_hash", "category"), name="unique entry"
+            ),
+        ),
+    ]

frog_api/models.py

@@ -77,6 +77,11 @@ class Entry(models.Model):
     class Meta:
         verbose_name_plural = "Entries"
         ordering = ["-timestamp"]
+        constraints = [
+            models.UniqueConstraint(
+                fields=["timestamp", "git_hash", "category"], name="unique entry"
+            )
+        ]
 
     def __str__(self) -> str:
         time_string = datetime.utcfromtimestamp(self.timestamp).strftime(