Folder structure fixes to checker and samples #2433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

hanyunfan merged 3 commits into master from structure_fixes

Jan 12, 2026

.github/workflows/test-resnet50.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -35,7 +35,10 @@ jobs: @@
             python-version: ${{ matrix.python-version }}
         - name: Install dependencies
           run: |
-            python3 -m pip install mlc-scripts
+            python3 -m pip install mlcflow
+            mlc pull repo mlcommons@mlperf-automations --branch=subcheckmod_change
+    #python3 -m pip install mlc-scripts
         - name: Test Resnet50 and end to end submission generation
           run: |
-            mlcr run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --hw_name=default --model=resnet50 --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src.version=custom --adr.inference-src-loadgen.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}
+            mlcr run,mlperf,inference,generate-run-cmds,_submission,_short,_r6.0 --quiet --submitter="MLCommons" --hw_name=default --model=resnet50 --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.submission_checker_refactor --adr.inference-src.version=custom --adr.inference-src-loadgen.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}
+    #mlcr run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --hw_name=default --model=resnet50 --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src.version=custom --adr.inference-src-loadgen.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}

.github/workflows/test-retinanet.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -33,7 +33,10 @@ jobs: @@
             python-version: ${{ matrix.python-version }}
         - name: Install dependencies
           run: |
-            python3 -m pip install mlc-scripts
+            python3 -m pip install mlcflow
+            mlc pull repo mlcommons@mlperf-automations --branch=subcheckmod_change
+    #python3 -m pip install mlc-scripts
         - name: Test Retinanet and end to end submission generation
           run: |
-            mlcr run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --hw_name=default --model=retinanet --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=10 --adr.compiler.tags=gcc  --adr.inference-src.version=custom --adr.inference-src.tags=_repo.${{ github.event.pull_request.head.repo.html_url }},_branch.${{ github.event.pull_request.head.ref }} --adr.inference-src-loadgen.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom
+            mlcr run,mlperf,inference,generate-run-cmds,_submission,_short,_r6.0 --quiet --submitter="MLCommons" --hw_name=default --model=retinanet --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=10 --adr.compiler.tags=gcc --adr.accuracy-check-src.tags=_branch.submission_checker_refactor --adr.inference-src.version=custom --adr.inference-src.tags=_branch.submission_checker_refactor --adr.inference-src-loadgen.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom
+    #mlcr run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --hw_name=default --model=retinanet --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=10 --adr.compiler.tags=gcc  --adr.inference-src.version=custom --adr.inference-src.tags=_repo.${{ github.event.pull_request.head.repo.html_url }},_branch.${{ github.event.pull_request.head.ref }} --adr.inference-src-loadgen.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom

.github/workflows/test-rgat.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -32,7 +32,10 @@ jobs: @@
             python-version: ${{ matrix.python-version }}
         - name: Install dependencies
           run: |
-            python3 -m pip install mlc-scripts
+            python3 -m pip install mlcflow
+            mlc pull repo mlcommons@mlperf-automations --branch=subcheckmod_change
+    #python3 -m pip install mlc-scripts
         - name: Test R-GAT and end to end submission generation
           run: |
-            mlcr run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --category=datacenter --hw_name=default --model=rgat --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}
+            mlcr run,mlperf,inference,generate-run-cmds,_submission,_short,_r6.0 --quiet --submitter="MLCommons" --category=datacenter --hw_name=default --model=rgat --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.submission_checker_refactor --adr.inference-src-loadgen.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}
+    #mlcr run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --category=datacenter --hw_name=default --model=rgat --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}

.github/workflows/test-submission-generation.yml

-Original file line number
+Diff line change
@@ -1,4 +1,5 @@
-    # This workflow will test the submission generation using MLPerf Automation
+    # This workflow will test the submission generation using MLPerf Automation.
+    # This test calls the test-mlperf-inference-submission-generation.yml file from automation repo
     name: MLC based Submission Generation
@@ Expand All / @@ -12,7 +13,9 @@ on: @@
     jobs:
       run-tests:
-        uses: mlcommons/mlperf-automations/.github/workflows/test-mlperf-inference-submission-generation.yml@dev
+        uses: mlcommons/mlperf-automations/.github/workflows/test-mlperf-inference-submission-generation.yml@subcheckmod_change
         with:
-          ref: ${{ github.event.pull_request.head.ref }}
-          repo-url: ${{ github.event.pull_request.head.repo.html_url }}
+          inf-branch: ${{ github.event.pull_request.head.ref }}
+          inf-repo-url: ${{ github.event.pull_request.head.repo.html_url }}
+          automation-repo: mlcommons@mlperf-automations
+          automation-repo-branch: subcheckmod_change

0 ...closed/AMD/code/llama2-70b-99.9/README.md → .../closed/AMD/src/llama2-70b-99.9/README.md

File renamed without changes.

0 ...1/closed/AMD/code/llama2-70b-99/README.md → ..._1/closed/AMD/src/llama2-70b-99/README.md

File renamed without changes.

0 ..._1/closed/AMD/code/mixtral-8x7b/README.md → ...e_1/closed/AMD/src/mixtral-8x7b/README.md

File renamed without changes.

0 ...ed/AMD/code/stable-diffusion-xl/README.md → ...sed/AMD/src/stable-diffusion-xl/README.md

File renamed without changes.

0 ...-186GB_aarch64x72_Dynamo_Interactive.json → ...ama3.1-405b/Interactive/measurements.json

File renamed without changes.

0 ...8b-edge/SingleStream/Jetson-Thor_TRT.json → ....1-8b-edge/SingleStream/measurements.json

File renamed without changes.

0 .../open/NVIDIA/code/llama3.1-405b/README.md → ...1/open/NVIDIA/src/llama3.1-405b/README.md

File renamed without changes.

0 ...en/NVIDIA/code/llama3.1-8b-edge/README.md → ...pen/NVIDIA/src/llama3.1-8b-edge/README.md

File renamed without changes.

tools/submission/preprocess_submission.py

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -305,6 +305,8 @@ def clean_invalid_results(args, log_path, config, system_desc, system_json,
  
                    if not args.nomove_failed_to_open:

                        target_code_path = change_first_directory_to_open(

                            model_code_path)

                        target_code_path = change_folder_name_in_path(

                            log_path, "code", "src")

                        target_results_path = change_first_directory_to_open(

                            model_results_path)

                        target_measurements_path = change_first_directory_to_open(

    @@ -543,9 +545,9 @@ def infer_scenario_results(args, config):
  
                                        shutil.copytree(high_accuracy_model_path,

                                                        low_accuracy_model_path)

                                    high_accuracy_model_code_path = os.path.join(log_path, "..",

                                                                                 "code", model)

                                                                                 "src", model)

                                    low_accuracy_model_code_path = os.path.join(log_path, "..",

                                                                                "code", low_accuracy_model)

                                                                                "src", low_accuracy_model)

                                    if not os.path.exists(

                                            low_accuracy_model_code_path):

                                        shutil.copytree(high_accuracy_model_code_path,

tools/submission/submission_checker.py

-Original file line number
+Diff line change
@@ Expand Up / @@ -3093,15 +3093,8 @@ def check_measurement_dir( @@
                     is_valid = False
         for i in files:
-            if i.startswith(system_desc) and i.endswith(
-                    "_" + scenario + ".json"):
+            if i.lower() == "measurements.json":
                 system_file = i
-                end = len("_" + scenario + ".json")
-                break
-            elif i.startswith(system_desc) and i.endswith(".json"):
-                system_file = i
-                end = len(".json")
-                break
         weight_data_types = None
         if system_file:
@@ Expand All / @@ -3117,7 +3110,7 @@ def check_measurement_dir( @@
                         log.error(
                             "%s, field %s is missing meaningful value", fname, k)
-            code_dir = os.path.join(root, "code", model)
+            code_dir = os.path.join(root, "src", model)
             if not os.path.exists(code_dir):
                 # see if the code dir is per model
                 if not os.path.exists(os.path.dirname(code_dir)):
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Folder structure fixes to checker and samples #2433

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!