Merge branch 'main' into ignore-large-json

Author: tdruez

CHANGELOG.rst

@@ -27,6 +27,22 @@ v34.9.4 (unreleased)
 - Add a download action on project list to enable bulk download of Project output files.
   https://github.com/aboutcode-org/scancode.io/issues/1518
 
+- Add labels to Project level search.
+  The labels are now always presented in alphabetical order for consistency.
+  https://github.com/aboutcode-org/scancode.io/issues/1520
+
+- Add a ``batch-create`` management command that allows to create multiple projects
+  at once from a directory containing input files.
+  https://github.com/aboutcode-org/scancode.io/issues/1437
+
+- Add a "TODOS" sheet containing on REQUIRES_REVIEW resources in XLSX.
+  https://github.com/aboutcode-org/scancode.io/issues/1524
+
+- Improve XLSX output for Vulnerabilities.
+  Replace the ``affected_by_vulnerabilities`` field in the PACKAGES and DEPENDENCIES
+  sheets with a dedicated VULNERABILITIES sheet.
+  https://github.com/aboutcode-org/scancode.io/issues/1519
+
 v34.9.3 (2024-12-31)
 --------------------
 

docs/command-line-interface.rst

@@ -57,6 +57,7 @@ ScanPipe's own commands are listed under the ``[scanpipe]`` section::
       add-input
       add-pipeline
       archive-project
+      batch-create
       check-compliance
       create-project
       create-user
@@ -83,7 +84,8 @@ For example::
     $ scanpipe create-project --help
     usage: scanpipe create-project [--input-file INPUTS_FILES]
         [--input-url INPUT_URLS] [--copy-codebase SOURCE_DIRECTORY]
-        [--pipeline PIPELINES] [--execute] [--async]
+        [--pipeline PIPELINES] [--label LABELS] [--notes NOTES]
+        [--execute] [--async]
         name
 
     Create a ScanPipe project.
@@ -124,6 +126,10 @@ Optional arguments:
 - ``--copy-codebase SOURCE_DIRECTORY`` Copy the content of the provided source directory
   into the :guilabel:`codebase/` work directory.
 
+- ``--notes NOTES`` Optional notes about the project.
+
+- ``--label LABELS`` Optional labels for the project.
+
 - ``--execute`` Execute the pipelines right after project creation.
 
 - ``--async`` Add the pipeline run to the tasks queue for execution by a worker instead
@@ -133,6 +139,90 @@ Optional arguments:
 .. warning::
     Pipelines are added and are executed in order.
 
+.. _cli_batch_create:
+
+`$ scanpipe batch-create [--input-directory INPUT_DIRECTORY] [--input-list FILENAME.csv]`
+-----------------------------------------------------------------------------------------
+
+Processes files from the specified ``INPUT_DIRECTORY`` or rows from ``FILENAME.csv``,
+creating a project for each file or row.
+
+- Use ``--input-directory`` to specify a local directory. Each file in the directory
+  will result in a project, uniquely named using the filename and a timestamp.
+
+- Use ``--input-list`` to specify a ``FILENAME.csv``. Each row in the CSV will be used
+  to create a project based on the data provided.
+
+Supports specifying pipelines and asynchronous execution.
+
+Required arguments (one of):
+
+- ``input-directory`` The path to the directory containing the input files to process.
+  Ensure the directory exists and contains the files you want to use.
+
+- ``input-list`` Path to a CSV file with project names and input URLs.
+  The first column must contain project names, and the second column should list
+  comma-separated input URLs (e.g., Download URL, PURL, or Docker reference).
+
+  **CSV content example**:
+
+  +----------------+---------------------------------+
+  | project_name   | input_urls                      |
+  +================+=================================+
+  | project-1      | https://url.com/file.ext        |
+  +----------------+---------------------------------+
+  | project-2      | pkg:deb/debian/[email protected]      |
+  +----------------+---------------------------------+
+
+Optional arguments:
+
+- ``--project-name-suffix`` Optional custom suffix to append to project names.
+  If not provided, a timestamp (in the format [YYMMDD_HHMMSS]) will be used.
+
+- ``--pipeline PIPELINES`` Pipelines names to add on the project.
+
+- ``--notes NOTES`` Optional notes about the project.
+
+- ``--label LABELS`` Optional labels for the project.
+
+- ``--execute`` Execute the pipelines right after project creation.
+
+- ``--async`` Add the pipeline run to the tasks queue for execution by a worker instead
+  of running in the current thread.
+  Applies only when ``--execute`` is provided.
+
+Example: Processing Multiple Docker Images
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Assume multiple Docker images are available in a directory named ``local-data/`` on
+the host machine.
+To process these images with the ``analyze_docker_image`` pipeline using asynchronous
+execution::
+
+    $ docker compose run --rm \
+        --volume local-data/:/input-data:ro \
+        web scanpipe batch-create input-data/ \
+            --pipeline analyze_docker_image \
+            --label "Docker" \
+            --execute --async
+
+**Explanation**:
+
+- ``local-data/``: A directory on the host machine containing the Docker images to
+  process.
+- ``/input-data/``: The directory inside the container where ``local-data/`` is
+  mounted (read-only).
+- ``--pipeline analyze_docker_image``: Specifies the ``analyze_docker_image``
+  pipeline for processing each Docker image.
+- ``--label "Docker"``: Tagging all the projects with the "Docker" label to enable
+  easy search and filtering.
+- ``--execute``: Runs the pipeline immediately after creating a project for each
+  image.
+- ``--async``: Adds the pipeline run to the worker queue for asynchronous execution.
+
+Each Docker image in the ``local-data/`` directory will result in the creation of a
+project with the specified pipeline (``analyze_docker_image``) executed by worker
+services.
 
 `$ scanpipe list-pipeline [--verbosity {0,1,2,3}]`
 --------------------------------------------------

docs/faq.rst

@@ -108,6 +108,33 @@ It does not compute such summary.
 You can also have a look at the different steps for each pipeline from the
 :ref:`built_in_pipelines` documentation.
 
+How to create multiple projects at once?
+-----------------------------------------
+
+You can use the :ref:`cli_batch_create` command to create multiple projects
+simultaneously.
+This command processes all files in a specified input directory, creating one project
+per file.
+Each project is uniquely named using the file name and a timestamp by default.
+
+For example, to create multiple projects from files in a directory named
+``local-data/``::
+
+    $ docker compose run --rm \
+        --volume local-data/:/input-data:ro \
+        web scanpipe batch-create input-data/
+
+**Options**:
+
+- **Custom Pipelines**: Use the ``--pipeline`` option to add specific pipelines to the
+  projects.
+- **Asynchronous Execution**: Add ``--execute`` and ``--async`` to queue pipeline
+  execution for worker processing.
+- **Project Notes and Labels**: Use ``--notes`` and ``--label`` to include metadata.
+
+Each file in the input directory will result in the creation of a corresponding project,
+ready for pipeline execution.
+
 Can I run multiple pipelines in parallel?
 -----------------------------------------
 
@@ -279,7 +306,7 @@ data older than 7 days::
   See :ref:`command_line_interface` chapter for more information about the scanpipe
   command.
 
-How can I provide my license policies ?
+How can I provide my license policies?
 ---------------------------------------
 
 For detailed information about the policies system, refer to :ref:`policies`.

scanpipe/api/views.py

@@ -232,15 +232,17 @@ def packages(self, request, *args, **kwargs):
     @action(detail=True, filterset_class=None)
     def dependencies(self, request, *args, **kwargs):
         project = self.get_object()
-        queryset = project.discovereddependencies.all()
+        queryset = project.discovereddependencies.prefetch_for_serializer()
         return self.get_filtered_response(
             request, queryset, DependencyFilterSet, DiscoveredDependencySerializer
         )
 
     @action(detail=True, filterset_class=None)
     def relations(self, request, *args, **kwargs):
         project = self.get_object()
-        queryset = project.codebaserelations.all()
+        queryset = project.codebaserelations.select_related(
+            "from_resource", "to_resource"
+        )
         return self.get_filtered_response(
             request, queryset, RelationFilterSet, CodebaseRelationSerializer
         )

scanpipe/filters.py

@@ -245,15 +245,19 @@ def filter_queryset(self, queryset):
         `empty_value` to any filters.
         """
         for name, value in self.form.cleaned_data.items():
-            field_name = self.filters[name].field_name
-            if value == self.empty_value:
+            filter_field = self.filters[name]
+            field_name = filter_field.field_name
+
+            if isinstance(filter_field, QuerySearchFilter):
+                queryset = filter_field.filter(queryset, value)
+            elif value == self.empty_value:
                 queryset = queryset.filter(**{f"{field_name}__in": EMPTY_VALUES})
             elif value == self.any_value:
                 queryset = queryset.filter(~Q(**{f"{field_name}__in": EMPTY_VALUES}))
             elif value == self.other_value and hasattr(queryset, "less_common"):
                 return queryset.less_common(name)
             else:
-                queryset = self.filters[name].filter(queryset, value)
+                queryset = filter_field.filter(queryset, value)
 
         return queryset
 
@@ -266,7 +270,7 @@ def filter_for_lookup(cls, field, lookup_type):
         return super().filter_for_lookup(field, lookup_type)
 
 
-def parse_query_string_to_lookups(query_string, default_lookup_expr, default_field):
+def parse_query_string_to_lookups(query_string, default_lookup_expr, search_fields):
     """Parse a query string and convert it into queryset lookups using Q objects."""
     lookups = Q()
     terms = shlex.split(query_string)
@@ -295,11 +299,14 @@ def parse_query_string_to_lookups(query_string, default_lookup_expr, default_fie
                 field_name = field_name[1:]
                 negated = True
 
+            lookups &= Q(
+                **{f"{field_name}__{lookup_expr}": search_value}, _negated=negated
+            )
+
         else:
             search_value = term
-            field_name = default_field
-
-        lookups &= Q(**{f"{field_name}__{lookup_expr}": search_value}, _negated=negated)
+            for field_name in search_fields:
+                lookups |= Q(**{f"{field_name}__{lookup_expr}": search_value})
 
     return lookups
 
@@ -323,18 +330,22 @@ class QuerySearchFilter(django_filters.CharFilter):
 
     field_class = QuerySearchField
 
+    def __init__(self, search_fields=None, lookup_expr="icontains", *args, **kwargs):
+        super().__init__(lookup_expr=lookup_expr, *args, **kwargs)
+        self.search_fields = search_fields or []
+
     def filter(self, qs, value):
         if not value:
             return qs
 
         lookups = parse_query_string_to_lookups(
             query_string=value,
             default_lookup_expr=self.lookup_expr,
-            default_field=self.field_name,
+            search_fields=self.search_fields,
         )
 
         try:
-            return qs.filter(lookups)
+            return qs.filter(lookups).distinct()
         except FieldError:
             return qs.none()
 
@@ -347,7 +358,7 @@ class ProjectFilterSet(FilterSetUtilsMixin, django_filters.FilterSet):
     ]
 
     search = QuerySearchFilter(
-        label="Search", field_name="name", lookup_expr="icontains"
+        label="Search", search_fields=["name", "labels__name"], lookup_expr="icontains"
     )
     sort = django_filters.OrderingFilter(
         label="Sort",
@@ -412,8 +423,10 @@ def __init__(self, data=None, *args, **kwargs):
         if not data or data.get("is_archived", "") == "":
             self.queryset = self.queryset.filter(is_archived=False)
 
-        active_count = Project.objects.filter(is_archived=False).count()
-        archived_count = Project.objects.filter(is_archived=True).count()
+        counts = Project.objects.get_active_archived_counts()
+        active_count = counts["active_count"]
+        archived_count = counts["archived_count"]
+
         self.filters["is_archived"].extra["widget"] = BulmaLinkWidget(
             choices=[
                 ("", f'<i class="fa-solid fa-seedling"></i> {active_count} Active'),
@@ -508,7 +521,7 @@ class ResourceFilterSet(FilterSetUtilsMixin, django_filters.FilterSet):
 
     search = QuerySearchFilter(
         label="Search",
-        field_name="path",
+        search_fields=["path"],
         lookup_expr="icontains",
     )
     sort = django_filters.OrderingFilter(
@@ -615,15 +628,7 @@ def filter(self, qs, value):
         if value.startswith("pkg:"):
             return qs.for_package_url(value)
 
-        if ":" in value:
-            return super().filter(qs, value)
-
-        search_fields = ["type", "namespace", "name", "version"]
-        lookups = Q()
-        for field_names in search_fields:
-            lookups |= Q(**{f"{field_names}__{self.lookup_expr}": value})
-
-        return qs.filter(lookups)
+        return super().filter(qs, value)
 
 
 class GroupOrderingFilter(django_filters.OrderingFilter):
@@ -662,7 +667,9 @@ class PackageFilterSet(FilterSetUtilsMixin, django_filters.FilterSet):
     ]
 
     search = DiscoveredPackageSearchFilter(
-        label="Search", field_name="name", lookup_expr="icontains"
+        label="Search",
+        search_fields=["type", "namespace", "name", "version"],
+        lookup_expr="icontains",
     )
     sort = GroupOrderingFilter(
         label="Sort",
@@ -746,7 +753,7 @@ class DependencyFilterSet(FilterSetUtilsMixin, django_filters.FilterSet):
     ]
 
     search = QuerySearchFilter(
-        label="Search", field_name="name", lookup_expr="icontains"
+        label="Search", search_fields=["name"], lookup_expr="icontains"
     )
     sort = GroupOrderingFilter(
         label="Sort",
@@ -803,7 +810,7 @@ class Meta:
 
 class ProjectMessageFilterSet(FilterSetUtilsMixin, django_filters.FilterSet):
     search = QuerySearchFilter(
-        label="Search", field_name="description", lookup_expr="icontains"
+        label="Search", search_fields=["description"], lookup_expr="icontains"
     )
     sort = django_filters.OrderingFilter(
         label="Sort",
@@ -855,7 +862,7 @@ class RelationFilterSet(FilterSetUtilsMixin, django_filters.FilterSet):
 
     search = QuerySearchFilter(
         label="Search",
-        field_name="to_resource__path",
+        search_fields=["to_resource__path"],
         lookup_expr="icontains",
     )
     sort = django_filters.OrderingFilter(

scanpipe/forms.py

@@ -272,6 +272,23 @@ class ProjectOutputDownloadForm(forms.Form):
     )
 
 
+class ProjectReportForm(forms.Form):
+    model_name = forms.ChoiceField(
+        label="Choose the object type to include in the XLSX file",
+        choices=[
+            ("discoveredpackage", "Packages"),
+            ("discovereddependency", "Dependencies"),
+            ("codebaseresource", "Resources"),
+            ("codebaserelation", "Relations"),
+            ("projectmessage", "Messages"),
+            ("todos", "TODOs"),
+        ],
+        required=True,
+        initial="discoveredpackage",
+        widget=forms.RadioSelect,
+    )
+
+
 class ListTextarea(forms.CharField):
     """
     A Django form field that displays as a textarea and converts each line of input