'clean' operations don't complete properly if the number of keys in the S3 bucket is large

[kpfleming]

I'm trying to clean out 50,000 (or more) stale assets from a bucket, but the process won't complete because it uses the ListObjectsV2Command which limits the number of returned keys, and the code does not attempt to use pagination to get the full list.

Something like this which uses paginateListObjectsV2 should be used in order to obtain the full list of keys in the bucket.

[kpfleming]

Here is a very hacky implementation I used:

    async getStorageKeys(prefix) {
      const client = this.getS3Client();
      const paginator = paginateListObjectsV2(
        { client, },
        { Bucket: this.s3Bucket,
          Prefix: prefix,
        },
      );

      const objectKeys = [];
      for await (const { Contents } of paginator) {
        objectKeys.push(...Contents.map((obj) => (obj.Key)));
      }

      return objectKeys.filter(c => c != null);
    }