Skip to content

GET_ALL_PAGES_IN_DIR doesn't work for "dirs" full of "subdirs" #73

@hartzell

Description

@hartzell

The fix in #31 doesn't seem to work if you attempt to list a directory (prefix) full of only subdirectories (longer prefixes) as page.Contents is 0 and it stops after the first page.

Various AWS SDK documentation and examples (e.g. here on GitHub just pass in a function that always returns true, apparently the SDK will stop calling it after the last page.

A better fix is something like this:

--- a/internal/service/amazon-s3.go
+++ b/internal/service/amazon-s3.go
@@ -37,7 +37,7 @@ func (c client) S3listObjects(bucket, prefix string) (*s3.ListObjectsOutput, err
                func(page *s3.ListObjectsOutput, lastPage bool) bool {
                        result.CommonPrefixes = append(result.CommonPrefixes, page.CommonPrefixes...)
                        result.Contents = append(result.Contents, page.Contents...)
-                       return len(page.Contents) == 1000
+                       return true
                })
        return result, err
 }

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions