S3 log forwarder timeout

[dsotirho-ucsc]

Failed while processing large (2-3 MB) files from the "logs" bucket.
The files in question contain thousands of lines, each for a S3.EXPIRE.OBJECT action for objects in the "trail" bucket.

log-group-names: /aws/lambda/azul-indexer-prod-forward_s3_logs

@timestamp	@message
2026-05-21 02:36:12.005	START RequestId: ff2f42b6-ef79-4f3a-8b93-a84c26ea6791 Version: 29
2026-05-21 02:36:12.008	[INFO] 2026-05-21T02:36:12.008Z ff2f42b6-ef79-4f3a-8b93-a84c26ea6791 azul.boto3 s3.GetObject: Making GET request to https://s3.amazonaws.com/edu-ucsc-gi-platform-hca-prod-logs.us-east-1/s3/access/prod/cloudtrail/2026-05-21-02-36-11-3A5DF20F3AF46A4F
2026-05-21 02:36:12.008	[INFO] 2026-05-21T02:36:12.008Z ff2f42b6-ef79-4f3a-8b93-a84c26ea6791 azul.boto3 s3.GetObject: … without a request body
2026-05-21 02:36:12.187	[INFO] 2026-05-21T02:36:12.187Z ff2f42b6-ef79-4f3a-8b93-a84c26ea6791 azul.boto3 s3.GetObject: Got 200 response
2026-05-21 02:36:12.187	[INFO] 2026-05-21T02:36:12.187Z ff2f42b6-ef79-4f3a-8b93-a84c26ea6791 azul.boto3 s3.GetObject: … with a response body of type (<class 'botocore.response.StreamingBody'>)
2026-05-21 02:36:43.111	REPORT RequestId: ff2f42b6-ef79-4f3a-8b93-a84c26ea6791 Duration: 31000.00 ms Billed Duration: 31000 ms Memory Size: 160 MB Max Memory Used: 150 MB Status: timeout
2026-05-21 02:36:43.111	END RequestId: ff2f42b6-ef79-4f3a-8b93-a84c26ea6791
...
2026-05-21 22:43:40.913	START RequestId: b8dd20e9-9603-4e6a-bcd2-93a42c91e068 Version: 29
2026-05-21 22:43:40.915	[INFO] 2026-05-21T22:43:40.915Z b8dd20e9-9603-4e6a-bcd2-93a42c91e068 azul.boto3 s3.GetObject: Making GET request to https://s3.amazonaws.com/edu-ucsc-gi-platform-hca-prod-logs.us-east-1/s3/access/prod/cloudtrail/2026-05-21-22-43-39-73CA272783151DAA
2026-05-21 22:43:40.915	[INFO] 2026-05-21T22:43:40.915Z b8dd20e9-9603-4e6a-bcd2-93a42c91e068 azul.boto3 s3.GetObject: … without a request body
2026-05-21 22:43:40.938	[INFO] 2026-05-21T22:43:40.938Z b8dd20e9-9603-4e6a-bcd2-93a42c91e068 azul.boto3 s3.GetObject: Got 200 response
2026-05-21 22:43:40.938	[INFO] 2026-05-21T22:43:40.938Z b8dd20e9-9603-4e6a-bcd2-93a42c91e068 azul.boto3 s3.GetObject: … with a response body of type (<class 'botocore.response.StreamingBody'>)
2026-05-21 22:44:12.006	REPORT RequestId: b8dd20e9-9603-4e6a-bcd2-93a42c91e068 Duration: 31000.00 ms Billed Duration: 31000 ms Memory Size: 160 MB Max Memory Used: 160 MB Status: timeout
2026-05-21 22:44:12.006	END RequestId: b8dd20e9-9603-4e6a-bcd2-93a42c91e068
...
2026-05-22 13:53:31.145	START RequestId: 59bb287e-bcf1-4577-8c37-20af45a78cac Version: 29
2026-05-22 13:53:31.148	[INFO] 2026-05-22T13:53:31.148Z 59bb287e-bcf1-4577-8c37-20af45a78cac azul.boto3 s3.GetObject: … without a request body
2026-05-22 13:53:31.148	[INFO] 2026-05-22T13:53:31.148Z 59bb287e-bcf1-4577-8c37-20af45a78cac azul.boto3 s3.GetObject: Making GET request to https://s3.amazonaws.com/edu-ucsc-gi-platform-hca-prod-logs.us-east-1/s3/access/prod/cloudtrail/2026-05-22-13-53-29-1DAC74C81A18E17B
2026-05-22 13:53:31.196	[INFO] 2026-05-22T13:53:31.196Z 59bb287e-bcf1-4577-8c37-20af45a78cac azul.boto3 s3.GetObject: … with a response body of type (<class 'botocore.response.StreamingBody'>)
2026-05-22 13:53:31.196	[INFO] 2026-05-22T13:53:31.196Z 59bb287e-bcf1-4577-8c37-20af45a78cac azul.boto3 s3.GetObject: Got 200 response
2026-05-22 13:54:02.265	END RequestId: 59bb287e-bcf1-4577-8c37-20af45a78cac
2026-05-22 13:54:02.265	REPORT RequestId: 59bb287e-bcf1-4577-8c37-20af45a78cac Duration: 31000.00 ms Billed Duration: 31000 ms Memory Size: 160 MB Max Memory Used: 149 MB Status: timeout

[achave11-ucsc]

@hannes-ucsc: "Spike to come up with a reliable reproduction. I am thinking using the invoke API action to send the copy of the originally failing event to the Lambda function. The first part of the spike is about figuring out how to do this, using a lower deployment. The second part is executing those steps in prod for the files listed above. We may need to synthesize the event but we may also be able to extract it from the CloudTrail event for the forwarder Lambda execution."