Why

Making post requests to KV is slow. Our workers spend most of the time waiting for the POST requests to finish. We want them to run faster. So that we can avoid hitting the timeout limit of 15 minutes. When the workers hit the timelimit the entire job crashes disabling the entire feature.

How

By accepting post messages within amazon AWS cloud instead

• The service quickly enqueue the messages to AWS-SQS and upload the payload to AWS-S3 (if necessary)
• Running a schedule within the timelimit the dequeue service will upload as many entries as it can manage to KV
• If needed the services can more workers in parallell to process the queue

Features

• Large payload sizes with S3 buckets
• Enqueue and Dequeue parallelizm
• Streaming read and write payload-data to S3
• Skipping and logging of failed entries
• Logging of all successful entries
• Skipping of entries that timeouts
• Debugging flag for direct upload to KV instead of queuing the message (the service will act as a proxy)