Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP connections less persistent with >1 worker #797

Open
docbert opened this issue Feb 19, 2025 · 0 comments
Open

HTTP connections less persistent with >1 worker #797

docbert opened this issue Feb 19, 2025 · 0 comments

Comments

@docbert
Copy link

docbert commented Feb 19, 2025

s5cmd supports HTTP persistent connections, which are good for performance (especially for HTTPS), however when using >1 workers the number of requests sent per connection drops significantly, and thus the number of new TCP connections opened goes up. This add latency and overhead due to the additional TCP connections, but also due to TLS negotiation that needs to occur on each new connection.

Testing using s5cmd cp to upload 50,000 objects to an S3 bucket with --workers 1 results in a 2 TCP connection being opened, and half of the requests being sent over each connection.

The same test with --workers 16 results in 10,500 TCP connections being opened, and an average of ~5 requests per connection (although they vary from 1 to around 10 requests per connection). With 64 workers it goes up to ~15,000 connections.

tcpdump confirms the connections are being closed by the client-end, and they are being closed immediately after the previous successful PUT request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant