You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
s5cmd supports HTTP persistent connections, which are good for performance (especially for HTTPS), however when using >1 workers the number of requests sent per connection drops significantly, and thus the number of new TCP connections opened goes up. This add latency and overhead due to the additional TCP connections, but also due to TLS negotiation that needs to occur on each new connection.
Testing using s5cmd cp to upload 50,000 objects to an S3 bucket with --workers 1 results in a 2 TCP connection being opened, and half of the requests being sent over each connection.
The same test with --workers 16 results in 10,500 TCP connections being opened, and an average of ~5 requests per connection (although they vary from 1 to around 10 requests per connection). With 64 workers it goes up to ~15,000 connections.
tcpdump confirms the connections are being closed by the client-end, and they are being closed immediately after the previous successful PUT request.
The text was updated successfully, but these errors were encountered:
s5cmd supports HTTP persistent connections, which are good for performance (especially for HTTPS), however when using >1 workers the number of requests sent per connection drops significantly, and thus the number of new TCP connections opened goes up. This add latency and overhead due to the additional TCP connections, but also due to TLS negotiation that needs to occur on each new connection.
Testing using
s5cmd cp
to upload 50,000 objects to an S3 bucket with--workers 1
results in a 2 TCP connection being opened, and half of the requests being sent over each connection.The same test with
--workers 16
results in 10,500 TCP connections being opened, and an average of ~5 requests per connection (although they vary from 1 to around 10 requests per connection). With 64 workers it goes up to ~15,000 connections.tcpdump confirms the connections are being closed by the client-end, and they are being closed immediately after the previous successful PUT request.
The text was updated successfully, but these errors were encountered: