libssh2 read performance. query..

[Kannamraju P via libssh2-devel]

Hi All,

I was running a test to check the speed of SFTP read , A single read
of 60K bytes
with 64K buffer is resulting in these many packets on the wire . Any input
on optimizing
read would be really helpful.




[image: Inline image 1]

[Daniel Stenberg]

I was running a test to check the speed of SFTP read , A single read of
60K bytes with 64K buffer is resulting in these many packets on the wire .
Any input on optimizing read would be really helpful.

It behaves like that to optimize reading! libssh2 will send a lot of small
reads in a "pipelining" manner so that you don't have to wait for any single
large packet to return but it can return data as soon as the first packet has
arrived.

[Kannamraju P via libssh2-devel]

It seems that the reads are done in sequential order , in networks which
have latency this slows down the read quite a bit.
Is there any workaround or configuration at library level or TCP level to
tune this.

I was running a test to check the speed of SFTP read , A single read of

60K bytes with 64K buffer is resulting in these many packets on the wire .
Any input on optimizing read would be really helpful.

It behaves like that to optimize reading! libssh2 will send a lot of small
reads in a "pipelining" manner so that you don't have to wait for any
single large packet to return but it can return data as soon as the first
packet has arrived.
--
/ daniel.haxx.se

[Daniel Stenberg]

Post by Kannamraju P via libssh2-devel
It seems that the reads are done in sequential order , in networks which
have latency this slows down the read quite a bit.

Yes of course they're sequential, since the API implies reading sequentially!

And it is a work-around to *help* when you have long latencies, as I tried to
explain as it allows the first responses to arrive earlier than otherwise. It
is especially effective if you're reading more data in a loop (in a
non-blocking fashion), and not just that single function call.

SFTP is notoriously bad for long-latency connections (since each individual
packet needs to be individually acked the in SFTP protocol layer). This
approach is a way to try to make the effects of this less bad.

Post by Kannamraju P via libssh2-devel
Is there any workaround or configuration at library level or TCP level to
tune this.

We've previously discussed adding some way to allow applications to tweak this
behavior, and in particular the block sizes and read-ahead length libssh2
does, but we've never gotten around to actually do it.

[Kannamraju P via libssh2-devel]

I see some documentation online that some other client libraries like
openssh query these reads parallely and assemble them .
this gives better throughput I believe. Do we have something similar in
libssh2.

Thanks.

Post by Kannamraju P via libssh2-devel
It seems that the reads are done in sequential order , in networks which

Post by Kannamraju P via libssh2-devel
have latency this slows down the read quite a bit.

Yes of course they're sequential, since the API implies reading sequentially!
And it is a work-around to *help* when you have long latencies, as I tried
to explain as it allows the first responses to arrive earlier than
otherwise. It is especially effective if you're reading more data in a loop
(in a non-blocking fashion), and not just that single function call.
SFTP is notoriously bad for long-latency connections (since each
individual packet needs to be individually acked the in SFTP protocol
layer). This approach is a way to try to make the effects of this less bad.
Is there any workaround or configuration at library level or TCP level to

Post by Kannamraju P via libssh2-devel
tune this.

We've previously discussed adding some way to allow applications to tweak
this behavior, and in particular the block sizes and read-ahead length
libssh2 does, but we've never gotten around to actually do it.
--
/ daniel.haxx.se

[Daniel Stenberg]

I see some documentation online that some other client libraries like
openssh query these reads parallely and assemble them . this gives better
throughput I believe. Do we have something similar in libssh2.

That's exactly what this system does. It sends out multiple read packets at
the same time and assembles them when the responses come back. That's what
OpenSSH does as well.