How to increase performance of libssh2 SFTP Read/Write

[Kamal Bhatt]

I am reading/writing to Remote SFTP server using libssh2 library. However
there is very poor performance compared to the sftp utility comes along with
UNIX.

I am using *libssh2-0.14 *version.

I tried to investigate the library and found that for Read maximum packet
size is 40000 bytes and for write there is a loop going which will send the
data in chunks of 32KB.

I then tried to increase the SFTP Packet size by changing
LIBSSH2_SFTP_PACKET_MAXLEN to 64KB hoping that then we can read/write in
chunks of 64KB but of no use since after building the library even it was
not able to establish the session with the remote SFTP server.

I also tried changing various constants like
LIBSSH2_PACKET_MAXPAYLOAD
LIBSSH2_PACKET_MAXCOMP
LIBSSH2_PACKET_MAXDECOMP
LIBSSH2_CHANNEL_WINDOW_DEFAULT

But still I was not able to increase the performance.

Is there some way that by changing these constants or some other things we
can increase the performance SFTP read/write using libssh2 or any other
workaround so as to how to increase the performance?

Any pointers from you will be of great help.

( Please help me about workarounds on *libssh2-0.14 *version for increasing
perofrmance as I have tested my entire application using this version of
library and moving to new version will be very difficult for me)

Thanks in advance.

[Mononen Jussi]

Post by Kamal Bhatt
I am reading/writing to Remote SFTP server using libssh2
library. However there is very poor performance compared to
the sftp utility comes along with UNIX.
I am using libssh2-0.14 version.
I tried to investigate the library and found that for Read
maximum packet size is 40000 bytes and for write there is a
loop going which will send the data in chunks of 32KB.
I then tried to increase the SFTP Packet size by changing
LIBSSH2_SFTP_PACKET_MAXLEN to 64KB hoping that then we can
read/write in chunks of 64KB but of no use since after
building the library even it was not able to establish the
session with the remote SFTP server.
I also tried changing various constants like
LIBSSH2_PACKET_MAXPAYLOAD
LIBSSH2_PACKET_MAXCOMP
LIBSSH2_PACKET_MAXDECOMP
LIBSSH2_CHANNEL_WINDOW_DEFAULT
But still I was not able to increase the performance.
Is there some way that by changing these constants or some
other things we can increase the performance SFTP read/write
using libssh2 or any other workaround so as to how to
increase the performance?

Hi,

maximum packet size is just the protocol (SSH2) maximum packet size. If
packet is larger than this it must be rejected, this is defined in the
SECSH RFC's (recommended maxlength is 35000 bytes). But the main problem
with read-function performance is that it reads the data in 16 byte
blocks (see packet.c line 733). This is fixed in the latest CVS version
where it reads 4096 bytes in (afaik).

Also, when comparing to command line tools (like OpenSSH sftp), libssh2
is much slower because its approach to sending/receiving is synchronous
where as the command line tools use asynchronous approach. When libssh2
sends onepacket, it can not send another before the previous one is
acknowledged. Command line tools do nothave this restriction and they
utilize the bandwidth to the maximum.

The libssh2_packet_read performance problem can be fixed easily, but the
async vs. sync transfer is a bit tougher issue to solve.

br,

/jUSSi
--
perl -e 'print for reverse @{[@{[@{[@{
[@{[Hacker,Perl,Another,Just]}]}]}]}]};'

____________________________________________________________________________________
Comptel Finland is moving to new premises.
Starting from July 9 our new visiting address is Salmisaarenaukio 1, 00180 Helsinki,
and the postal address is P.O.Box 1000, FI-00181 Helsinki
____________________________________________________________________________________

Disclaimer: This message and any attachments thereto are intended solely for
the addressed recipient(s) and may contain confidential information. If you
are not the intended recipient, please notify the sender by reply e-mail and
delete the e-mail (including any attachments thereto) without producing,
distributing or retaining any copies thereof. Any review, dissemination or
other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient(s) is prohibited.
Thank you.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Mononen Jussi]

Post by Mononen Jussi
maximum packet size is just the protocol (SSH2) maximum
packet size. If packet is larger than this it must be rejected

Just to correct myself, it is not a MUST to reject larger packets, but
application MUST be able to handle atleast 35000 byte packets.

/jUSSi
--
perl -e '$@=[[0,0112,0,0],[0,0101,0,0],
[0,0120,0,0],[0,0110,0,0]];$[=256/8-0x1D;
print(chr(@{$_}[+($[+1-1+1)]))for(@{$@});'

____________________________________________________________________________________
Comptel Finland is moving to new premises.
Starting from July 9 our new visiting address is Salmisaarenaukio 1, 00180 Helsinki,
and the postal address is P.O.Box 1000, FI-00181 Helsinki
____________________________________________________________________________________

Disclaimer: This message and any attachments thereto are intended solely for
the addressed recipient(s) and may contain confidential information. If you
are not the intended recipient, please notify the sender by reply e-mail and
delete the e-mail (including any attachments thereto) without producing,
distributing or retaining any copies thereof. Any review, dissemination or
other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient(s) is prohibited.
Thank you.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Daniel Stenberg]

Also, when comparing to command line tools (like OpenSSH sftp), libssh2 is
much slower because its approach to sending/receiving is synchronous where
as the command line tools use asynchronous approach. When libssh2 sends
onepacket, it can not send another before the previous one is acknowledged.

At what packet level is this "acknowledgement" done? I don't see any such ACK
packets in RFC4253, or am I just blind?

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Mononen Jussi]

Post by Mononen Jussi
Also, when comparing to command line tools (like OpenSSH

sftp), libssh2 is

Post by Mononen Jussi
much slower because its approach to sending/receiving is

synchronous where

Post by Mononen Jussi
as the command line tools use asynchronous approach. When

libssh2 sends

Post by Mononen Jussi
onepacket, it can not send another before the previous one

is acknowledged.
At what packet level is this "acknowledgement" done? I don't
see any such ACK
packets in RFC4253, or am I just blind?

"Acknowledgment" is a wrong word, sorry. Libssh2 blocks while it waits
for response to a sent request. Command line tools do not have this
limitation. They have their own "stack" implementation where they fire
away tens of requests for data after first few successfull requests and
they take care of the ordering of received responses based on TCP
sequences. Thus the bandwidth is never unused as the requests and
responses are fulfilling it to its capacity.

Sorry for the incorrect and confusing wording.

/jUSSi
--
perl -e '$@=[[0,0112,0,0],[0,0101,0,0],
[0,0120,0,0],[0,0110,0,0]];$[=256/8-0x1D;
print(chr(@{$_}[+($[+1-1+1)]))for(@{$@});'

____________________________________________________________________________________
Comptel Finland is moving to new premises.
Starting from July 9 our new visiting address is Salmisaarenaukio 1, 00180 Helsinki,
and the postal address is P.O.Box 1000, FI-00181 Helsinki
____________________________________________________________________________________

Disclaimer: This message and any attachments thereto are intended solely for
the addressed recipient(s) and may contain confidential information. If you
are not the intended recipient, please notify the sender by reply e-mail and
delete the e-mail (including any attachments thereto) without producing,
distributing or retaining any copies thereof. Any review, dissemination or
other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient(s) is prohibited.
Thank you.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Daniel Stenberg]

"Acknowledgment" is a wrong word, sorry. Libssh2 blocks while it waits for
response to a sent request. Command line tools do not have this limitation.
They have their own "stack" implementation where they fire away tens of
requests for data after first few successfull requests and they take care of
the ordering of received responses based on TCP sequences. Thus the
bandwidth is never unused as the requests and responses are fulfilling it to
its capacity.

Ok thanks, I'm trying to understand this to see what can be done about this to
improve libssh2 performance.

What requests do libssh2 send that it waits for a response to that the openssh
tools don't? I'm looking at the code for a simple SCP download case, and I
don't see it.

Can you be a bit more specific and mention exact protocol details or perhaps
libssh2 source code functions/snippets that is the culprit of this mentioned
restriction?

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Mononen Jussi]

Post by Daniel Stenberg
Ok thanks, I'm trying to understand this to see what can be
done about this to
improve libssh2 performance.
What requests do libssh2 send that it waits for a response
to that the openssh
tools don't? I'm looking at the code for a simple SCP
download case, and I
don't see it.
Can you be a bit more specific and mention exact protocol
details or perhaps
libssh2 source code functions/snippets that is the culprit
of this mentioned
restriction?

The most critical part is file transfer. Consider that you are
transferring a 2MB file, you send one request (libssh2_sftp_read,
SSH_FXP_READ) for 32kb of data. Then you wait for the response packet
(libssh2_sftp_packet_requirev, SSH_FXP_DATA, SSH_FXP_STATUS). Then you
send the request again and wait a while to get the response. And so on.

If the first request is successfull you could send three requests in a
row and then wait for the responses and data. If they are successfull,
then let's send 10 requests and wait for responses. This would consume
the bandwidth more efficiently as the bandwidth would be in use while
the server processes our request. This is what I mean with asynchronous
transfer.

This could be implemented by adding a full function that gets/puts one
file similarly to command line tools. Now that the API provides
read/write functions to get a certain amount of data we are bound to use
synchronous approach.

Hopefully this makes sense :-)

/jUSSi
--
perl -e '@_=qw{Just Perl Another Hacker}; sub r{+{@_}}
print keys %{&r}; print values %{&r};'


____________________________________________________________________________________
Comptel Finland is moving to new premises.
Starting from July 9 our new visiting address is Salmisaarenaukio 1, 00180 Helsinki,
and the postal address is P.O.Box 1000, FI-00181 Helsinki
____________________________________________________________________________________

Disclaimer: This message and any attachments thereto are intended solely for
the addressed recipient(s) and may contain confidential information. If you
are not the intended recipient, please notify the sender by reply e-mail and
delete the e-mail (including any attachments thereto) without producing,
distributing or retaining any copies thereof. Any review, dissemination or
other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient(s) is prohibited.
Thank you.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Daniel Stenberg]

On Wed, 13 Jun 2007, Mononen Jussi wrote:

Thanks for your patience and explanations.

Post by Mononen Jussi
The most critical part is file transfer.

You're then referring to SFTP here. But AFAIK, libssh2 is also slower on pure
SCP transfers... Isn't it? Did anyone actually measure libssh2 transfers
recently and compared them to the openssh tools?

Post by Mononen Jussi
Consider that you are transferring a 2MB file, you send one request
(libssh2_sftp_read, SSH_FXP_READ) for 32kb of data. Then you wait for the
response packet (libssh2_sftp_packet_requirev, SSH_FXP_DATA,
SSH_FXP_STATUS). Then you send the request again and wait a while to get the
response. And so on.

I get it, it is quite similar to doing HTTP with or without pipelining. Each
request/response adds a protocol roundtrip.

Post by Mononen Jussi
If the first request is successfull you could send three requests in a row
and then wait for the responses and data. If they are successfull, then
let's send 10 requests and wait for responses. This would consume the
bandwidth more efficiently as the bandwidth would be in use while the server
processes our request. This is what I mean with asynchronous transfer.

I wouldn't call it asynchronous, I would all it pipelining (since the data
would still be synchronously sent) but then I might just be http damaged! ;-)
But yeah, I can see how that approach can boost performance a fair amount.
Especially on connections with high latency.

Post by Mononen Jussi
This could be implemented by adding a full function that gets/puts one file
similarly to command line tools. Now that the API provides read/write
functions to get a certain amount of data we are bound to use synchronous
approach.

I disagree with that conclusion. First, there's of course nothing that
prevents us from adding another or a modified API that somehow makes this
easier, but I could very well consider just an added function/option to the
library that would make libssh2 ask for several chunks of data at once and
just build an internal buffer to return to the API when asked for (of course
limited to a certain extent beyond what has been asked for). The option would
be needed because it would of course risk asking for and transferring more
data than what the application would otherwise want. And when sending data to
a peer, the function could accept a large chunk of data split it up and send
it off in many pieces more or less at once without necessarily waiting for
each packet to first receive success. That will of course require that apps
send very large data chunks to the libssh2 API, but should be a matter of
documentation.

Of course this kind of pipelining approach will make error handling somewhat
tricker.

Looking at the code, the current approach is even more limited than what has
been explained here. Each call to libssh2_sftp_read() (and libssh2_sftp_write)
is directly mapped to an underlying SFTP packet data request size, so if you
happen to call libssh2_sftp_read() with a buffer size less than 32KB or 40KB
or so, you simply will get worse performance since every call to this function
will cause a protocol round-trip.

Personally I would prefer to start with improving SCP as that's a much simpler
protocol and if that is still slower than openssh I think we should focus on
getting that up to speed first and then attack SFTP.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Mononen Jussi]

Post by Daniel Stenberg

Post by Mononen Jussi
The most critical part is file transfer.

You're then referring to SFTP here. But AFAIK, libssh2 is
also slower on pure
SCP transfers... Isn't it? Did anyone actually measure
libssh2 transfers
recently and compared them to the openssh tools?

You are right, it affects scp also. When we started using libssh2 in our
product we received complaints about the performance. We did some tests
with libssh2-0.12 (with our own patch to change the 16 byte read loop
into 32kb read loop) and we came to a conclusion that libssh2 is roughly
50% slower in file transfer than the command line counterparts. Eg. we
had one test run where we transferred a 100MB file, command line scp
took 58 seconds and libssh2 a bit over 2 minutes.

Post by Daniel Stenberg

Post by Mononen Jussi
Consider that you are transferring a 2MB file, you send

I wouldn't call it asynchronous, I would all it pipelining
(since the data
would still be synchronously sent) but then I might just be
http damaged! ;-)

Terminology scherminology ;-) but pipelining is a better term for it.

Post by Daniel Stenberg
I disagree with that conclusion. First, there's of course
nothing that prevents us from adding another or a modified
API that somehow makes this easier, but I could very well
consider just an added function/option to the library that
would make libssh2 ask for several chunks of data at once
and just build an internal buffer to return to the API when
asked for (of course limited to a certain extent beyond
what has been asked for). The option would be needed
because it would of course risk asking for and transferring
more data than what the application would otherwise want.
And when sending data to a peer, the function could accept
a large chunk of data split it up and send it off in many
pieces more or less at once without necessarily waiting for
each packet to first receive success. That will of course
require that apps send very large data chunks to the
libssh2 API, but should be a matter of documentation.

libssh2_sftp_read_many(LIBSSH2_SFTP_HANDLE *handle,
char *buffer,
size_t buffer_maxlen,
ulong upto)

Where 'upto' would specify the expected size of the target of the read
operation, for example the file size. Appropriate timeouts would be
applied, just a bit longer than for a normal read opreation. If upto is
zero, function would fall back to libssh2_sftp_read().

Would that make any sense?

Post by Daniel Stenberg
Of course this kind of pipelining approach will make error
handling somewhat tricker.

well, it might block a bit longer, but basicly the requests either
succeed or fail. The function could remove all the received data and
return an error or it could always return the amount successfully
received sequential data and the application should check whether it
matches the expected amount of data (eg. upto).

Post by Daniel Stenberg
Personally I would prefer to start with improving SCP as
that's a much simpler
protocol and if that is still slower than openssh I think we
should focus on
getting that up to speed first and then attack SFTP.

But SCP is not compatible with all SSH implementations. We are using
libssh2 in mixed environments and sometimes OpenSSH jsuts isn't an
option.

br,

/jUSSi
--
perl -e '$@=[[0,0112,0,0],[0,0101,0,0],
[0,0120,0,0],[0,0110,0,0]];$[=256/8-0x1D;
print(chr(@{$_}[+($[+1-1+1)]))for(@{$@});'

____________________________________________________________________________________
Comptel Finland is moving to new premises.
Starting from July 9 our new visiting address is Salmisaarenaukio 1, 00180 Helsinki,
and the postal address is P.O.Box 1000, FI-00181 Helsinki
____________________________________________________________________________________

Disclaimer: This message and any attachments thereto are intended solely for
the addressed recipient(s) and may contain confidential information. If you
are not the intended recipient, please notify the sender by reply e-mail and
delete the e-mail (including any attachments thereto) without producing,
distributing or retaining any copies thereof. Any review, dissemination or
other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient(s) is prohibited.
Thank you.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Chris Nystrom]

Post by Mononen Jussi
But SCP is not compatible with all SSH implementations. We are using
libssh2 in mixed environments and sometimes OpenSSH jsuts isn't an
option.

I am not doubting what you say, I am just curious: what environments
have ssh and sftp, but not scp?

Chris
--
E-Mail: Chris Nystrom <cnystrom-***@public.gmane.org>
Saving the world from web programming.
http://www.newio.org/ - AIM: nystromchris

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Mononen Jussi]

Post by Mononen Jussi

Post by Mononen Jussi
But SCP is not compatible with all SSH implementations. We

are using

Post by Mononen Jussi
libssh2 in mixed environments and sometimes OpenSSH jsuts isn't an
option.

I am not doubting what you say, I am just curious: what environments
have ssh and sftp, but not scp?

The uncompatibility is not about the presence of the components. At
least the older versions of Tectia product family do not support OpenSSH
SCP. Basicly SCP is just an implementation of rcp and OpenSSH and SSH
Corp. have done it differently. I have a vague recollection that the
latest versions of Tectia product family can be confiugured to support
OpenSSH SCP. SCP is not a standard where as SFTP is (or at least a draft
;-).

And some of our customers decline to install OpenSSH because they use
commercial products.

br,

/jUSSi
--
perl -e '$@=[[0,0112,0,0],[0,0101,0,0],
[0,0120,0,0],[0,0110,0,0]];$[=256/8-0x1D;
print(chr(@{$_}[+($[+1-1+1)]))for(@{$@});'

____________________________________________________________________________________
Comptel Finland is moving to new premises.
Starting from July 9 our new visiting address is Salmisaarenaukio 1, 00180 Helsinki,
and the postal address is P.O.Box 1000, FI-00181 Helsinki
____________________________________________________________________________________

Disclaimer: This message and any attachments thereto are intended solely for
the addressed recipient(s) and may contain confidential information. If you
are not the intended recipient, please notify the sender by reply e-mail and
delete the e-mail (including any attachments thereto) without producing,
distributing or retaining any copies thereof. Any review, dissemination or
other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient(s) is prohibited.
Thank you.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Gutjahr, Troy]

I'm using libssh2 on Solaris to retrieve files via sftp from a
telecommunications network element that has (adapted) OpenSSH code
embedded in it. As far as I know, I can't use scp with this device.
Also, my requirements explictly force me to use sftp.

Faster sftp would be a good thing but so far my tests indicate that
libssh2 is fast enough for my purposes, although it eats lots of CPU
cycles when I'm getting several large files simultaneously in separate
threads.

-- Troy

-----Original Message-----
From: libssh2-devel-bounces-5NWGOfrQmneRv+***@public.gmane.org
[mailto:libssh2-devel-bounces-5NWGOfrQmneRv+***@public.gmane.org] On Behalf Of Chris
Nystrom
Sent: Thursday, June 14, 2007 3:02 AM
To: libssh2-devel-5NWGOfrQmneRv+***@public.gmane.org
Cc: Mononen Jussi
Subject: Re: [libssh2] How to increase performance of libssh2 SFTP
Read/Write

Post by Mononen Jussi
But SCP is not compatible with all SSH implementations. We are using
libssh2 in mixed environments and sometimes OpenSSH jsuts isn't an
option.

I am not doubting what you say, I am just curious: what environments
have ssh and sftp, but not scp?

Chris

--
E-Mail: Chris Nystrom <cnystrom-***@public.gmane.org> Saving the world from web
programming.
http://www.newio.org/ - AIM: nystromchris

------------------------------------------------------------------------
-
This SF.net email is sponsored by DB2 Express Download DB2 Express C -
the FREE version of DB2 express and take control of your XML. No limits.
Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
libssh2-devel mailing list
libssh2-devel-5NWGOfrQmneRv+***@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/libssh2-devel
============================================================
The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader
of this message is not the intended recipient, or an employee
or agent responsible for delivering this message to the
intended recipient, you are hereby notified that any reproduction,
dissemination or distribution of this communication is strictly
prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer. Thank you. Tellabs
============================================================

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Gutjahr, Troy]

These statements at the end of libssh2_sftp_close_handle() seem like a
bug to me. You can't modify the memory to which handle points after you
free it, right?

LIBSSH2_FREE(session, handle->handle);
LIBSSH2_FREE(session, handle);

handle->close_state = libssh2_NB_state_idle;

Jim: What do you think?

By the way, I used the libumem.so.1 library of Solaris 9 to find this
bug. It's quite nifty. Here is some info about it:
http://access1.sun.com/techarticles/libumem.html .

-- Troy
============================================================
The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader
of this message is not the intended recipient, or an employee
or agent responsible for delivering this message to the
intended recipient, you are hereby notified that any reproduction,
dissemination or distribution of this communication is strictly
prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer. Thank you. Tellabs
============================================================

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Mononen Jussi]

Post by Gutjahr, Troy
I'm using libssh2 on Solaris to retrieve files via sftp from a
telecommunications network element that has (adapted) OpenSSH code
embedded in it. As far as I know, I can't use scp with this device.
Also, my requirements explictly force me to use sftp.

Also, SCP supports only get/put functionality, where as SFTP provides
much richer interface with moving, deletion, listing of files and so on.

Post by Gutjahr, Troy
Faster sftp would be a good thing but so far my tests indicate that
libssh2 is fast enough for my purposes, although it eats lots of CPU
cycles when I'm getting several large files simultaneously
in separate threads.

I have come across cases where our customers need to transfer thousands
of gigabytes of data during one day across their networks. In these
cases we've had to switch to command line SCP and SFTP in order to meet
their performance requirements.

/jUSSi
--
perl -e 'print for reverse @{[@{[@{[@{
[@{[Hacker,Perl,Another,Just]}]}]}]}]};'

____________________________________________________________________________________
Comptel Finland is moving to new premises.
Starting from July 9 our new visiting address is Salmisaarenaukio 1, 00180 Helsinki,
and the postal address is P.O.Box 1000, FI-00181 Helsinki
____________________________________________________________________________________

Disclaimer: This message and any attachments thereto are intended solely for
the addressed recipient(s) and may contain confidential information. If you
are not the intended recipient, please notify the sender by reply e-mail and
delete the e-mail (including any attachments thereto) without producing,
distributing or retaining any copies thereof. Any review, dissemination or
other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient(s) is prohibited.
Thank you.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Daniel Stenberg]

Post by Gutjahr, Troy
I'm using libssh2 on Solaris to retrieve files via sftp from a
telecommunications network element that has (adapted) OpenSSH code embedded
in it. As far as I know, I can't use scp with this device. Also, my
requirements explictly force me to use sftp.
Faster sftp would be a good thing but so far my tests indicate that libssh2
is fast enough for my purposes, although it eats lots of CPU cycles when I'm
getting several large files simultaneously in separate threads.

I just wanted to make this clear:

I'm fully agreeing that better SFTP performance is wanted, but since SCP is a
much simpler protocol and is suspected to also suffer from performance
problems it seems suitable to me to start with the more basic stuff and get
that up to speed first, before diving into SFTP. Without a well-performing
SCP, we won't have a well-performing SFTP.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Daniel Stenberg]

We did some tests with libssh2-0.12 (with our own patch to change the 16
byte read loop into 32kb read loop) and we came to a conclusion that libssh2
is roughly 50% slower in file transfer than the command line counterparts.
Eg. we had one test run where we transferred a 100MB file, command line scp
took 58 seconds and libssh2 a bit over 2 minutes.

I think we should:

1 - do updated SCP measurements on current libssh2 CVS code and compare to
recent openssh tools.

2 - work on making SCP perform roughly at the same speed as openssh - or more.
We should do profiling etc to see where time is spent and where we should
focus our efforts.

3 - measure SFTP performance (when large buffers are used by the client)

4 - adjust the SFTP API slightly and adapt code to improve SFTP performance

5 - smile

Now, all we need is lots of time...

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Mononen Jussi]

Post by Daniel Stenberg
1 - do updated SCP measurements on current libssh2 CVS code
and compare to recent openssh tools.
2 - work on making SCP perform roughly at the same speed as
openssh - or more. We should do profiling etc to see where
time is spent and where we should focus our efforts.
3 - measure SFTP performance (when large buffers are used by
the client)
4 - adjust the SFTP API slightly and adapt code to improve
SFTP performance

Just a thought:

If the libssh2 sftp API is non-blocking (including packet_require) the
application could use the non-blocking mode to send multiple requests.
It is easy to calculate the required request amount to transfer a file
of certain size and when the application knows how many packets it needs
to get the whole file transferred, it could easily send all the requests
(or a moderate amount, maybe less than 20). As the functions would not
block, the receiving can be done after the desired amount of requests
are sent.

Is the current non-blocking functionality suitable for this approach? If
yes, one could easily implement get/put operations as "atomic" from the
user's point of view and still provide access to single read/write
operations? (I have a Perl interface using libssh2 imitating Perl's
Net::FTP module, this is why I am eager to get performant versions of
get/put ;-)

br,

/jUSSi
--
TIMTOWTDI

____________________________________________________________________________________
Comptel Finland is moving to new premises.
Starting from July 9 our new visiting address is Salmisaarenaukio 1, 00180 Helsinki,
and the postal address is P.O.Box 1000, FI-00181 Helsinki
____________________________________________________________________________________

Disclaimer: This message and any attachments thereto are intended solely for
the addressed recipient(s) and may contain confidential information. If you
are not the intended recipient, please notify the sender by reply e-mail and
delete the e-mail (including any attachments thereto) without producing,
distributing or retaining any copies thereof. Any review, dissemination or
other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient(s) is prohibited.
Thank you.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Mononen Jussi]

* src/packet.c 1.39 - reading bigger blocks from
network. Significantly improves performance due to lesser syscalls
(and probably it influences the TCP window changes too). This is a
temporary measure while libssh2 is mostly synchronous.
I made diff of vers 1.38 and 1.39 of packet.c to see what
specific changes are made for the above performance
workaround and made the same changes to packet.c of
libssh2-0.14 release.
But still there was no performance boost up as such while
reading the file from remote SFTP server.

I have also changed the LIBSSH2_CHANNEL_WINDOW_DEFAULT to 128kb.

How much data you request with one call to libssh2_sftp_read -function?
You should read at least 32kb, ie. the max packet size.

/jUSSi
--
perl -e '$@=[[0,0112,0,0],[0,0101,0,0],
[0,0120,0,0],[0,0110,0,0]];$[=256/8-0x1D;
print(chr(@{$_}[+($[+1-1+1)]))for(@{$@});'

____________________________________________________________________________________
Comptel Finland is moving to new premises.
Starting from July 9 our new visiting address is Salmisaarenaukio 1, 00180 Helsinki,
and the postal address is P.O.Box 1000, FI-00181 Helsinki
____________________________________________________________________________________

Disclaimer: This message and any attachments thereto are intended solely for
the addressed recipient(s) and may contain confidential information. If you
are not the intended recipient, please notify the sender by reply e-mail and
delete the e-mail (including any attachments thereto) without producing,
distributing or retaining any copies thereof. Any review, dissemination or
other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient(s) is prohibited.
Thank you.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[James Housley]

Some quick test with the sftp and sftp_nonblocking programs in
example/simple shows that the amount of data requested in the call to
libssh2_sftp_read() is much more important the the SSH packet size.
All test were between two computers on the same network connected by
100MBs switch and transferring a 616M zip compressed file

sftp/sftp_nonblock read size: 1K
================================
sftp sftp_nonblock
+---------+--------------
libssh2 max | 1:06.98 | 1:05.01
packet: 4K | 1:07.37 | 1:05.14
| 1:05.14 | 1:05.00
-------------+---------+--------------
libssh2 max | 1:05.71 | 1:11.32
packet: 32K | 1:05.19 | 1:04.83
| 1:05.30 | 1:05.22

sftp/sftp_nonblock read size: 16K
=================================
sftp sftp_nonblock
+---------+--------------
libssh2 max | 0:06.03 | 0:06.03
packet: 4K | 0:06.11 | 0:06.03
| 0:06.56 | 0:05.96
-------------+---------+--------------
libssh2 max | 0:06.14 | 0:06.01
packet: 32K | 0:06.04 | 0:05.93
| 0:06.11 | 0:05.92

Although these tests were simple, it shows that when reading the more
data you ask the library for in each call the better the performance.

Jim

--

/"\ ASCII Ribbon Campaign .
\ / - NO HTML/RTF in e-mail .
X - NO Word docs in e-mail .
/ \ -----------------------------------------------------------------
http://www.FreeBSD.org The Power to Serve
jim-moqyQx4kVrsuQSGLnC+***@public.gmane.org http://www.TheHousleys.net
---------------------------------------------------------------------
In theory there is no difference between theory and practice.
In practice there is no similarity.
-- From the "I wish I'd said that" archives.



-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Daniel Stenberg]

Post by James Housley
Some quick test with the sftp and sftp_nonblocking programs in
example/simple shows that the amount of data requested in the call to
libssh2_sftp_read() is much more important the the SSH packet size.

What about a comparison with openssh's sftp next to our sftp_nonblocking?

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[James Housley]

Post by Daniel Stenberg

Post by James Housley
Some quick test with the sftp and sftp_nonblocking programs in
example/simple shows that the amount of data requested in the call
to libssh2_sftp_read() is much more important the the SSH packet
size.

What about a comparison with openssh's sftp next to our
sftp_nonblocking?

OpenSSH sftp: 0:01.06, 0:01.04, 0:01.01

libssh2 sftp_nonblock (libssh2 packet size:4K, sftp_nonblock read 64K
"static"):
0:01.60, 0:01.60, 0:01.58

That looks close enough to be called the same.

Jim
--

/"\ ASCII Ribbon Campaign .
\ / - NO HTML/RTF in e-mail .
X - NO Word docs in e-mail .
/ \ -----------------------------------------------------------------
http://www.FreeBSD.org The Power to Serve
jim-moqyQx4kVrsuQSGLnC+***@public.gmane.org http://www.TheHousleys.net
---------------------------------------------------------------------
Fortune Not Found:
Abort, Retry, Ignore?




-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Mononen Jussi]

Post by James Housley

Post by Daniel Stenberg
What about a comparison with openssh's sftp next to our
sftp_nonblocking?

OpenSSH sftp: 0:01.06, 0:01.04, 0:01.01
libssh2 sftp_nonblock (libssh2 packet size:4K, sftp_nonblock
read 64K "static"): 0:01.60, 0:01.60, 0:01.58
That looks close enough to be called the same.

Hi,

61 seconds (0:01:01) is not the same as 118 seconds (0:01:58). The
latter is ~100% slower. If this ratio endures when the file sizes grow
or the netowrk speed decreases the performance can be too weak. What if
OpenSSH sftp transfers one file in 10 minutes, libssh2 would take 20
minutes. From our/my point of view this is not acceptable and we/I have
had to rely on wrapping command line tools for certain performance
critical systems.

But I guess the packet pipelining is out of the scope of 0.15 release.

br,

/jUSSi
--
perl -e '$@=[[0,0112,0,0],[0,0101,0,0],
[0,0120,0,0],[0,0110,0,0]];$[=256/8-0x1D;
print(chr(@{$_}[+($[+1-1+1)]))for(@{$@});'

____________________________________________________________________________________
Comptel Finland is moving to new premises.
Starting from July 9 our new visiting address is Salmisaarenaukio 1, 00180 Helsinki,
and the postal address is P.O.Box 1000, FI-00181 Helsinki
____________________________________________________________________________________

Disclaimer: This message and any attachments thereto are intended solely for
the addressed recipient(s) and may contain confidential information. If you
are not the intended recipient, please notify the sender by reply e-mail and
delete the e-mail (including any attachments thereto) without producing,
distributing or retaining any copies thereof. Any review, dissemination or
other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient(s) is prohibited.
Thank you.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[James Housley]

Post by Mononen Jussi

Post by James Housley

Post by Daniel Stenberg
What about a comparison with openssh's sftp next to our
sftp_nonblocking?

OpenSSH sftp: 0:01.06, 0:01.04, 0:01.01
libssh2 sftp_nonblock (libssh2 packet size:4K, sftp_nonblock
read 64K "static"): 0:01.60, 0:01.60, 0:01.58
That looks close enough to be called the same.

Hi,
61 seconds (0:01:01) is not the same as 118 seconds (0:01:58). The

No those are all 1 second plus 6/100ths for OpenSSH. While libssh2
is 1 second plus 60/100ths

Post by Mononen Jussi
latter is ~100% slower. If this ratio endures when the file sizes grow
or the netowrk speed decreases the performance can be too weak. What if
OpenSSH sftp transfers one file in 10 minutes, libssh2 would take 20
minutes. From our/my point of view this is not acceptable and we/I have

So it would take 15 minutes, which is still slower. Not that was
with a libssh2 with an internal 4k Packet and reading 64K at a time.
Still available for testing is libssh2 with 32k packets and even
reading 256k at a time.

Post by Mononen Jussi
had to rely on wrapping command line tools for certain performance
critical systems.
But I guess the packet pipelining is out of the scope of 0.15 release.

I would think so too.

Jim

--

/"\ ASCII Ribbon Campaign .
\ / - NO HTML/RTF in e-mail .
X - NO Word docs in e-mail .
/ \ -----------------------------------------------------------------
http://www.FreeBSD.org The Power to Serve
jim-moqyQx4kVrsuQSGLnC+***@public.gmane.org http://www.TheHousleys.net
---------------------------------------------------------------------
The wise man built his network upon Un*x.
The foolish man built his network upon Windows.




-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[Daniel Stenberg]

Post by Mononen Jussi
But I guess the packet pipelining is out of the scope of 0.15 release.

Yes, I would think that such pipelining support will take some thinking, some
internal restructuring and more, so we better save that until after 0.15...


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/