SSH client is not reading full data if executed command generates big output

[Subrata Dasgupta]

Hi All,I am new to libssh2.I am using the example program SmallSimpleSSH.c to simulate a SSH client terminal to execute several commands. Please find the attached code.Normally program is working fine. But when output of the remotely executed program is very big then the client is not able to read all the data. Seems server is sending the partial data and then the command prompt. But if I connect the same server using PUTTY and execute the same command then PUTTY terminal is showing "more" at the end of terminal and wait for my "return" key press.My objective is to read all the output of the command without blocking on "more" ( means "return" key press). How can I achieve this ? Please help.ThanksSubrata

[Peter Stuge]

Your attached example requests a VT100 pseudo-terminal (pty) from the
server, but does not include any terminal emulation whatsoever. That
will never work reliably in the general case.

PuTTY includes significant terminal emulation code. Linux/Unix systems
always have a terminal emulator. Windows never does.

Post by Subrata Dasgupta
But when output of the remotely executed program is very big then
the client is not able to read all the data. Seems server is sending
the partial data and then the command prompt.

In other words: Without terminal emulation, communication is unreliable.

Post by Subrata Dasgupta
But if I connect the same server using PUTTY and execute the same
command then PUTTY terminal is showing "more" at the end
of terminal and wait for my "return" key press.

In other words: With terminal emulation, communication is reliable.

Post by Subrata Dasgupta
My objective is to read all the output of the command without blocking
on "more" ( means "return" key press).
How can I achieve this ?

It depends on the capabilities of your server system, but quite
likely you will need to implement some if not full terminal emulation
and then you may have to make your program recognize the "more" prompt,
and then simulate a human pressing "return", while also filtering
that out of the data stream that you want to scrape.

If you're lucky then the server also supports a non-interactive terminal
type, where you can scrape all output without interaction. That's all
completely out of scope of the SSH protocol and thus libssh2, it is
exclusively a property of your particular server. You can try using
"raw", "ascii" and maybe some others instead of "vt100" in the example -
whatever your research on that topic finds may be worth a try.

I will recommend to search for an API offered by the server product
to do what you want instead of trying to automate around the
interactive interface.


Good luck.

//Peter
_______________________________________________
libssh2-devel https://cool.haxx.se/cgi-bin/mailman/listinfo/

[Subrata Dasgupta]

Hi Peter ,Many thanks for your explanation.My intention is to run multiple commands sequentially over ssh and get the output.It seems "libssh2_channel_exec" function can only execute a single command over a session or channel. So this option will not serve my purpose because commands may have some sub-commands. So I need to retain the session or channel to execute those sub-commands. Please let me know if my understanding is wrong and kindly let me know how to execute multiple commands and sub-commands over libssh2_channel_exec.It seems only "libssh2_channel_shell" function can serve my purpose because I can execute multiple commands on a established channel / session. Please let me know if I am wrong.It also seems from the different ssh tutorials from net that non-interactive session may serve my purpose because there is no pty associated with the channel. So I have tried to comment out the "libssh2_channel_request_pty" call , but unfortunately!
result is the same(server do not send all data). What could be the reason of such problem and how to fix it if possible ?Lastly is there anyway to get all output data of commands or sub-commands without using terminal emulation ?? Please advice if possible.Thanks a lot again,SubrataFrom: Peter Stuge <***@stuge.se>Sent: Sun, 28 Oct 2018 04:57:16To: libssh2-***@cool.haxx.seSubject: Re: SSH client is not reading full data if executed command generates big outputYour attached example requests a VT100 pseudo-terminal (pty) from theserver, but does not include any terminal emulation whatsoever. Thatwill never work reliably in the general case.PuTTY includes significant terminal emulation code. Linux/Unix systemsalways have a terminal emulator. Windows never does.Subrata Dasgupta wrote:> But when output of the remotely executed program is very big then> the client is not able to read all the data. Seems server is sending> the partial data and then the command p!
rompt.In other words: Without terminal emulation, communication is unreliable.> But if I connect the same server using PUTTY and execute the same> command then PUTTY terminal is showing "more" at the end> of terminal and wait for my "return" key press.In other words: With terminal emulation, communication is reliable.> My objective is to read all the output of the command without blocking> on "more" ( means "return" key press).> How can I achieve this ?It depends on the capabilities of your server system, but quitelikely you will need to implement some if not full terminal emulationand then you may have to make your program recognize the "more" prompt,and then simulate a human pressing "return", while also filteringthat out of the data stream that you want to scrape.If you're lucky then the server also supports a non-interactive terminaltype, where you can scrape all output without interaction. That's allcompletely out of scope!
of the SSH protocol and thus libssh2, it isexclusively a property of your particular server. You can try using"raw", "ascii" and maybe some others instead of "vt100" in the example -whatever your research on that topic finds may be worth a try.I will recommend to search for an API offered by the server productto do what you want instead of trying to automate around theinteractive interface.Good luck.//Peter_______________________________________________libssh2-devel https://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel

[Peter Stuge]

Post by Subrata Dasgupta
My intention is to run multiple commands sequentially over ssh and
get the output.It seems "libssh2_channel_exec" function
can only execute a single command over a session or channel. So
this option will not serve my purpose because commands may have
some sub-commands. So I need to retain the session or channel to
execute those sub-commands. Please let me know if my
understanding is wrong and kindly let me know how to execute
multiple commands and sub-commands over libssh2_channel_exec.

Your understanding is correct; libssh2_channel_exec() only ever
executes one command, but you can call it many times within one
session.

That allows you to take advantage of well-defined fit-for-purpose
behavior and interactions between your client and the server. This
way, your software has some chance to control the processes on the
server side.

Post by Subrata Dasgupta
It seems only "libssh2_channel_shell" function can serve
my purpose because I can execute multiple commands on a established
channel / session. Please let me know if I am wrong.

A shell channel is not needed to execute multiple commands in one
session, only to execute multiple commands in one *channel*. Study
the difference to see what is actually required in your case.

Post by Subrata Dasgupta
It also seems from the different ssh tutorials from net that
non-interactive session may serve my purpose because there is no
pty associated with the channel. So I have tried to comment out
the "libssh2_channel_request_pty" call , but unfortunately!
result is the same(server do not send all data). What could be
the reason of such problem and how to fix it if possible ?

One reason could be that the software you are executing on the server
simply is not written to support both interactive and programmed use.
You can't fix that in the client. There can be other reasons, you'll
have to study the particular server software you want to support in
detail.

Post by Subrata Dasgupta
Lastly is there anyway to get all output data of commands or
sub-commands without using terminal emulation ??

Please clarify what you mean by "sub-commands" ?


In general, if the software you want to execute on the server does
not explicitly support programmed use (and this is likely the case)
then your only option is to write a software that simulates interactive
use, which neccessarily requires terminal emulation to handle everything
that the server software outputs, as well as everything that your
simulator requires to output.

Shells are human interfaces, not programming interfaces, making them a
poor choice for automation. Sometimes there is may be no other way,
but it always requires a lot of (I think wasted) effort. Try to solve
the problem another way if possible.


//Peter
_______________________________________________
libssh2-devel https://cool.

[Subrata Dasgupta]

Hi Peter ,Many thanks for your explanation.My intention is to run multiple commands sequentially over ssh and get the output.It seems "libssh2_channel_exec" function can only execute a single command over a session or channel. So this option will not serve my purpose because commands may have some sub-commands. So I need to retain the session or channel to execute those sub-commands. Please let me know if my understanding is wrong and kindly let me know how to execute multiple commands and sub-commands over libssh2_channel_exec.It seems only "libssh2_channel_shell" function can serve my purpose because I can execute multiple commands on a established channel / session. Please let me know if I am wrong.It also seems from the different ssh tutorials from net that non-interactive session may serve my purpose because there is no pty associated with the channel. So I have tried to comment out the "libssh2_channel_request_pty" call , but unfortunately!
result is the same(server do not send all data). What could be the reason of such problem and how to fix it if possible ?Lastly is there anyway to get all output data of commands or sub-commands without using terminal emulation ?? Please advice if possible.Thanks a lot again,SubrataFrom: Peter Stuge <***@stuge.se>Sent: Sun, 28 Oct 2018 04:57:16To: libssh2-***@cool.haxx.seSubject: Re: SSH client is not reading full data if executed command generates big outputYour attached example requests a VT100 pseudo-terminal (pty) from theserver, but does not include any terminal emulation whatsoever. Thatwill never work reliably in the general case.PuTTY includes significant terminal emulation code. Linux/Unix systemsalways have a terminal emulator. Windows never does.Subrata Dasgupta wrote:> But when output of the remotely executed program is very big then> the client is not able to read all the data. Seems server is sending> the partial data and then the command p!
rompt.In other words: Without terminal emulation, communication is unreliable.> But if I connect the same server using PUTTY and execute the same> command then PUTTY terminal is showing "more" at the end> of terminal and wait for my "return" key press.In other words: With terminal emulation, communication is reliable.> My objective is to read all the output of the command without blocking> on "more" ( means "return" key press).> How can I achieve this ?It depends on the capabilities of your server system, but quitelikely you will need to implement some if not full terminal emulationand then you may have to make your program recognize the "more" prompt,and then simulate a human pressing "return", while also filteringthat out of the data stream that you want to scrape.If you're lucky then the server also supports a non-interactive terminaltype, where you can scrape all output without interaction. That's allcompletely out of scope!
of the SSH protocol and thus libssh2, it isexclusively a property of your particular server. You can try using"raw", "ascii" and maybe some others instead of "vt100" in the example -whatever your research on that topic finds may be worth a try.I will recommend to search for an API offered by the server productto do what you want instead of trying to automate around theinteractive interface.Good luck.//Peter_______________________________________________libssh2-devel https://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel

[Subrata Dasgupta]

Hi Peter,Thanks a lot for beautiful explanation.My intention is to run several commands on a small router like device. The device provide a restricted environment and allows user to run only specific set of commands. Some commands may also have sub-commands, for an example until and unless 'enable' command is executed user are not allowed to execute few commands. One more example of sub-command is , until and unless "configure terminal" command is executed users are not allowed to run any configuration related commands. So I need to retain the state otherwise I can not execute some commands.I do not fully understand the difference between session and channel. If possible please explain a bit.Is it possible to use "libssh2_channel_exec" in the scenario explained above (means in sub-command case) ? If yes please let me know how.Does "libssh2_channel_exec" is able to provide all the data if command output is very big ?ThanksSubrataFrom: Pet!
er Stuge <***@stuge.se>Sent: Mon, 29 Oct 2018 02:24:07To: libssh2-***@cool.haxx.seSubject: Re: SSH client is not reading full data if executed command generates big outputSubrata Dasgupta wrote:> My intention is to run multiple commands sequentially over ssh and> get the output.It seems "libssh2_channel_exec" function> can only execute a single command over a session or channel.  So> this option will not serve my purpose because commands may have> some sub-commands.  So I need to retain the session or channel to> execute those sub-commands.  Please let me know if my> understanding is wrong and kindly let me know how to execute> multiple commands and sub-commands over libssh2_channel_exec.Your understanding is correct; libssh2_channel_exec() only everexecutes one command, but you can call it many times within onesession.That allows you to take advantage of well-defined fit-for-purposebehavior and interactions!
between your client and the server. Thisway, your software has some chance to control the processes on theserver side.> It seems only "libssh2_channel_shell" function can serve> my purpose because I can execute multiple commands on a established> channel / session.  Please let me know if I am wrong.A shell channel is not needed to execute multiple commands in onesession, only to execute multiple commands in one *channel*. Studythe difference to see what is actually required in your case.> It also seems from the different ssh tutorials from net that> non-interactive session may serve my purpose because there is no> pty associated with the channel.  So I have tried to comment out> the "libssh2_channel_request_pty" call , but unfortunately!> result is the same(server do not send all data).  What could be> the reason of such problem and how to fix it if possible ?One reason could be that the software you are executing on the serversimply is not written to support b!
oth interactive and programmed use.You can't fix that in the client. There can be other reasons, you'llhave to study the particular server software you want to support indetail.> Lastly is there anyway to get all output data of commands or> sub-commands without using terminal emulation ??Please clarify what you mean by "sub-commands" ?In general, if the software you want to execute on the server doesnot explicitly support programmed use (and this is likely the case)then your only option is to write a software that simulates interactiveuse, which neccessarily requires terminal emulation to handle everythingthat the server software outputs, as well as everything that yoursimulator requires to output.Shells are human interfaces, not programming interfaces, making them apoor choice for automation. Sometimes there is may be no other way,but it always requires a lot of (I think wasted) effort. Try to solvethe problem another way if possible.//Peter________!
_______________________________________libssh2-devel https://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel

[Peter Stuge]

Post by Subrata Dasgupta
My intention is to run several commands on a small router like device.
The device provide a restricted environment and allows user to run only
specific set of commands. Some commands may also have sub-commands, for
an example until and unless 'enable' command is executed user
are not allowed to execute few commands. One more example of sub-command
is, until and unless "configure terminal" command is executed
users are not allowed to run any configuration related commands.

Oh now I understand. Thanks for the explanation.

Post by Subrata Dasgupta
So I need to retain the state otherwise I can not execute some commands.
I do not fully understand the difference between session and channel.
If possible please explain a bit.

They are two SSH protocol level concepts, I recommend reading RFC4254
chapter 5 and 6 where channels are explained:

https://tools.ietf.org/rfc/rfc4254.txt

Please note that SSH sessions are different from "terminal sessions"
and "Interactive Sessions" also explained in RFC4254.

An SSH session can be considered to represent the TCP connection that
has been authenticated by the SSH server, usually by one or more of
password, public key and keyboard-interactive credentials. Within one
such SSH session there can be multiple SSH channels, if server policy
permits. You can e.g. take advantage of this in the OpenSSH ssh client
using the "ControlMaster" and "ControlPath" directives. It's a nice
feature of the SSH protocol.

Post by Subrata Dasgupta
Is it possible to use "libssh2_channel_exec" in the scenario
explained above (means in sub-command case) ?

No, that will not work with the type of shell you describe. Your target
device should in fact probably not even allow you to open a channel,
since that application-specific shell is rich in context.

Post by Subrata Dasgupta
Does "libssh2_channel_exec" is able to provide all the data
if command output is very big ?

Large data transfer does work well with all channel types in libssh2,
the function at the center of data transfer is libssh2_channel_read(),
which reads data from open channels. But it does not behave exactly like
the read() system call, and the differences can be tricky to deal with.

Differences have to do with how the SSH protocol can also send other
messages than what you are waiting for (like data for another channel,
or a message unrelated to any channel such as re-keying the SSH session)
and while libssh2 handles them in the background, things like timeout
usually have to be implemented "by hand" in the application - the kernel
APIs such as select() and poll() can't be used, because one TCP connection
can carry many SSH message types, and the kernel doesn't know which bytes
belong to a particular channel. SSH is a very useful protocol, but this
part can make it a little inconvenient in applications.

Try using some of the examples that are included with libssh2, first
unchanged, then move on to change them.

In the end, for an interactive shell like the one you describe, I
think you will have to do a lot of work to create a reliable tool,
including some kind of terminal emulation (but maybe not very much)
to satisfy the expectations of the shell running on the device.


//Peter
_______________________________________________
libssh2-devel https://co

[Subrata Dasgupta]

Hi Peter,Thanks a lot for beautiful explanation.My intention is to run several commands on a small router like device. The device provide a restricted environment and allows user to run only specific set of commands. Some commands may also have sub-commands, for an example until and unless 'enable' command is executed user are not allowed to execute few commands. One more example of sub-command is , until and unless "configure terminal" command is executed users are not allowed to run any configuration related commands. So I need to retain the state otherwise I can not execute some commands.I do not fully understand the difference between session and channel. If possible please explain a bit.Is it possible to use "libssh2_channel_exec" in the scenario explained above (means in sub-command case) ? If yes please let me know how.Does "libssh2_channel_exec" is able to provide all the data if command output is very big ?ThanksSubrataFrom: Pet!
er Stuge <***@stuge.se>Sent: Mon, 29 Oct 2018 02:24:07To: libssh2-***@cool.haxx.seSubject: Re: SSH client is not reading full data if executed command generates big outputSubrata Dasgupta wrote:> My intention is to run multiple commands sequentially over ssh and> get the output.It seems "libssh2_channel_exec" function> can only execute a single command over a session or channel.  So> this option will not serve my purpose because commands may have> some sub-commands.  So I need to retain the session or channel to> execute those sub-commands.  Please let me know if my> understanding is wrong and kindly let me know how to execute> multiple commands and sub-commands over libssh2_channel_exec.Your understanding is correct; libssh2_channel_exec() only everexecutes one command, but you can call it many times within onesession.That allows you to take advantage of well-defined fit-for-purposebehavior and interactions!
between your client and the server. Thisway, your software has some chance to control the processes on theserver side.> It seems only "libssh2_channel_shell" function can serve> my purpose because I can execute multiple commands on a established> channel / session.  Please let me know if I am wrong.A shell channel is not needed to execute multiple commands in onesession, only to execute multiple commands in one *channel*. Studythe difference to see what is actually required in your case.> It also seems from the different ssh tutorials from net that> non-interactive session may serve my purpose because there is no> pty associated with the channel.  So I have tried to comment out> the "libssh2_channel_request_pty" call , but unfortunately!> result is the same(server do not send all data).  What could be> the reason of such problem and how to fix it if possible ?One reason could be that the software you are executing on the serversimply is not written to support b!
oth interactive and programmed use.You can't fix that in the client. There can be other reasons, you'llhave to study the particular server software you want to support indetail.> Lastly is there anyway to get all output data of commands or> sub-commands without using terminal emulation ??Please clarify what you mean by "sub-commands" ?In general, if the software you want to execute on the server doesnot explicitly support programmed use (and this is likely the case)then your only option is to write a software that simulates interactiveuse, which neccessarily requires terminal emulation to handle everythingthat the server software outputs, as well as everything that yoursimulator requires to output.Shells are human interfaces, not programming interfaces, making them apoor choice for automation. Sometimes there is may be no other way,but it always requires a lot of (I think wasted) effort. Try to solvethe problem another way if possible.//Peter________!
_______________________________________libssh2-devel https://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel