Picture

Hi, I'm Boopathi Rajaa.

Hacker, Dancer, Biker, Adventurist

Threaded SSH

Kindly note, this doesn't mean implementation of threading in SSH.

Problem statement

When we have multiple servers to manage, we configure it in such a way that they are easily manageable. To meet this requirement, as a first thought, we make similarities in few of the configurations, packages and repositories used in/by those servers. This is mostly applicable in cases where/whenever High Availability, Sharing of resources and Horizontal Scaling comes into picture. Once everything is set, we might run into a situation, where we would want to run a script on all those servers and get the output and error  to one place for reviewing. Now, one way to do this is, while setting up those servers, we could use a tool, called MCollective, that pertains for these kinds of operations. The quick and dirty way is to ssh into every server, write the script, create a cron job that invokes this script, and throw the output to some static page that can be fetched over HTTP. Now, being a system administrator, we would NOT want to do that. We would end up writing a tiny script in bash, which sends the command over ssh, executes it, and standard output is redirected to the system(callee). We have to find a better way (and a Simple way) to do this.

Sequential SSH

As discussed above, whenever we would like to execute something over SSH, we write something like this.

    for i in {1..50}; do
        ssh boopathi@server-$1@myhost.com "top -b -n1 | head -n5";
    done;

The code, sequentially, after getting authorized, executes the command within quotes (the top 5 processes in that system), and echoes the output to the callee. This takes lots of time where the callee waits until the previous instance has exited.

Threaded SSH (Solution)

Threading seems to solve the issue of waiting for every instance to be completed. So, one of the possible SIMPLE solutions, would be to invoke the ssh function in separate threads. As, each of the thread spawned is further going to invoke a sub-process that runs SSH, and also, as there is nothing to be shared between these instances, we can safely write a SIMPLE module that would facilitate this feature. So, now, all connections are instantiated in parallel or whenever a new thread can be spawned. The output is thrown to STDOUT of the callee whenever a thread exits. And it obviously becomes much faster on a multi-core system.

Code (Python)

You could as well follow this gist, where the revisions are maintained.

https://gist.github.com/4046754

Code:

    #!/usr/bin/python

    # This doesn't mean threading in ssh. What it means is, every thread
    # that this program spawns, creates a new subprocess to perform the
    # system "ssh" command.

    # Sample Usages
    # =============
    #
    # 1.
    # for i in server1 server2 server3 server4;
    # do echo username@$i;
    # done | ./tssh "echo 'Good night :P'; rm -rf /; "
    #
    # 2.
    # cat connection_list | ./tssh "ifconfig lo"
    #
    # 3.
    # cat server_list | while read server;
    # do echo "username@$server";
    # done | ./tssh "top -b -n1 | head -n10" --noheader

    import sys,os,threading,shlex,subprocess


    basecommand = "ssh -o StrictHostKeyChecking=no"
    header = "echo \"%s says,\"; echo '=========================================='; "

    nohead=len(sys.argv)>2 and sys.argv[2]=='--noheader'

    class Runner(threading.Thread):
        def __init__ (self, server, command):
            super(Runner,self). __init__ ()
            self.command = shlex.split(basecommand)
            self.command += [server]
            self.command += [(header % server, "")[nohead] + str(command) +";"]
        def run(self):
            sout = subprocess.Popen(self.command, stdout=subprocess.PIPE).stdout
            for i in sout:
                print i.strip()

    if __name__ == ' __main__':
        while True:
            try:
                server=raw_input()
            except EOFError:
                break
            t=Runner(server=server, command=len(sys.argv)>1 and sys.argv[1] or "echo ")
            t.start()

 Internals

The code reads from STDIN (Standard Input) - the list of servers, till EOF (End of File). Once this collection is made, it spawns a thread(for every server), which constructs a sub-process that runs the SSH command. The --noheader  flag is just to prevent pretty print option. Use it if you've to parse the output or use it on another process.

Installation

No big deal. Save the source. Give execute permissions. Those who are struggling to install, use this script to download and install.

    curl https://raw.github.com/gist/4046754/af7b26f723c9da37a9e34102be93285ed92be756/tssh.py > /usr/bin/tssh && chmod +x /usr/bin/tssh