Python's Hand in Redirecting Command Output Over the Network

Introduction

If you say out loud at a party that you created an interesting application using Python in Linux and the people at the party want to know more, you are definitely with the right croud. The application that you are going to see implemented in Python involves network programming (UDP) and GUI development. What else can be more interesting than a Python application, right? A testament to its power, scripting in Python feels like writing code in a programming language. You can make many of the system calls that you can when using a programming language such as C++. Most of the time, the effect of making such calls is similar regardless of the OS you are on.

If I start blabbering on how to do something using Linux commands, a number of you will be able to show me better commands to perform the same tasks. Linux provides with a plethora of commands for everyday use. On top of that, you can pipe and redirect output of one command and pass it to another command in a sub-process. Imagine what would be the number of all the combinations of commands in Linux. In general, I go by the motto, “Leave to Linux what Linux can do best,” rather than jumping to writing code in Java or C++. But, I recognize that Linux is not a panacea, and my next step in the plan of attack to a hard problem is to see if the problem can be tackled with a script. Mine and many other programmers' choice of a scripting language is Python. I am attracted to Python for its readability as a consequence of the strict indentation rules.

Here we go now to the crux of the matter. What I would like to do is run a command in Linux and have the output be displayed in another process potentially running on a different computer. I am envisioning being able to use a command dubbed sotp (send output to process) like so:

echo "Bello mondo" | sotp

When this happens, if the program ReceDisplay is running, it should receive and display Bello mondo because that is the output of echo "Bello mondo".

Help me out here, will you? We know that one of the ways two processes, being on the same or different computers, can communicate is through a network protocol. I am sure you would agree that UDP, the connection-less protocol, suffices our needs so, we are not going to bother about TCP. For this application, we are also not going to be concerned with IPV6; our network addresses will just be IPV4. During development, we are going to be using the IPV4 address 127.0.0.1—which is the loop-back address or home. Data packets sent to this address will just loop around and come back to the originating machine. At this time, for convenience sake, we will be running both the source and destination applications on the same machine. The address can easily be modified to an appropriate number just before deployment.

The computer interacts with a network for transmitting and receiving data packets using its NIC (Network Interface Card). Each NIC is tagged with a supposedly unique MAC (Media Access Control) number by its manufacturer. Theoretically, two computer on a network should be able to successfully communicate by exchanging data packets if the network has a knowledge of their MAC numbers. But, instead, we assign IP addresses to the NICs so as to have routing information for exchanges outside the network. Another important number in network programming is the port number. So, when the data packets reach the destination IP address, the resident OS should then take them on the last leg of their trip to the application that can use them. The NIC or IP address of a machine is shared amongst the many applications running on the machine. These applications each take a slot of the NIC identified by the port number. In our applications, we will be using 51001 as our port number. If more port numbers are needed, we will be moving up one number to 51002, 51003, 51004, and so on.

Sender and Receiver

With these considerations in mind, let us now create the application that is going to receive and display the output data.

#!/usr/bin/env python3
#filename ReceDisplay.py

import socket

IP_ADDR = ""
PORT    = 51001

sock = socket.socket(socket.AF_INET,    # IPv4
                     socket.SOCK_DGRAM) # UDP
sock.bind((IP_ADDR, PORT))

print "Server Started: ", PORT

while True:
    (data_line, addr) = sock.recvfrom(1024) # buffer size is 1024 bytes
    print(data_line.decode())

As you can see, a socket with PORT 51001 and IP_ADDR “any” is being established. This socket will sniff at data on all the IP addresses of the machine and receive 1024 bytes at a time. This is enough number of characters for one line. We will make sure to send data one line at a time.

We cannot test a receiver without a sender and, for that, we have the next program.

#!/usr/bin/env python3
#filename sotp.py

import socket
import sys

IP_ADDR = "127.0.0.1"
PORT    = 51001
MESSAGE = "Transfer successful, but no message passed."

num_args = len(sys.argv)

# Check if stdin might be filled
if not sys.stdin.isatty():
    MESSAGE = sys.stdin.read()
    # Remove the newline character
    MESSAGE = MESSAGE[:-1]

print("Target port:", PORT)
print("Message:", MESSAGE)

sock = socket.socket(socket.AF_INET,    # IPv4
                     socket.SOCK_DGRAM) # UDP
sock.sendto(MESSAGE.encode(), (IP_ADDR, PORT))

This program will send MESSAGE to the application with port number 51001 on IP address 127.0.0.1 which we have said is the home address. We can simply modify this number to the desired target if we would like to send it to a different one. We can also do clever things like broadcasting MESSAGE to multiple IP addresses.

Notice that, in sotp.py, MESSAGE is being read from stdin, the standard input stream. Well, when piping commands, essentially, what goes on is that the output of the command in the current process is transferred to the command after the pipe symbol in a subprocess as input. For example, if you consider command1 and command2 to be valid Linux commands, piping command1 and command2 like this:

command1 | command2

redirects the output of command1 from stdout of the current process to stdin of the subprocess in which command2 runs. Now, replace command2 with python sotp.py and, it will be evident why MESSAGE is being read from stdin.

At this point, we have everything needed to run the application. First, start the program ReceDisplay.py and leave it open. Next, run the commands:

echo "Bello mondo" | python3 sotp.py

You will observe the output of B<span style="text-decoration: none;">ello mondo</span> being displayed on ReceDisplay.py.

Convenience of GUI

Let us see if we can further improve on our application. With the current set up, ReceDisplay can only handle the output of one command. But, if we turn ReceDisplay into a notebook with multiple pages, we can print the output of a couple more commands on each page. Python GUI programs end with .pyw. ReceDisplay.pyw invokes tkinter.

#!/usr/bin/env python3
#filename ReceDisplay.pyw

from tkinter import *
from tkinter.ttk import *

root = Tk()

nb = Notebook(root, height=700, width=600)
nb.pack(fill='both', expand='yes')

page1 = Frame()
page2 = Frame()
page3 = Frame()
page4 = Frame()
page5 = Frame()
page6 = Frame()

nb.add(page1, text='Page 1')
nb.add(page2, text='Page 2')
nb.add(page3, text='Page 3')
nb.add(page4, text='Page 4')
nb.add(page5, text='Page 5')
nb.add(page6, text='Page 6')

t1 = Text(page1)
t2 = Text(page2)
t3 = Text(page3)
t4 = Text(page4)
t5 = Text(page5)
t6 = Text(page6)

t1.insert(END, "Server 1: work in progress")
t1.see(END)
t2.insert(END, "Server 2: work in progress")
t2.see(END)
t3.insert(END, "Server 3: work in progress")
t3.see(END)
t4.insert(END, "Server 4: work in progress")
t4.see(END)
t5.insert(END, "Server 5: work in progress")
t5.see(END)
t5.insert(END, "Server 6: work in progress")
t5.see(END)

t1.pack(fill='both', expand='yes')
t2.pack(fill='both', expand='yes')
t3.pack(fill='both', expand='yes')
t4.pack(fill='both', expand='yes')
t5.pack(fill='both', expand='yes')
t6.pack(fill='both', expand='yes')

root.mainloop()

This is the skeleton of the application. It only generates the GUI; there is no meat to it, so as to speak. To flesh it out, the first thing that you need to note is that the code is currently single-threaded. We would like to run six servers (one on each page) to receive MESSAGEs from six commands. Not to interfere with the main thread that is drawing our GUI, we want to run our little servers on six separate threads. But, notice how the code is growing in blocks of six. Pretty soon, we will have a bloated code of blocks of six that do the same thing (just repeated six times). Well, Python's other strength is that it has facilities for object-oriented programming. If only we implement our design as a class, we can reuse code by creating objects of the class. And that is exactly what we will be doing.

#!/usr/bin/env python3
#filename ReceDisplay.pyw

from tkinter import *
from tkinter.ttk import *
from queue import Queue
import socket
import threading

IP_ADDR = ""
PORTS = [51001, 51002, 51003, 51004, 51005, 51006]

class LittleServer(): # Many instances of this class can be created
   def __init__(self, PORTX): # PORT is passed during object initialization
      self.sockx = socket.socket(socket.AF_INET,    # IPv4
                                 socket.SOCK_DGRAM) # UDP
      self.sockx.bind((IP_ADDR, PORTX))

      self.bthreadx = threading.Thread(target=self.run_server, args=())
      self.bthreadx.daemon = True  # Daemonize thread
      self.bthreadx.start() # Thread starter with run_server()
 
      self.eventx = threading.Event()

      self.pagex = Frame() # Pages to be added to notebook
      self.textx = Text(self.pagex)
      self.textx.pack(fill='both', expand='yes')

      self.textx.insert(END, "Server started - " + str(PORTX) + "\n")
      self.textx.see(END)
      self.textx.update_idletasks()

      self.queuex = Queue()

   def run_server(self):
      buf_size = 1024 # buffer size is 1024 bytes
      while True:
         (data, addr) = self.sockx.recvfrom(buf_size)
         self.queuex.put(data.decode()) # Put messages in queue for display_message()
         self.eventx.set() # Set event to signal run_disp_msgs() on another thread

   def display_message(self):
      self.thx = threading.Thread(target=self.run_disp_msgs, args=())
      self.thx.daemon = True  # Daemonize thread
      self.thx.start() # Thread started with run_disp_msgs()

   def run_disp_msgs(self):
      while True:
         self.eventx.wait()
         while not self.queuex.empty():
            s = self.queuex.get()
            self.textx.insert(END, s + "\n")
            self.textx.see(END)
            self.textx.update_idletasks()
         self.eventx.clear()

root = Tk()

nb = Notebook(root, height=700, width=600)
nb.pack(fill='both', expand='yes')

lservs = [] # this list will contain LittleServers

for PORT in PORTS:
   lserv = LittleServer(PORT)
   nb.add(lserv.pagex, text='Page ' + str(PORT))
         # add each server's page to notebook
   lservs.append(lserv)

for lserv in lservs:
   lserv.display_message() # display server content

root.mainloop()

It is cool that each page has a socket associated with it; six commands are now able to display their output by sending their MESSAGEs to the six PORTs. But, to accomplish this, port number must be passed as argument to the modified sotp.py program. Before going any further though, let us test our application by using the ping command.

ping bing.com | python3 sotp.py

Notice the one line output on Page 51001. Not quite what we expected. Normally, the output of a ping command is non-ending lines upon lines of text that we usually break with a Ctrl+C.

Dispair Not

Obviously, we can no longer trust that piping will work. We are not going to give up that easily, though. We are going to change strategy. The new strategy, as can be seen in the modified sotp.py, employs stdout instead of stdin. The other thing that stands out is that we will be passing the commands to be executed as argument (that is besides PORT), and the passed command runs as a subprocess (this may be the one thing that remained constant).

#!/usr/bin/env python3
#filename sotp.py

import socket
import subprocess
import sys

IP_ADDR = "127.0.0.1"
PORT    = int(sys.argv[1]) # First argument is the PORT
MESSAGE = "Transfer successful, but no message passed."

sock = socket.socket(socket.AF_INET,    # IPv4
                     socket.SOCK_DGRAM) # UDP

print("Target port:", PORT)

cmd = []
if len(sys.argv) > 2:
   cmd = sys.argv[2:]
   proc = subprocess.Popen(cmd, 
                           stdout=subprocess.PIPE,
                           stderr=subprocess.STDOUT, # stderr combined with stdout
                           universal_newlines=True)
while True:
   if cmd != []:
      MESSAGE = proc.stdout.readline()
      if MESSAGE == '' and proc.poll() != None:
         break
   if MESSAGE:
      # Remove the newline character
      MESSAGE = MESSAGE[:-1]
      sock.sendto(MESSAGE.encode(), (IP_ADDR, PORT))
      if cmd == []:
         break

War Ended

Now, finally, if we create the alias sotp:

alias sotp=python3 sotp.py

and issue our command like:

sotp 51004 ping bing.com

there is nothing left to do but reap the fruits of our labor.