FTP-based Replication with Sharp FTP Server

• Download source files - 21 Kb
• Download compiled project - 16.9 Kb

Introduction

This article demonstrates how to create a file replication process that uses FTP as the transport mechanism. The development of the Sharp FTP Server and its unique replication functionality was mainly a journey of discovery for me to learn more about C#. In the end, I learned a little about Windows services, parsing XML, multi-threading, and garbage collection.

Background

The Sharp FTP Server is a C# Windows Service that handles all the basic things that a FTP server should. The code that handles the server part of the service is based on an FTP server implementation called Light FTP Server. The client side of the code [used to push files to other servers] is based on the FtpClient code written by Jaimon Matthew.

The main parts that I coded revolved around creating a service and implementing a FTP-based replication. The replication works by taking an incoming PUT [STOR in the FTP spec] and redirecting/duplicating it to other FTP servers. The server does this by seeing what directory the incoming client connection is going to and looking for possible alternate destinations in a "router" configuration XML file. If it doesn't find any entries, then it stores the file where the client told the server to put it. Otherwise, it opens FTP connections to the alternate destinations and writes the data coming in on the upload stream to the other outgoing upload streams.

Using the code

The following code snippets illustrate the bulk of my replication logic. The first snippet shows the UploadFile method located in the FtpServerControl class. Its main purpose used to be to just write an incoming file to disk. But I added a step for the method to check a routing configuration file (defined in the main config.xml file) for the incoming directory and username, to see if the file really needs to go elsewhere. If so, the RouteTransfer method is called.

private bool UploadFile(string file, FtpType fileType)
{
    Socket socket;
    XmlDocument mapdoc = new XmlDocument();
    XmlNode root;
    XmlNodeList remoteDirs;
    String fileDir;

    if(file == "")
    {
        return false;
    }

    //get config

    mapdoc.Load(m_routerConfigFile);
    root = mapdoc.DocumentElement;

    if (!(file.IndexOf("/")>= 0))
    {
        fileDir = m_currentDirectory + "/";
        file = fileDir + file;
    }
    else
    {
        fileDir = file.Substring(0,file.LastIndexOf("/")) + "/";
    }

    if (!(AllowAccessToDirectory(fileDir)))
        return false;

    file = file.Replace("/","\\");

    try 
    {
        remoteDirs = root.SelectNodes("directory[@user='" +
            m_clientAuthenticationToken.Username + 
            "' and @dirname='" + fileDir + "']/remotedir");

        if(remoteDirs.Count > 0)
            return RouteTransfer(file, remoteDirs);
    }
    catch (Exception ex)
    {
        m_logOutput.WriteDebugOutput("No routing found for " 
                                           + file + "\n" + ex);
    }

    try
    {  
        socket = GetClientConnection();

        if ((fileType == FtpType.Binary) || (fileType == FtpType.Image))
        {
            BinaryWriter writer = 
              new BinaryWriter(new StreamWriter(file, false).BaseStream);
            byte[] buffer = new byte;
            int bytes;

            while ((bytes = socket.Receive(buffer))>0)
            {
                writer.Write(buffer, 0, bytes);
            }

            writer.Close();
            socket.Shutdown(SocketShutdown.Both);
            socket.Close();
            return true;
        }
        else
        {
            StreamWriter writer = new StreamWriter(file, false);
            byte[] buffer = new byte;
            int bytes;

            while ((bytes = socket.Receive(buffer))>0)
            {
                writer.Write(System.Text.Encoding.ASCII.GetChars(buffer, 
                                                                0, bytes));
            }

            writer.Close();
            socket.Shutdown(SocketShutdown.Both);
            socket.Close();
            return true;
        }
    }
    catch (Exception ex)
    {
        m_logOutput.WriteDebugOutput("UploadFile failed on file " 
                                          + file + "\n" + ex.Message);
        m_logOutput.WriteErrorOutput(m_clientAuthenticationToken.Username, 
          "UploadFile failed on file " + file);
        return false;
    }
}

The RouteTransfer method takes the incoming file and the list of servers where the file should really go. It then sets up connections to the remote servers using the FtpClientControl class. Once all the FtpClientControl classes are setup (and stored in an ArrayList), the upload is really started. The RouteTransfer method loops through the incoming data, and then it loops through the array of FtpClientControl classes and sends that data off to the other servers.

private bool RouteTransfer(string file, XmlNodeList remoteDirs)
{        
    Socket socket;
    byte[] buffer = new byte;
    int bytes;
    ArrayList remoteConns = new ArrayList();;

    //Get the real filename

    if(file.IndexOf("\\") >= 0)
    {
        file = file.Substring(file.LastIndexOf("\\") + 1);
    }

    //Loop through each remote node and connect to the remote server
    foreach(XmlNode remoteDirectory in remoteDirs)
    {
        XmlAttributeCollection attrColl = remoteDirectory.Attributes;

        string server = "";
        string username = "";
        string password = "";
        int timeoutSeconds = 10;
        int port = 21;

        server = attrColl["server"].Value;
        username = attrColl["user"].Value;
        password = attrColl["pass"].Value;
        timeoutSeconds = 
          System.Convert.ToInt32((attrColl["timeoutSeconds"].Value));
        port = System.Convert.ToInt32((attrColl["port"].Value));

        if(server != "" && username != "" && password != "")
        {
            FtpClientControl ftpClient = 
              new FtpClientControl(server, username, 
              password, timeoutSeconds, port);

            ftpClient.RemotePath = remoteDirectory.InnerText;
            ftpClient.Login();
            ftpClient.StartUpload(file);
            remoteConns.Add(ftpClient);
        }
        else
        {
            m_logOutput.WriteErrorOutput(m_clientAuthenticationToken.Username, 
              "Missing a server, username, or password for the " + 
              remoteDirectory.Value + " remotedirectory.");

            return false;
        }
    }

    socket = GetClientConnection();

    //read data from client and send data elsewhere
    while ((bytes = socket.Receive(buffer)) > 0)
    {
        foreach(FtpClientControl ftc in remoteConns)
        {
            ftc.SendUploadData(buffer, bytes);
        }
    }

    //When finished, close remote client conns
    foreach(FtpClientControl ftc in remoteConns)
    {
        ftc.FinishUpload();
    }

    socket.Shutdown(SocketShutdown.Both);
    socket.Close();

    return true;
}

The methods listed below are StartUpload, SendUploadData, and FinishUpload. They are methods contained in the FtpClientControl class and are used by the RouteTransfer method in FtpServerControl to replicate incoming uploads to other servers.

public void StartUpload(string fileName)
{
    if ( !m_loggedin ) Login();

    m_transferSocket = createDataSocket();

    sendCommand("STOR " + fileName);

    if ( m_resultCode != 125 && m_resultCode != 150 ) 
        throw new FtpException(m_result.Substring(4));

    Debug.WriteLine( "Uploading file " + fileName + 
              " to " + m_remotePath, "FtpClient" );
}

public void SendUploadData(byte[] buffer, int bytes)
{
    m_transferSocket.Send(buffer, bytes, 0);
}

public void FinishUpload()
{
    if (m_transferSocket.Connected)
    {
        m_transferSocket.Close();
    }
    readResponse();

    if( m_resultCode != 226 && m_resultCode != 250 ) 
        throw new FtpException(m_result.Substring(4));

    m_transferSocket = null;
}

Points of Interest

My first point of interest would be to note that I used the SharpDevelop development environment to develop this program. While SharpDevelop isn't quite as stable as VS.NET, it is definitely cheaper. Check it out here.

Another note is that I spent a lot of time on this project fidgeting with the OnStart and OnStop methods in the service stub, mainly due to the fact that the FTP server code was multi-threaded. It kept hanging in the OnStart, and when it went to stop, well it wouldn't (at least not immediately). So, I had to add the SharpServerListener class as a member class, so I could cause it to shutdown its client threads.

I'm not 100% that everything is coded correctly in the whole project, but I do know that it works (or at least it works on my PC.) It is definitely a security no-no to keep all the local and remote usernames and passwords out in plain site.

If I had to add more enhancements to the server, here is what I would add:

1. A separate mechanism for storing, updating, and retrieving security information.
2. The ability to route other actions (renaming and deleting files).
3. A GUI admin console.
4. An installation process for the service.
5. A better way for the server to process incoming commands.
6. A better way to manage the client threads (ThreadPool?).
7. The ability to "buffer" the file to disk and send to the remote servers later.