Home Automation with Netduino and Kinect

Introduction

Home automation has been an interest of mine for a long time. There is a bunch of bad technology in the marketplace and the products are too expensive so I decided to build my own. I started out with the Arduino microcontroller which was really fun but the code quickly became hard to maintain because it was not object oriented. Additionally it could not do multithreading or real debugging with breakpoints and such. I refactored the code for C# and the .NET Micro Framework. I choose the netduino plus, http://www.netduino.com/netduinoplus/specs.htm, for the microcontroller which has a built in Ethernet adapter for network communication.

Please be sure to also see the sequels to this article:

• Using jQuery Mobile with MVC and Netduino for Home Automation
• Home Automation with Microsoft Kinect Point Cloud and Speech Recognition
• IoT for Home Automation

The figure below shows an early prototype of the project.

Netduino Controlled Squirt Gun

The first project that I built was a servo controlled squirt gun for the pool. The code that I wrote for the netduino controls the servos to spray the gun in different patterns in the pool. I then built a Windows Phone 7 interface to aim the servos to the position on the screen where you touch. I used IIS live smooth streaming to stream video to the phone so you could remotely nail the kids in the pool from anywhere. I had mixed results with the video piece and at some point I need to spend more time perfecting and reducing the buffering time to make it more real time.

Garden

My next project was to control the irrigation of the garden. My code schedules the times to water the garden and controls the duration of the watering.

Kinect

One of my colleagues in the office started doing projects with the Microsoft Kinect which has a rich SDK complete with drivers, APIs and plenty of good sample code. The Kinect has a bunch of sensors including a RGB camera, depth sensor and multi-array microphone. With a Kinect, you are the controller! I got the idea of using the Kinect for the controller of the squirt gun in the pool. You can now aim the gun by pointing to where you want it to shoot. The trigger is controlled by bending your other arm so that the hand is above the elbow joint. Plugging in the Kinect for the controller was really simple because of the rich Kinect API and because I had already written the back end tiers to communicate with the netduino microcontroller.

Watch this video of the Squirt Gun

Speech Recognition on the Kinect

One of the other features on the Kinect is the multi-array microphone with speech recognition. I played around with speech commands to control the squirt gun and to open the garage.

Watch this video of Simon Says Kinect

Android Garage Door Opener

I wanted to learn a little about Android development so I wrote a native Android app to call a REST web service (WCF) that talks to the netduino to open the garage.

Putting it all Together

The image below shows the communication between the components.

The image below shows the devices that the Netduino controls. The fireplace project is in progress and I just started working on it.

Watch this video to see how it all comes together

Ethernet Communication

The Ethernet communication with the netduino was the hardest part of the project.

using System;
using System.Net;
using System.Net.Sockets;
using System.Text;
using Microsoft.SPOT;
using Microsoft.SPOT.Hardware;
using SecretLabs.NETMF.Hardware;
using SecretLabs.NETMF.Hardware.Netduino;
using Microsoft.SPOT.Net.NetworkInformation;
using System.Threading;

namespace Netduino.Controller
{
    public delegate void MessageEventHandler(string Message);

    class EthernetCommunication
    {
        #region Private Variables
        private string _hostAddress = null;
        private int _port = 80;
        private string _netduinoStaticIPAddress = null;
        private string _subnetMask = null;
        private string _gatewayAddress = null;
        private Thread _listeningThread;
        private Socket _clientSocket = null;
        private static EthernetCommunication _ethernetCommunication;
        #endregion

        #region Constructors
        //This keeps other classes from creating an instance
        private EthernetCommunication()
        {
        }
        #endregion

        #region Public Properties
        public string HostAddress
        {
            set { _hostAddress = value; }
            get { return _hostAddress; }
        }
        public int Port
        {
            set { _port = value; }
            get { return _port; }
        }
        public string NetduinoStaticIPAddress
        {
            set 
            { 
                _netduinoStaticIPAddress = value;
                SetNetduinoStaticIPConfiguration();
            }
            get { return _netduinoStaticIPAddress; }
        }
        public string SubnetMask
        {
            set
            {
                _subnetMask = value;
                SetNetduinoStaticIPConfiguration();
            }
            get { return _subnetMask; }
        }
        public string GatewayAddress
        {
            set 
            {
                _gatewayAddress = value;
                SetNetduinoStaticIPConfiguration();
            }
            get { return _gatewayAddress; }
        }
        #endregion

        #region Events
        public static event MessageEventHandler EventHandlerMessageReceived;
        #endregion

        #region Public Methods
        
        private void StartListening()
        {
            _listeningThread = new Thread(new ThreadStart(ReceiveSocketsInListeningThreadAndHandleSocketExceptions));
            _listeningThread.Start();
        }

        private void InitializeConfiguration()
        {
            if (_netduinoStaticIPAddress == null)
                throw new Exception("The netduino Static IP Address nust be set!");

            if (_subnetMask == null)
                throw new Exception("The Subnet Mask must be set!");

            if (_gatewayAddress == null)
                throw new Exception("The Gateway address must be set.");

            SetNetduinoStaticIPConfiguration();
            NetworkInterface networkInterface = NetworkInterface.GetAllNetworkInterfaces()[0];

            if (_netduinoStaticIPAddress != networkInterface.IPAddress)
                throw new Exception("Problem setting the static IP.");

            if (_subnetMask != networkInterface.SubnetMask)
                throw new Exception("Problem setting the subnet mask.");

            if (_gatewayAddress != networkInterface.GatewayAddress)
                throw new Exception("Problem setting the gateway address.");
        }
        #endregion

        #region Public Static Methods
        public static EthernetCommunication GetInstance()
        {
            if (_ethernetCommunication == null)
            {
                _ethernetCommunication = new EthernetCommunication();
                _ethernetCommunication.HostAddress = Config.HostAddress;
                _ethernetCommunication.Port = Config.Port;
                _ethernetCommunication.NetduinoStaticIPAddress = Config.NetduinoStaticIPAddress;
                _ethernetCommunication.SubnetMask = Config.SubnetMask;
                _ethernetCommunication.GatewayAddress = Config.GatewayAddress;
                _ethernetCommunication.InitializeConfiguration();
                _ethernetCommunication.StartListening();
            }
            return _ethernetCommunication;
        }

        public static void SendMessage(string message)
        {
            GetInstance().SendEthernetMessage(message);
        }
        #endregion

        #region Private Methods
        private bool IsSocketConnected(Socket socket)
        {
            bool connectionNotClosedResetOrTerminated = !socket.Poll(1000, SelectMode.SelectRead);
            bool socketHasDataAvailableToRead = (socket.Available != 0);
            return (connectionNotClosedResetOrTerminated || socketHasDataAvailableToRead);
        }

        private void ReceiveSocketsInListeningThreadAndHandleSocketExceptions()
        {
            try
            {
                ReceiveSocketsInListeningThread();
            }
            catch (SocketException se)
            {
                Debug.Print("Socket Exception!  Probably WiFi or Ethernet connection not working?");
                Debug.Print(se.StackTrace);

                Debug.Print("Rebooting netduino to recover.");
                PowerState.RebootDevice(false);
            }
            catch (Exception ex)
            {
                Debug.Print("Non socket exception.");
                Debug.Print(ex.StackTrace);
            }
        }

        private void ReceiveSocketsInListeningThread()
        {
            string receiveMessage = "";
            bool exitProgram = false;

            using (System.Net.Sockets.Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
            {
                socket.Bind(new IPEndPoint(IPAddress.Any, _port));
                socket.Listen(10);

                while (!exitProgram)
                {
                    Debug.Print("Waiting for message from socket...");

                    // This call is "blocking" and will will wait for a socket.  
                    // The thread will wait here until a message is received
                    _clientSocket = socket.Accept();

                    Debug.Print("Message received!");

                    using (_clientSocket)
                    {
                        while (IsSocketConnected(_clientSocket))
                        {
                            int availablebytes = _clientSocket.Available;
                            byte[] buffer = new byte[availablebytes];
                            _clientSocket.Receive(buffer);
                            if (buffer.Length > 0)
                            {
                                receiveMessage = new string(Encoding.UTF8.GetChars(buffer));
                                RaiseMessageReceivedEvent(receiveMessage);
                                if (receiveMessage.ToUpper() == "EXIT")
                                {
                                    exitProgram = true;
                                }
                            }
                            
                        }
                    }
                }
            }
        }

        private void RaiseMessageReceivedEvent(string message)
        {
            // Event will be null if there are no subscribers
            if (EventHandlerMessageReceived != null)
            {
                EventHandlerMessageReceived(message);
            }
        }

        private void SetNetduinoStaticIPConfiguration()
        {
            //Exit if not all of the configuration properties are set
            if (_netduinoStaticIPAddress == null || _subnetMask == null || _gatewayAddress == null)
                return;

            NetworkInterface networkInterface = NetworkInterface.GetAllNetworkInterfaces()[0];

            bool _ipAddressAlreadySet = _netduinoStaticIPAddress == networkInterface.IPAddress;
            bool _subnetMaskAlreadySet = _subnetMask == networkInterface.SubnetMask;
            bool _gatewayAlreadySet = _gatewayAddress == networkInterface.GatewayAddress;

            if (_ipAddressAlreadySet && _subnetMaskAlreadySet && _gatewayAlreadySet)
                return;

            // Set our IP address to a new value
            // This will be saved in the config sector of the netduino and will survive reboots 
            networkInterface.EnableStaticIP(_netduinoStaticIPAddress, _subnetMask, _gatewayAddress);
        }

        private void SendEthernetMessage(string message)
        {
            if (_hostAddress != null && _port > 0)
            {
                using (System.Net.Sockets.Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
                {
                    IPHostEntry entry = Dns.GetHostEntry(_hostAddress);
                    IPAddress address = entry.AddressList[0];
                    IPEndPoint endpoint = new IPEndPoint(address, _port);

                    try
                    {
                        socket.Connect(endpoint);
                        socket.Send(Encoding.UTF8.GetBytes(message));
                        socket.Close();
                        Debug.Print(message);
                    }
                    catch (SocketException se)
                    {
                        Debug.Print("Socket Exception!  Probably no server or bad ip?");
                        Debug.Print(se.StackTrace);

                        Debug.Print("Rebooting netduino to recover.");
                        PowerState.RebootDevice(false);
                    }
                    catch (Exception ex)
                    {
                        Debug.Print("Non socket exception.");
                        Debug.Print(ex.StackTrace);
                    }
                }
            }
        }
        #endregion
    }
}

If you want to see an example of how to talk with a desktop application, download the source code and look at the Netduino.Desktop.Messenger project.

Servo Communication

The servos were fun to program. I wrote a servo class that you set the Angle and the Minimum and Maximum degrees. I added the Inverted property to invert the angle because I have an indoor version of the squirt gun that is mounted from the floor while the outdoor version is mounted upside down.

using System;
using Microsoft.SPOT.Hardware;
using SecretLabs.NETMF.Hardware;

namespace Netduino.Controller
{
    public class Servo : IDisposable
    {
        #region Private Variables
        private PWM _servo;
        private bool _invertAngle = false;
        private int _degreeMin = Config.DegreeMinDefault;
        private int _degreeMax = Config.DegreeMaxDefault;
        private uint _durationMin = Config.DurationMinDefault;
        private uint _durationMax = Config.DurationMaxDefault;
        private uint _angle = Config.HomeDefaultAngle;
        private uint _period = Config.PeriodDefault;
        #endregion

        #region Constructors
        public Servo(Cpu.Pin pin)
        {
            _servo = new PWM(pin);
            _servo.SetDutyCycle(0);
        }
        #endregion

        #region Public Methods
        public void Dispose()
        {
            DisengageServo();
            _servo.Dispose();
        }

        /// <summary> 
        /// Disengage the servo.  
        /// The servo motor will stop trying to maintain an angle 
        ///  
        public void DisengageServo()
        {
            _servo.SetDutyCycle(0);
        }

        public void EngageServo()
        {
            SetPulse();
        }
        #endregion

        #region Private Methods
        private void SetPulse()
        {
            uint angle = _invertAngle ? 180 - _angle: _angle;
            uint duration = (angle) * (_durationMax - _durationMin) / 180 + _durationMin;
            _servo.SetPulse(period: _period, duration: duration);  
        }
        #endregion

        #region Public Properties
        public int Angle
        {
            set
            {
                if (value > _degreeMax)
                    value = _degreeMax;

                if (value < _degreeMin)
                    value = _degreeMin;

                if (value < 0)
                    value = 0;

                _angle = (uint)value;
                SetPulse();
            }
            get
            {
                return (int)_angle;
            }
        }

        public bool InvertAngle
        {
            set {_invertAngle = value;}
            get { return _invertAngle; }
        }

        public int DegreeMin
        {
            set {_degreeMin  = value;}
            get { return _degreeMin; }
        }

        public int DegreeMax
        {
            set { _degreeMax = value; }
            get { return _degreeMax; }
        }

        public uint durationMin
        {
            set { _durationMin = value; }
            get { return _durationMin; }
        }

        public uint durationMax
        {
            set { _durationMax = value; }
            get { return _durationMax; }
        }

        public uint period
        {
            set { _period = value; }
            get { return _period; }
        }
        #endregion
    }
}

Controlling the Garden

I wrote a library of time commands for the .NET Micro Framework. Note that the .net micro framework is very rich but does not have generics.

using System;
using System.Threading;
using System.Collections;
using Microsoft.SPOT;

namespace Netduino.Controller
{
    public delegate void AlarmCallback();

    class AlarmData
    {
        public int Key { get; set; }
        public ExtendedTimer ExtendedTimer { get; set; }
        public bool RemoveAfterRun { get; set; }
        public AlarmCallback Callback { get; set; }
    }

    class Time
    {
        #region Private Variables
        private static Hashtable _alarmHashtable;
        private static int _key;
        #endregion

        #region Constructors
        //This keeps other classes from creating an instance
        private Time()
        {
        }
        #endregion

        #region Public Static Methods
        public static void SetTime(int year, int month, int day, int hour,int minute, int second, int millisecond )
        {
            DateTime presentTime = new DateTime( year, month, day, hour, minute, second, millisecond);
            Microsoft.SPOT.Hardware.Utility.SetLocalTime(presentTime);
        }

        public static void RunDaily(AlarmCallback alarmCallback, int hour, int minute, int second)
        {
            DateTime alarmTime = new DateTime(DateTime.Now.Year, DateTime.Now.Month, DateTime.Now.Day, hour, minute, second, 0);
            
            //If we already missed today then tomorrow is the first day to run
            if(alarmTime<DateTime.Now)
            {
                alarmTime = alarmTime.AddDays(1);
            }

            TimeSpan dailyTimeSpan = new TimeSpan(24, 0, 0);
            CreateAlarm(alarmCallback, alarmTime, dailyTimeSpan, false);
        }

        public static void RunOnDelay(AlarmCallback alarmCallback, int runInMilliseconds)
        {
            DateTime alarmTime = DateTime.Now.AddMilliseconds(runInMilliseconds);
            CreateAlarm(alarmCallback, alarmTime, TimeSpan.Zero, true);
        }

        public static void RunRepetitively(AlarmCallback alarmCallback, int repeatMilliseconds)
        {
            DateTime alarmTime = DateTime.Now.AddMilliseconds(repeatMilliseconds);
            TimeSpan repeatTimeSpan = new TimeSpan(0, 0, 0, 0, repeatMilliseconds);
            CreateAlarm(alarmCallback, alarmTime, repeatTimeSpan, false);
        }
        #endregion

        #region Private Methods
        private static void CreateAlarm(AlarmCallback alarmCallback, DateTime alarmTime, TimeSpan timeSpan, bool removeAfterRun)
        {
            if (_alarmHashtable == null)
                _alarmHashtable = new Hashtable();
            
            _key=_key+1;

            AlarmData alarmData = new AlarmData();
            alarmData.Key = _key;
            alarmData.Callback = alarmCallback;
            alarmData.ExtendedTimer = new ExtendedTimer(OnExecuteAlarm, alarmData, alarmTime, timeSpan);
            alarmData.RemoveAfterRun = removeAfterRun;

            _alarmHashtable.Add(_key, alarmData);
        }

        private static void OnExecuteAlarm(object target)
        {
            AlarmData alarmData = (AlarmData)target;
            
            if (alarmData.RemoveAfterRun)
                _alarmHashtable.Remove(alarmData.Key);

            alarmData.Callback.Invoke();
        }
        #endregion
    }
}

The Main .NET Micro Framework Program

There is an event handler for the Ethernet Communication that runs when a message is received.

EthernetCommunication.EventHandlerMessageReceived += new MessageEventHandler(OnMessageReceived);

The OnMessageReceived method parses the message and calls the methods to execute the commands. The code snippet below is only partial, but the full source is available for download in this article.

private static void OnMessageReceived(string message)
{
    string[] parts = message.Split(' ');
    switch(parts[0].ToUpper())
    {
        case "M":
        case "MOVE":
            if (parts.Length != 3)
            {
                EthernetCommunication.SendMessage("The move command takes 3 arguments.");
                break;
            }
            int leftRightAngle = int.Parse(parts[1]);
            int upDownAngle = int.Parse(parts[2]);
            _squirtGun.MoveToPosition(leftRightAngle, upDownAngle);
            break;

        case "U":
        case "UP":
            int upDelta = parts.Length > 1 ? int.Parse(parts[1]) : 1;
            _squirtGun.UpDownAngle = _squirtGun.UpDownAngle + upDelta;
            break;

About the Author

Follow Dan on twitter: @LogicalDan

Dan graduated summa cum laude from North Carolina State University with dual degrees in Electrical Engineering and Computer Engineering. Dan attended NC State on full scholarship program with General Motors. After working with GM, Dan served as application development director for the largest Microsoft Business Solutions Partner in the Carolinas. During this time, Dan's team won two Microsoft Pinnacle awards. For the past 12 years, as Co-Founder and Chief Technology Officer of, Logical Advantage (www.logicaladvantage.com), a software consulting business, Dan has successfully architected and delivered web-based and mobile applications for many Fortune 500 companies. Dan focuses his energies on emerging technologies, and ensuring that all projects are architected to meet the client's current and future needs. Dan collaborates with his Chief Solutions Officer and other architects to create technical standards, including coding standards, tools, and platforms. He holds a leadership role in the local Microsoft Enterprise Developer's Guild and has been on the steering committee for over a dozen years.

Download the Source Code

Click this to download the source code for the netduino .net micro framework projects.

Continue Reading my Home Automation Series:

Click this link to read part 2 of my home automation projects: Using jQuery Mobile with MVC and Netduino for Home Automation

Click this link to read part 3 of my home automation projects: Home Automation with Microsoft Kinect Point Cloud and Speech Recognition

Click this link to read part 4 of my home automation projects: IoT for Home Automation