This article is a tutorial on doing a battery of advanced tasks with the Arduino Mega 2560+WiFi R3. We will create a clock that can automatically configure itself on your network, sync using NTP, report the time local temperature and humidity, and expose a local website that reports the same, both as a dynamic webpage, and a JSON based web service.
Introduction
Sometimes, there's more to a device than meets the eye. We're using a clock as an excuse to explore some advanced techniques for IoT devices, even the humble and ironically named Mega. We'll be putting together a clock based on some tutorials, and then writing some witchy code to give it some magic. I'll also be providing you with some libraries that ease the automatic network configuration portion in your own apps.
Prerequisites
You'll need an Arduino Mega 2560+WiFi R3.
You'll need a DS1307 or similar Real-time Clock.
You'll need a DHT11 temperature and humidity sensor.
You'll need a 16x2 LCD display with a hitachi interface.
You'll need the usual wires and prototype board.
You'll need the Arduino IDE. You must go to File|Preferences and add the following to the Additional Board Managers text box:
If there is already a URL present, delimit them with commas.
Now get yourself the latest ESP8266FS zip file here.
If you're on Linux, you'll want to get the ESP8266FS folder in the zip, and find your Arduino application directory. That should be off your home directory. Mine is ~/arduino-1.8.13. Do not get it mixed up with ~/Arduino. Under there, there is a tools folder and that's where you want your ESP8266FS folder to go.
I've never done it on Windows, but this is how you do it: You'll need the ESP8266FS folder from the zip. Find your program directory. It is probably something like C:\Program Files (x86)\arduino-1.8.13. Inside, there is a tools folder. That is where the ESP8266FS folder needs to go.
Either way, you'll need to restart the Arduino IDE. You'll know it took if you now have a Tools|ESP8266 Sketch Data Upload option.
You'll also need to install all of the libraries in ESP8266AutoConfLibs.zip. After unzipping this, you can go to Sketch|Include Library|Add Zip Library... and select each of the zips to install them.
Conceptualizing this Mess
Connecting and Configuring
This is the most involved part of the project, and the main reason I created this project and article, so we're going to spend some time with this subtopic in particular.
My IoT devices typically autoconfigure their WiFi connections and support WPS. What I mean is my devices don't require anything to be done to them other than to be switched on. Once the WPS button on the router is pressed, they will connect. This is relatively straightforward for devices where a network connection is required because we can stall the rest of the application until the connection and configuration takes place, but what about an application like a clock that needs to run continuously while ideally looking for Internet connections or WPS signals in the background? With the default ESP8266WiFi library, this is pretty much impossible.
Here are the steps we take to connect to a network:
- Read the SSID and the password from the configuration.
- Use that if available to attempt to make a connection to that network, otherwise skip this step.
- If #2 times out or fails or we skipped #2, attempt to run a scan for a WPS signal and connect if we find it.
- If we timeout/fail to find a WPS signal, go to #2.
- Otherwise, save the new SSID and password into the configuration and continue.
In the foreground where we block until the process completes, this is relatively straightforward as I said. It's considerably more complicated when it must be done in the background. The ESP8266WiFi library's beginWPSconfig()
is synchronous, so I struggled for awhile before making an asynchronous version like begin()
is. My library is called ESP8266AsyncWiFi and the aforementioned method is the only behavior that I changed. When I first tried this, I tried to do it without forking the WiFi library but the result ended up randomly crashing because I couldn't call some private methods that made it work.
That wasn't the end of the mess however. In order to make all the timeouts work and everything without using delay()
, and also to manage the different transitions between the steps above, I built a state machine that is called from loop()
. The state machine moves from one state to the next as the connection and WPS negotiation process is navigated. There are two timeouts involved - one for the connection and one for the WPS search. Basically, we just go back and forth between trying to connect and trying to find a WPS signal but we do so in a way that doesn't block. It's not pretty - state machines rarely are, but at least it's not more complicated than it needs to be.
I've turned all of my autoconfiguration stuff into several libraries which I've included with this distribution. You can install each of the zips using the Arduino IDE by going to Sketch|Include Library|Add Zip Library...
- ESP8266AsyncWifi is required for both of the included async autoconf libraries (see below).
- ESP8266AsyncAutoConf is a background auto WiFi config library that uses raw flash to store the network credentials.
- ESP8266AsyncAutoConfFS is a background auto WiFi config library that uses the SPIFFS flash filesystem to store the network credentials. This is useful if you already intend to use the filesystem for things such as serving web pages.
- ESP8266AutoConf is a foreground auto WiFi config library that uses raw flash to store the network credentials.
- ESP8266AutoConfFS is a foreground auto WiFi config library that uses the SPIFFS flash filesystem to store the network credentials.
How to pick which of the AutoConf libraries is right for your project:
- If your device doesn't require a network in order to operate, you should use #2 or #3 above.
- If your device does require a network in order to operate, you should use #4 or #5 above.
- If your device will need a filesystem, use either #3 or #5 above
- If your device will not need a filesystem, use #2 or #4 above
That's just a general guide to using them. We'll be selecting #3 for our project since we don't require a network connection in order to function and because we need a filesystem. This way, our clock can serve a small website and webservice while automatically searching for a WPS signal or a usable WiFi connection in the background.
The Clock Hardware
I've basically cobbled together the hardware from several example projects, here, here, and here. I'll expect you can follow them, and combine them onto one prototype board. Remember on the Mega when you go to attach the clock, you'll want to set it to the second I2C interface (SDA20
/SCL21
), not the first set. You'll also want the DHT sensor's S line to be plugged into A0
and the corresponding code will need to be updated to change the pin to A0
in the code. If your clock isn't a DS1307, you'll need to adjust your code and wiring accordingly.
Once you have it wired up and tested with some throwaway code, we can get to the meat. Here's some throwaway code for testing:
#include <LiquidCrystal.h>
#include <dht.h>
#include <RTClib.h>
RTC_DS1307 RTC;
dht DHT;
float _temperature;
float _humidity;
#define DHT11_PIN A0
LiquidCrystal LCD(8, 9, 4, 5, 6, 7);
void setup() {
Serial.begin(115200);
LCD.begin(16, 2);
pinMode(DHT11_PIN, INPUT);
if (! RTC.begin()) {
Serial.println(F("Couldn't find RTC"));
while (true);
}
RTC.adjust(DateTime(__DATE__, __TIME__));
}
void loop()
{
int chk = DHT.read11(DHT11_PIN);
float f = DHT.temperature;
if (-278 < f) {
_temperature = f;
_humidity = DHT.humidity;
}
DateTime now = RTC.now();
char sz[16];
memset(sz,' ',16);
sprintf(sz, "%d:%02d:%02d",
now.hour(),
now.minute(),
now.second());
int c = strlen(sz);
if(c<16)
sz[c]=' ';
LCD.setCursor(0,0);
LCD.print(sz);
LCD.setCursor(0, 1);
memset(sz,' ',16);
if (1==(millis() / 1000L) % 2) {
sprintf(sz,"Temp: %dC/%dF",(int)_temperature,(int)((int)_temperature * 1.8) + 32);
int c = strlen(sz);
if(c<16)
sz[c]=' ';
} else {
sprintf(sz,"Humidity: %d",(int)_humidity);
int c = strlen(sz);
if(c<16)
sz[c]=' ';
}
LCD.print(sz);
}
That should display the time, temperature, and humidity on the display if everything is working.
The ATMega2560
This processor will be responsible for managing the LCD output and communicating the clock and sensor readings with the XDS 160Micro processor on the WiFi module. We'll be using that processor to do almost all the hard work because it's much more capable and it's got WiFi basically built right in instead of needing to be accessed over serial. We will be using a serial port though but just to get sensor data and clock information back and forth. For the most part, aside from updating the clock's display, this simply listens on a serial port for an incoming 255 value. If it gets that, it reads an opcode next, and then command bytes which depend on the opcode. Any other value gets forwarded to the serial port that's exposed via USB.
The XDS 160Micro
This processor will be responsible for negotiating the WiFi network, running the webserver, communicating with an NTP server, and setting the clock when necessary. Any time it needs sensor or clock information it must query the ATMega2560 over a serial line. It does this by sending the byte 255, followed by 0 and then receives all of the clock and sensor data. If it sends a 255 followed by a 1 and then a 32-but number representing the Unix time, it will set the clock. It runs a web server at http://clock.local which will present the time, temperature and humidity. It runs a JSON service at http://clock.local/time.json.
Coding this Mess
The ESPAsyncAutoConf Facility
This facility is what automatically scans for WiFi and WPS, and manages the SSID and password stored in the device.
Using It
Before we dive into what makes it work, let's look at how to use it:
#include <ESP8266AsyncAutoConfFS.h> // header chosen from #3 above
In our setup()
method:
ESPAsyncAutoConf.begin();
If you weren't using the async version, you'd invoke ESPAutoConf.begin()
instead.
In the loop()
method:
ESPAsyncAutoConf.update();
Similarly as above, if you weren't using the async versions, you'd refer to ESPAutoConf
instead.
You can see if you're connected to the network as normal:
if(WL_CONNECTED == WiFi.status()) Serial.println("Connected!");
With the synchronous versions after the update()
call, you will always be connected. For the asynchronous versions, they very well may not be connected after update()
is called.
Remember to always check whether you're connected before you do a network operation when using the asynchronous libraries!
That's about it for using it. Let's dive into how it was made.
Building It
I've already covered the mechanics of the synchronous versions of these libraries here. The code in the libraries is updated a bit since the article but the concept hasn't changed. I will be focusing on the asynchronous configuration process in this article.
Most of the meat of these asynchronous libraries is in update()
which is typically called from loop()
:
#include "ESP8266AsyncAutoConfFS.h"
_ESPAsyncAutoConf ESPAsyncAutoConf;
void _ESPAsyncAutoConf::begin() {
SPIFFS.begin();
int i = 0;
_cfgssid[0] = 0;
_cfgpassword[0] = 0;
File file = SPIFFS.open("/wifi_settings", "r");
if (file) {
if (file.available()) {
int l = file.readBytesUntil('\n', _cfgssid, 255);
_cfgssid[l] = 0;
if (file.available()) {
l = file.readBytesUntil('\n', _cfgpassword, 255);
_cfgpassword[l] = 0;
}
}
file.close();
}
WiFi.mode(WIFI_STA);
_wifi_timestamp = 0;
_wifi_conn_state = 0;
}
void _ESPAsyncAutoConf::update() {
switch (_wifi_conn_state) {
case 0: if (WL_CONNECTED != WiFi.status())
{
if (0 < strlen(_cfgssid)) {
Serial.print(F("Connecting to "));
Serial.println(_cfgssid);
if (WiFi.begin(_cfgssid, _cfgpassword)) {
_wifi_conn_state = 1;
_wifi_timestamp = 0;
}
} else {
_wifi_conn_state = 2;
_wifi_timestamp = 0;
}
}
break;
case 1: if (WL_CONNECTED != WiFi.status()) {
if (!_wifi_timestamp)
_wifi_timestamp = millis();
else if (20000 <= (millis() - _wifi_timestamp)) {
Serial.println(F("Connect attempt timed out"));
_wifi_conn_state = 2;
_wifi_timestamp = 0;
}
} else {
Serial.print(F("Connected to "));
Serial.println(WiFi.SSID());
strcpy(_cfgssid,WiFi.SSID().c_str());
strcpy(_cfgpassword,WiFi.psk().c_str());
File file = SPIFFS.open("/wifi_settings", "w");
if (file) {
file.print(_cfgssid);
file.print("\n");
file.print(_cfgpassword);
file.print("\n");
file.close();
}
_wifi_conn_state = 4;
_wifi_timestamp = 0;
}
break;
case 2: Serial.println(F("Begin WPS search"));
if (WL_CONNECTED != WiFi.status()) {
WiFi.beginWPSConfig();
_wifi_conn_state = 3;
_wifi_timestamp = 0;
}
break;
case 3: if (WL_CONNECTED != WiFi.status()) {
if (!_wifi_timestamp)
_wifi_timestamp = millis();
else if (30000 <= (millis() - _wifi_timestamp)) {
Serial.println(F("WPS search timed out"));
_wifi_conn_state = 0;
_wifi_timestamp=0;
}
} else {
_wifi_conn_state = 1;
_wifi_timestamp = 0;
}
break;
case 4:
if (WL_CONNECTED != WiFi.status()) {
_wifi_conn_state = 0;
_wifi_timestamp = 0;
}
break;
}
}
You may notice that this is a state machine. We've covered what it does in the conceptualization section prior. The logic is a bit messy but necessarily so in order to handle all the cases. I love state machines in concept but not so much in practice because they can be difficult to read. However, sometimes they are just the right tool for the job.
The whole idea of this routine is to break up the process of connecting, scanning WPS, connecting, scanning WPS ad nauseum into a coroutine - a "restartable method" - that's where the state machine comes in. Each time loop()
is called, we pick up where we left off last time because we're tracking the state with _wifi_conn_state
. It's not really the most self explanatory code but if you pay close attention, you can follow it.
The ATmega2560
Here is the processing code for the ATmega2560 CPU which we've already covered prior, so let's get to the code:
#include <dht.h>
#include <RTClib.h>
#include <LiquidCrystal.h>
#define DHT11_PIN A0
typedef union {
float fp;
byte bin[4];
} binaryFloat;
typedef union {
uint32_t ui;
byte bin[4];
} binaryUInt;
float _temperature;
float _humidity;
dht DHT;
RTC_DS1307 RTC;
LiquidCrystal LCD(8, 9, 4, 5, 6, 7);
void setup() {
Serial.begin(115200);
Serial3.begin(115200);
pinMode(DHT11_PIN,INPUT);
if (! RTC.begin()) {
Serial.println(F("Couldn't find the clock hardware"));
while (1);
}
if (! RTC.isrunning())
Serial.println(F("The clock is not running!"));
LCD.begin(16, 2);
}
void loop() {
int chk = DHT.read11(DHT11_PIN);
float tmp = DHT.temperature;
if (-278 < tmp) {
_temperature = tmp;
_humidity = DHT.humidity;
}
DateTime now = RTC.now();
char sz[16];
memset(sz,' ',16);
sprintf(sz, "%d:%02d:%02d",
now.hour(),
now.minute(),
now.second());
int c = strlen(sz);
if(c<16)
sz[c]=' ';
LCD.setCursor(0,0);
LCD.print(sz);
LCD.setCursor(0, 1);
memset(sz,' ',16);
if (1==(millis() / 2000L) % 2) {
sprintf(sz,"Temp: %dC/%dF",(int)_temperature,(int)((int)_temperature * 1.8) + 32);
int c = strlen(sz);
if(c<16)
sz[c]=' ';
} else {
sprintf(sz,"Humidity: %d",(int)_humidity);
int c = strlen(sz);
if(c<16)
sz[c]=' ';
}
LCD.print(sz);
if (Serial3.available()) {
byte b = Serial3.read();
if (255 != b)
{
Serial.write(b);
return;
}
b = Serial3.read();
switch (b) {
case 0: binaryUInt data;
data.ui = RTC.now().unixtime();
Serial3.write(data.bin, 4);
binaryFloat data2;
data2.fp = _temperature;
Serial3.write(data2.bin, 4);
data2.fp = _humidity;
Serial3.write(data2.bin, 4);
break;
case 1: Serial3.readBytes((char*)data.bin,4);
RTC.adjust(DateTime(data.ui));
break;
}
}
}
Remember to set your dips to 1-4 ON and 5-8 OFF. Also select the "Arduino Mega or Mega 2560" from the boards menu before flashing the above code.
The XDS Micro160
Here's the code for the XDS Micro160. As you can see, this is a lot more involved. This is due to all the features of the clock more than anything.
#include <ESP8266AsyncAutoConfFS.h>
#include <ESPAsyncTCP.h>
#include <ESPAsyncWebServer.h>
#include <ESP8266mDNS.h>
#include <WiFiUdp.h>
typedef union {
float fp;
byte bin[4];
} binaryFloat;
typedef union {
uint32_t ui;
byte bin[4];
} binaryUInt;
#define HOSTNAME "clock"
unsigned int localPort = 2390;
IPAddress timeServerIP;
const char* ntpServerName = "time.nist.gov";
const int NTP_PACKET_SIZE = 48;
byte packetBuffer[ NTP_PACKET_SIZE];
WiFiUDP udp;
unsigned long _ntp_timestamp;
unsigned long _mdns_timestamp;
bool _net_begin;
AsyncWebServer server(80);
void sendNTPpacket(IPAddress& address);
String process(const String& arg);
void setup() {
_ntp_timestamp = 0;
_mdns_timestamp = 0;
_net_begin = false;
Serial.begin(115200);
ESPAsyncAutoConf.begin();
server.on("/", HTTP_GET, [](AsyncWebServerRequest * request) {
request->send(SPIFFS, F("/index.html"), String(), false, process);
});
server.on("/time.json", HTTP_GET, [](AsyncWebServerRequest * request) {
request->send(SPIFFS, F("/time.json"), String(), false, process);
});
}
void loop() {
if (WL_CONNECTED != WiFi.status()) {
_mdns_timestamp = 0;
_net_begin = false;
}
ESPAsyncAutoConf.update();
if (WL_CONNECTED == WiFi.status()) {
if (!_net_begin) {
_net_begin = true;
MDNS.begin(F(HOSTNAME));
server.begin();
udp.begin(localPort);
Serial.print(F("Started http://"));
Serial.print(F(HOSTNAME));
Serial.println(F(".local"));
}
if (!_ntp_timestamp)
_ntp_timestamp = millis();
else if (300000 <= millis() - _ntp_timestamp) {
WiFi.hostByName(ntpServerName, timeServerIP);
sendNTPpacket(timeServerIP); _ntp_timestamp = 0;
}
if (!_mdns_timestamp)
_mdns_timestamp = millis();
else if (1000 <= millis() - _mdns_timestamp) {
MDNS.update();
_mdns_timestamp = 0;
}
if (0 < udp.parsePacket()) {
udp.read(packetBuffer, NTP_PACKET_SIZE);
unsigned long highWord = word(packetBuffer[40], packetBuffer[41]);
unsigned long lowWord = word(packetBuffer[42], packetBuffer[43]);
unsigned long secsSince1900 = highWord << 16 | lowWord;
const unsigned long seventyYears = 2208988800UL;
unsigned long epoch = secsSince1900 - seventyYears;
Serial.write((byte)255);
Serial.write((byte)1);
binaryUInt data;
data.ui = epoch;
Serial.write((char*)data.bin, 4);
}
}
}
String process(const String& arg)
{
Serial.write((byte)255);
Serial.write((byte)0);
binaryUInt data;
Serial.read((char*)data.bin, 4);
unsigned long epoch = data.ui;
binaryFloat dataf;
Serial.read((char*)dataf.bin, 4);
float tmp = dataf.fp;
Serial.read((char*)dataf.bin, 4);
float hum = dataf.fp;
if (arg == "TIMESTAMP") {
char sz[256];
sprintf(sz, "%d:%02d:%02d", (epoch % 86400L) / 3600,
(epoch % 3600) / 60,
epoch % 60
);
return String(sz);
} else if (arg == "TEMPERATURE") {
char sz[256];
sprintf(sz, "%f",tmp);
return String(sz);
} else if(arg=="HUMIDITY") {
char sz[256];
sprintf(sz, "%f",hum);
return String(sz);
}
return String();
}
void sendNTPpacket(IPAddress& address) {
Serial.println("sending NTP packet...");
memset(packetBuffer, 0, NTP_PACKET_SIZE);
packetBuffer[0] = 0b11100011; packetBuffer[1] = 0; packetBuffer[2] = 6; packetBuffer[3] = 0xEC; packetBuffer[12] = 49;
packetBuffer[13] = 0x4E;
packetBuffer[14] = 49;
packetBuffer[15] = 52;
udp.beginPacket(address, 123); udp.write(packetBuffer, NTP_PACKET_SIZE);
udp.endPacket();
}
Remember to set your dips such that 1-4 are OFF, 5-7 are ON, and 8 is OFF. Select the "Generic ESP8266" from the boards menu before flashing the code. After flashing the code, make sure you upload the data directory to your flash memory as well. When you're done flashing, set the dips back such that 1-4 are ON and 5-8 are OFF.
Note that our "webservice" above is just a templated JSON file:
{
"time": "%TIMESTAMP%",
"temperature": %TEMPERATURE%,
"humidity": %HUMIDITY%
}
Those % delimited values are resolved using the process()
routine from above. It's a shameless way to make a dynamic webservice, but it works, and keeps our poor little CPU from having to do a bunch of JSON string concatenations.
Those get fetched using the following HTML and JavaScript. Please excuse my mess:
<!DOCTYPE html>
<html>
<head>
<title>Clock</title>
</head>
<body>
<span>Time:</span><span id="TIME">Loading</span><br />
<span>Temp:</span><span id="TEMPERATURE">Loading</span><br />
<span>Humidity:</span><span id="HUMIDITY">Loading</span>
<script>
function askTime() {
var xmlhttp = new XMLHttpRequest();
xmlhttp.onreadystatechange = function() {
if (this.readyState == 4 && (this.status==0 || this.status == 200)) {
var obj = JSON.parse(this.responseText);
document.getElementById("TIME").innerHTML = obj['time'];
var far = (obj['temperature'] * 1.8) + 32;
document.getElementById("TEMPERATURE").innerHTML =
Math.round(obj['temperature']) +
'C/'+
Math.round(far)+
'F';
document.getElementById("HUMIDITY").innerHTML =
Math.round(obj['humidity']);
}
};
xmlhttp.open("GET", "time.json", true);
xmlhttp.send();
setTimeout(function() {askTime();},1000);
}
askTime();
</script>
</body>
</html>
Nothing to see here, it's just the classic JSON based AJAX technique stripped of every fancy JS framework and down to the metal.
History
- 10th November, 2020 - Initial submission