Uncompressed 32bit 44.1kHz audio transmission wirelessly is not possible from Arduino MKR WiFi 1010 with existing NINA firmware and WiFiNINA library due to latency introduced by its communication protocol. This article describes minor modifications to firmware enabling high data-rate UDP transmissions.
Introduction
Arduino MKR WiFi 1010 is a powerful IoT device with both an ARM Cortex-M0 (SAMD21) and an ESP32 (NINA-W102) processor on a single compact board. Libraries provided for WiFi and Bluetooth connectivity make it easy to create IoT nodes communicating in IP protocols and as Bluetooth clients. One other thing that might catch our attention is the dual channel I2S port and corresponding libraries for reading out audio from the port. The I2S port can be used to read 32/24 bit audio at high sample rates of up-to 192kHz from supported I2S microphone devices.
A brick wall one will hit when trying to transmit this high data-rate I2S audio wirelessly using the WiFiNINA library, is the latency introduced by the SPI based command/response protocol between SAMD21 and NINA-W102. This limits the rate at which data can be sent to any remote port. Compressing the audio data is an option, but the audio will lose the fidelity that I2S offers.
This article describes modifications to Arduino NINA-W102 firmware which is based on ESP-IDF to enable sending uncompressed 44.1kHz 32bit audio via UDP packets to any listening IP Address and port. Instructions for compiling and flashing the modified firmware to NINA-W102 are also provided. Corresponding MKR WiFi 1010 code is also described. Link for precompiled modified firmware is provided for readers who do not want to compile it themselves.
Cloning and Building Original Firmware
The git repository for original NINA-W102 firmware is located here, along with instructions to build it. Once make command is successful, the built firmware will be located in ‘\build’ folder. Windows users will have easier time building the project using Ubuntu running on Windows Subsystem for Linux. Do not forget to add to PATH the tool-chain using the below example commands.
export PATH=$PATH:/mnt/d/LinuxFiles/esp/xtensa-esp32-elf/bin
export IDF_PATH=/mnt/d/LinuxFiles/GIT/esp-idf
make
In the last line of make command output, if successful, an example of flash command is displayed, and it looks like this:
python.exe D:\LinuxFiles\GIT\esp-idf\components\esptool_py\esptool\esptool.py
--chip esp32
--port COM5 --baud 115200 --before default_reset --after hard_reset write_flash -z
--flash_mode dio --flash_freq 40m --flash_size detect 0x1000 bootloader/bootloader.bin
0xf000 phy_init_data.bin 0x30000 nina-fw.bin 0x8000 partitions.bin
For this flash command to work, there exists a SerialNINAPassthrough sketch which as its name suggests, when uploaded to MKR WiFi 1010, will pass serial data directly from MKR WiFi 1010 Serial COM Port to NINA-W102 to enable flashing with esptool.py.
After esptool.py flash command is successful, regular bootloader mode of MKR WiFi 1010 can be re-activated by pressing the reset button twice quickly. Bootloader mode will enable Arduino sketches targeting MKR WiFi 1010 to be uploaded, as SerialNINAPassthrough sketch on the device will prevent that. Testing WiFiNINA library again with this newly flashed firmware is recommended. Once this process is in place, firmware modification can begin.
Description of Original Firmware
Entry point for NINA-W102 firmware is located at main/sketch.ino.cpp and it has setup() and loop() functions like any Arduino sketch. In the loop() function of firmware, protocol used for communication between SAMD21 and NINA-W102 is implemented. SPIS.transfer() function handles the SPI data sent from from SAMD21, making it available in commandBuffer as an uint8_t array, and CommandHandler.handle() processes this commandBuffer and in-turn uses native esp libraries to perform requested command, after which it creates an uint8_t array response which will be sent back to SAMD21 via the same SPIS.transfer() function.
void loop() {
memset(commandBuffer, 0x00, SPI_BUFFER_LEN);
int commandLength = SPIS.transfer(NULL, commandBuffer, SPI_BUFFER_LEN);
if (commandLength == 0) {
return;
}
if (debug) {
dumpBuffer("COMMAND", commandBuffer, commandLength);
}
memset(responseBuffer, 0x00, SPI_BUFFER_LEN);
int responseLength = CommandHandler.handle(commandBuffer, responseBuffer);
SPIS.transfer(responseBuffer, NULL, responseLength);
if (debug) {
dumpBuffer("RESPONSE", responseBuffer, responseLength);
}
}
Going through main/CommandHandler.cpp and the function CommandHandlerClass::handle(), reader can understand how CommandHandler object translates the commandBuffer to some action performed using esp libraries, and how responseBuffer is updated.
Sending UDP Packets with Original Firmware
To send a UDP packet to an IP Address and port, MKR WiFi 1010 should be connected to a WiFi network. This can be done as specified in the below example code in the setup() function.
#include <WiFiNINA.h>
...
void setup() {
...
while (status != WL_CONNECTED) {
Serial.print("Attempting to connect to SSID: ");
Serial.println(ssid);
status = WiFi.begin("MySSID", "SomePassword");
delay(5000);
}
...
}
...
After successful connection to a network, and after including, the below example code sends a packet of byte array to a port on remote listener.
#include <WiFiUdp.h>
...
WiFiUDP Udp;
...
void loop() {
...
uint8_t packetData[512];
uint32_t packetLength = 512;
Udp.beginPacket("192.168.1.102", 8002);
Udp.write(packetData, packetLength);
Udp.endPacket();
...
}
Issue with Sending UDP Packets using WiFiUdp Class
It might not be apparent to the reader yet the problem with using WiFiUdp class to send UDP packets as described above. beginPacket(), write() and endPacket() are all translated into SPI commands and responses, hence these three functions running on SAMD21 have to send corresponding command via SPI to NINA-W102, and then wait for a response from NINA-W102 via SPI to proceed further. This introduces latency and there will be an upper limit on the rate at which UDP packets can be sent.
The goal of this article is to send I2S data at a sampling rate of 44.1kHz and 32bits per sample to a UDP listener, hence the bottleneck involved will crash the program attempting to send UDP packets at such high data rate. This can be resolved with a minor firmware modification described here.
Reading I2S Audio Data in MKR WiFi 1010
I2S library for MKR WiFi 1010 can be included as #include <I2S.h>, and can be used initialize an I2S device to a supported sampling rate and bits per sample, and to read streaming data. The below diagram shows how to connect an example I2S device to MKR WiFi 1010.
To initialize an I2S device to 44.1kHz sampling rate and 32 bits per sample, the following lines can be added to setup() function.
#include <I2S.h>
...
void setup() {
...
if (!I2S.begin(I2S_PHILIPS_MODE, 44100, 32)) {
Serial.println("Failed to initialize I2S!");
while (1);
}
...
}
...
In the loop() function, the below lines will continuously make available the audio data for further processing or directly relaying.
#include <I2S.h>
...
void loop() {
...
int size = I2S.available();
byte I2SData[size];
I2S.read(I2SData, size);
if(size == 0){
return;
}
...
}
...
The data which is available as I2SData in every loop, can now be compressed and sent to any listening UDP port, or can be sent uncompressed after described firmware modification.
Description of Firmware Modification
Retaining all of original firmware features, for example, the functions for connecting to a WiFi network is preferable, as the easy to use WiFiNINA.h library can still be used for those workflows. After MKR WiFi 1010 connects to a WiFi network, command/response SPI protocol can be bypassed conditionally to send UDP packets more rapidly.
A new SPI command/response protocol command is introduced to set target IP Address and Port to which UDP packets containing audio data will be sent, and this target IP Address and Port will be remembered on custom firmware running on NINA-W102. This will be the final command after which SPI transfers can work in one way mode to send audio data directly to NINA-W102 and relayed as UDP packets. As there will be no waiting for response from SPI bus, latency is removed, enabling high data rate packet transfer to a UDP listening port on the network.
Custom Command for Setting the IP Address and Port of UDP Listener
An 8-byte command is chosen to accommodate an IP Address (4 bytes), a port (2 bytes) and unique start and end identifiers (2 bytes).
When the code described below, to be added to the loop() of the NINA-W102 firmware, detects an 8 byte command with the unique start and end identifiers, the IP Address and Port can be extracted and remembered.
...
char sendToIpAddress[50];
uint16_t sendToPort = 0;
...
void loop() {
memset(commandBuffer, 0x00, SPI_BUFFER_LEN);
int commandLength = SPIS.transfer(NULL, commandBuffer, SPI_BUFFER_LEN);
if (commandLength == 0) {
return;
}
if (commandLength == 8 && commandBuffer[0] == 0xD1 && commandBuffer[7] == 0xD2) {
sprintf(&sendToIpAddress[0], "%d.%d.%d.%d",
commandBuffer[1], commandBuffer[2] , commandBuffer[3], commandBuffer[4]);
sendToPort = 0;
sendToPort = sendToPort^commandBuffer[5];
sendToPort = sendToPort^(commandBuffer[6]<<8);
memset(responseBuffer, 0x00, SPI_BUFFER_LEN);
responseBuffer[0] = 'A';
responseBuffer[1] = 'B';
SPIS.transfer(responseBuffer, NULL, 2);
return;
}
...
}
...
And the way to send this new custom command from MKR WiFi 1010 code is given in a later section. Once firmware knows to which endpoint to send the UDP data packets, flags are set and can passively receive data through SPI and relay it.
Sending UDP Packets Directly
The firmware for NINA-W102 has a class WiFiUDP available after including #include<WiFiUdp.h> which can be used to send UDP packets to any IP Address and Port. This can be done using the following lines of code:
...
#include "CommandHandler.h"
#include <WiFiUdp.h>
...
WiFiUDP udp;
...
void loop() {
udp.beginPacket(sendToIpAddress, sendToPort);
udp.write(commandBuffer, commandLength);
udp.endPacket();
}
...
These three functions, as they are running directly on NINA-W102 will not have any unnecessary overhead.
Sending the Received Audio Data
The audio data sent via SPI from SAMD21 will be 256 bytes or 512 bytes long, and will be available in the commandBuffer like any other commands, but instead of representing a command, it will contain the audio data. This data, once received can be directly relayed in loop() of the modified firmware. This can be accomplished using the code below:
...
#include "CommandHandler.h"
#include <WiFiUdp.h>
...
WiFiUDP udp;
...
char sendToIpAddress[50];
uint16_t sendToPort = 0;
...
void loop() {
memset(commandBuffer, 0x00, SPI_BUFFER_LEN);
int commandLength = SPIS.transfer(NULL, commandBuffer, SPI_BUFFER_LEN);
if (commandLength == 0) {
return;
}
...
if (WiFi.status() == WL_CONNECTED && sendToPort != 0 && commandLength > 128) {
udp.beginPacket(sendToIpAddress, sendToPort);
udp.write(commandBuffer, commandLength);
udp.endPacket();
return;
}
...
}
...
Complete Firmware Modification Code
The complete snapshot of the modifications described above are given below:
...
#include "CommandHandler.h"
#include <WiFiUdp.h>
...
WiFiUDP udp;
...
char sendToIpAddress[50];
uint16_t sendToPort = 0;
...
void loop() {
memset(commandBuffer, 0x00, SPI_BUFFER_LEN);
int commandLength = SPIS.transfer(NULL, commandBuffer, SPI_BUFFER_LEN);
if (commandLength == 0) {
return;
}
if (commandLength == 8 && commandBuffer[0] == 0xD1 && commandBuffer[7] == 0xD2) {
sprintf(&sendToIpAddress[0], "%d.%d.%d.%d",
commandBuffer[1], commandBuffer[2] , commandBuffer[3], commandBuffer[4]);
sendToPort = 0;
sendToPort = sendToPort^commandBuffer[5];
sendToPort = sendToPort^(commandBuffer[6]<<8);
memset(responseBuffer, 0x00, SPI_BUFFER_LEN);
responseBuffer[0] = 'A';
responseBuffer[1] = 'B';
SPIS.transfer(responseBuffer, NULL, 2);
return;
}
...
if (WiFi.status() == WL_CONNECTED && sendToPort != 0 && commandLength > 128) {
udp.beginPacket(sendToIpAddress, sendToPort);
udp.write(commandBuffer, commandLength);
udp.endPacket();
return;
}
...
}
...
After these changes to firmware are done, complied and uploaded to NINA-W102, code for MKR WiFi 1010 can be written to send I2S data, to be streamed as UDP packets.
MKR WiFi 1010 Code to Send Audio Packets
To send or receive any arbitrary data array via SPI to NINA-W102, direct access to the SPI bus is needed. This direct access can be accomplished by including #include<utility/spi_drv.h>, which makes available SpiDrv class.
Sending and Receiving Data Directly using SpiDrv
The below code gives an example of how to send bytes via SPI to NINA-W102.
...
#include <utility/spi_drv.h>
#include <WiFiNINA.h>
...
void loop() {
...
byte spiData[4];
WAIT_FOR_SLAVE_SELECT();
SpiDrv::spiTransfer(spiData[0]);
SpiDrv::spiTransfer(spiData[1]);
SpiDrv::spiTransfer(spiData[2]);
SpiDrv::spiTransfer(spiData[3]);
SpiDrv::spiSlaveDeselect();
...
}
...
The below code gives an example of how to send bytes and read some response bytes from NINA-W102.
...
#include <utility/spi_drv.h>
#include <WiFiNINA.h>
...
void loop() {
...
byte spiData[1];
WAIT_FOR_SLAVE_SELECT();
SpiDrv::spiTransfer(spiData[0]);
SpiDrv::spiSlaveDeselect();
SpiDrv::waitForSlaveReady();
SpiDrv::spiSlaveSelect();
char responseByte1 = SpiDrv::spiTransfer(DUMMY_DATA);
char responseByte2 = SpiDrv::spiTransfer(DUMMY_DATA);
SpiDrv::spiSlaveDeselect();
...
}
...
Sending Custom Command for Setting Target IP Address and Port
As explained in a previous section of this article, modified firmware is primed up to detect a command 8 bytes long containing IP Address and Port. The function given below sends a command 8 bytes long and wait for a response two bytes long of specific values.
void setIPAddressPortCommand() {
WAIT_FOR_SLAVE_SELECT();
SpiDrv::spiTransfer(0xD1);
sendIPAddress(LISTENER_IP);
sendPort(LISTENER_PORT);
SpiDrv::spiTransfer(0xD2);
SpiDrv::spiSlaveDeselect();
SpiDrv::waitForSlaveReady();
SpiDrv::spiSlaveSelect();
char responseByte1 = SpiDrv::spiTransfer(DUMMY_DATA);
char responseByte2 = SpiDrv::spiTransfer(DUMMY_DATA);
SpiDrv::spiSlaveDeselect();
if (responseByte1 == 'A' && responseByte2 == 'B') {
Serial.print("Have set listener ipaddress and port: ");
Serial.print(LISTENER_IP);
Serial.print(":");
Serial.print(LISTENER_PORT);
Serial.println();
return;
}
while (true) {
Serial.print("Listener ipaddress and port not set, aborted!");
delay(2000);
}
}
The received response will help confirm if flashed custom firmware is working as intended, if not, recompiling and re-flashing firmware maybe required. Once this command is executed properly, firmware will be ready to receive and relay any arbitrary SPI data sent from SAMD21 to any IP Address and Port set by this custom command.
Sending Arbitrary Int32 Array Via SPI
The function below will send any arbitrary int32_t buffer via SPI and does not wait for any response.
void sendIntValue(int value) {
byte a,b,c,d;
a=(value&0xFF);
b=((value>>8)&0xFF);
c=((value>>16)&0xFF);
d=((value>>24)&0xFF);
SpiDrv::spiTransfer(a);
SpiDrv::spiTransfer(b);
SpiDrv::spiTransfer(c);
SpiDrv::spiTransfer(d);
}
void sendBuffer(int32_t* buffer, int length) {
WAIT_FOR_SLAVE_SELECT();
for(int i=0; i<length; i++) {
int value = buffer[i];
if(value != 0) {
value = value ^ CYPHER_KEY;
sendIntValue(value);
}
}
SpiDrv::spiSlaveDeselect();
}
At this point, it is better to get a UDP listener client on the IP Address and Port ready, to verify data being sent and received.
Sending I2S Audio Data Byte Array Via SPI
As the final step building up to this point, the below code explains how to send I2S data via SPI to NINA-W102 to be relayed without latency to a UDP port.
...
#include<utility/spi_drv.h>
#include<WiFiNINA.h>
#include<I2S.h>
...
void setup() {
...
if (!I2S.begin(I2S_PHILIPS_MODE, 44100, 32)) {
Serial.println("Failed to initialize I2S!");
while (1);
}
...
}
...
byte i2sDataBuffer[1024];
void loop() {
int size = I2S.available();
I2S.read(i2sDataBuffer, size);
if (size < 4) {
return;
}
sendBuffer((int32_t*)i2sDataBuffer, size/4);
}
...
Complete Example of MKR WiFi 1010 Code
Below is the complete snapshot where everything is brought together.
...
#include<utility/spi_drv.h>
#include<WiFiNINA.h>
#include<I2S.h>
...
void sendIPAddress(String ipAddress) {
IPAddress ip;
ip.fromString(ipAddress);
SpiDrv::spiTransfer(ip[0]);
SpiDrv::spiTransfer(ip[1]);
SpiDrv::spiTransfer(ip[2]);
SpiDrv::spiTransfer(ip[3]);
}
void sendPort(uint16_t port) {
byte a=(port&0xFF);
byte b=((port>>8)&0xFF);
SpiDrv::spiTransfer(a);
SpiDrv::spiTransfer(b);
}
void setIPAddressPortCommand() {
WAIT_FOR_SLAVE_SELECT();
SpiDrv::spiTransfer(0xD1);
sendIPAddress(LISTENER_IP);
sendPort(LISTENER_PORT);
SpiDrv::spiTransfer(0xD2);
SpiDrv::spiSlaveDeselect();
SpiDrv::waitForSlaveReady();
SpiDrv::spiSlaveSelect();
char responseByte1 = SpiDrv::spiTransfer(DUMMY_DATA);
char responseByte2 = SpiDrv::spiTransfer(DUMMY_DATA);
SpiDrv::spiSlaveDeselect();
if (responseByte1 == 'A' && responseByte2 == 'B') {
Serial.print("Have set listener ipaddress and port: ");
Serial.print(LISTENER_IP);
Serial.print(":");
Serial.print(LISTENER_PORT);
Serial.println();
return;
}
while (true) {
Serial.print("Listener ipaddress and port not set, aborted!");
delay(2000);
}
}
...
void setup() {
...
setIPAddressPortCommand();
if (!I2S.begin(I2S_PHILIPS_MODE, 44100, 32)) {
Serial.println("Failed to initialize I2S!");
while (1);
}
...
}
...
void sendIntValue(int value) {
byte a,b,c,d;
a=(value&0xFF);
b=((value>>8)&0xFF);
c=((value>>16)&0xFF);
d=((value>>24)&0xFF);
SpiDrv::spiTransfer(a);
SpiDrv::spiTransfer(b);
SpiDrv::spiTransfer(c);
SpiDrv::spiTransfer(d);
}
void sendBuffer(int32_t* buffer, int length) {
WAIT_FOR_SLAVE_SELECT();
for(int i=0; i<length; i++) {
int value = buffer[i];
if(value != 0) {
value = value ^ CYPHER_KEY;
sendIntValue(value);
}
}
SpiDrv::spiSlaveDeselect();
}
...
byte i2sDataBuffer[1024];
void loop() {
int size = I2S.available();
I2S.read(i2sDataBuffer, size);
if (size < 4) {
return;
}
sendBuffer((int32_t*)i2sDataBuffer, size/4);
}
...
Client Programs To Receive, Decode and Play Audio
An example for decoding and playing the audio transmitted in the explained manner is out of scope for this article, but do watch out for another separate article for such a client in Windows, made with NAudio soon.
Applications
As the audio transmitted in the above manner will be free from distortions, compression losses and will maintain high fidelity, further processing such as triangulation of noise sources and ambient audio surveillance can be done, if not just for fun!
The drawback of using MKR WiFi 1010 and a Wireless router in described method for any possible applications is that the router needs to be in close proximity to MKR WiFi 1010, to avoid UDP packet loss, which leads to loss of signal quality.
Links
The custom firmware and corresponding MKR WiFi 1010 sketch are available in the git repos here and here respectively. Pre-compiled custom firmware binaries are attached here to flash onto NINA-W102 using esptool.py without needing to compile the custom firmware yourself.
History
- 17th October, 2022: Initial version
