Getting video stream from USB web-camera on Arduino Due - Part 7: Completing enumeration process

Introduction

Part 6 is here.

So far I have all necessary information to finish USB device enumeration. For all USB devices enumeration process consists of setting device address, configuration and interface alternative setting. Video class devices include additional step called negotiation during which several control transfers are done: setting and getting video probe controls and committing controls before setting alternative setting.

OUT control transfers with and without data stage are introduced in this article too.

Setting address

Before this moment all transfers were IN transfers, meaning from device to host. On the other hand setting address transfer is OUT transfer: host sends address to device, device uses it. This transfer is interesting because it doesn't have a data stage, SETUP transaction (the only transaction in SETUP stage) includes address number in itself:

As can be seen, no data stage is present because DATA0 packet of SETUP stage contains all necessary information for Set Address transfer. Device must respond with empty DATA packet in HANDSHAKE stage upon host IN packet.

Function that starts SETUP packet of SETUP transaction and has data that goes into DATA0 packet looks like this:

void USBD_SetAddressBegin()
{
    usb_setup_req_t ControlRequest;
    
    ControlRequest.bmRequestType = USB_REQ_DIR_OUT | USB_REQ_TYPE_STANDARD | USB_REQ_RECIP_DEVICE;
    ControlRequest.bRequest = USB_REQ_SET_ADDRESS;
    ControlRequest.wValue = DeviceAddress;
    ControlRequest.wIndex = 0;
    ControlRequest.wLength = 0;
    
    HCD_SendCtrlSetupPacket(0, USB_REQ_DIR_OUT, ControlRequest, NULL, 0, USBD_SetAddressEnd);
}

DeviceAddress is defined as 0x05 (you can specify any number from 1 to 127). wIndex and wLength must be zero and because there is no data stage, parameter ptrBuffer is NULL and TransferBufferSize is zero in HCD_SendCtrlSetupPacket function call.

Such requests are described in [2, p.256].

This function is called immediately after full configuration is shown (see previous article):

void USBD_GetConfigurationDescriptorEnd(uint16_t ByteReceived)
{
    uint16_t TotalSize;
    
    if(4 == ByteReceived)
    {
        TotalSize = (Buffer[3] << 8);
        TotalSize |= Buffer[2];
        
        PrintStr("Configuration Total size is ");
        PrintDEC(TotalSize);
        PrintStr(".\r\n");
        
        PrintStr("\r\nGETTIN FULL CONFIGURATION.\r\n");
        USBD_GetConfigurationDescriptorBegin(TotalSize);
    }
    else
    {
        PrintStr("Full configuration has been obtained.\r\n");
        USBD_ParseFullConfigurationDescriptor(Buffer, ByteReceived);
        
        PrintStr("Setting device address.\r\n");
        USBD_SetAddressBegin();
    }
}

Because no OUT transfers have been done before, interrupt handler will not process it correctly and therefore it has to be altered:

//...

//Start DATA stage    
if(hcd_ControlStructure.ControlRequestStage == USB_CTRL_REQ_STAGE_SETUP)
{
    if(hcd_ControlStructure.Direction == USB_REQ_DIR_IN)    //Now need to send IN packet
    {
        //Change to IN phase
        hcd_ControlStructure.ControlRequestStage = USB_CTRL_REQ_STAGE_DATA_IN;
        HCD_SendCtrlInPacket();
        return;
    }
    if(hcd_ControlStructure.Direction == USB_REQ_DIR_OUT)
    {
        if(hcd_ControlStructure.ByteCount == 0)             //Empty data-stage
        {
            hcd_ControlStructure.ControlRequestStage = USB_CTRL_REQ_STAGE_ZLP_IN;
            HCD_SendCtrlInPacket();
            return;
        }
    }
}

//...

After SETUP packet has been sent, "SETUP sent interrupt" leads us to this piece of code inside HCD_HandleControlPipeInterrupt function. Bold is what I added to handle data-stageless OUT transfer. If transfer is OUT, it checks quantity of bytes to be sent, and if it is zero (which is true in this case), it specifies that next stage is HANDSHAKE and empty IN transaction must occur. HCD_SendCtrlInPacket function sends IN packet which in turn initiates HANDSHAKE stage, this function just sends IN token and was describes in article 5.

After IN token of HANDSAHE stage has been sent, device responds with empty DATA packet and interrupt handler must recognize that condition:

//...

//Received IN Data interrupt
if(UOTGHS->UOTGHS_HSTPIPISR[0] & UOTGHS_HSTPIPISR_RXINI)    
{
    UOTGHS->UOTGHS_HSTPIPICR[0] = UOTGHS_HSTPIPICR_RXINIC;        //Ack interrupt    
       
    //Data stage: IN token has been send and DATA0/1 came in    
    if(hcd_ControlStructure.ControlRequestStage == USB_CTRL_REQ_STAGE_DATA_IN)
    {
        HCD_GetCtrlInDataPacket();                                //Getting data
        return;
    }
    if(hcd_ControlStructure.ControlRequestStage == USB_CTRL_REQ_STAGE_ZLP_IN)
    {
        (*hcd_ControlStructure.TransferEnd)(0);                   //Notifies about completion
        return;
    }        
    return;
}

//...   

If handler receives IN packet from device and stage is ZLP_IN (zero length packet IN), means HANDSHAKE stage is completed and caller can be notified about transfer completion. TransferEnd function pointer was specified in USBD_SetAddressBegin as USBD_SetAddressEnd function:

void USBD_SetAddressBegin()
{
    usb_setup_req_t ControlRequest;
    
    ControlRequest.bmRequestType = USB_REQ_DIR_OUT | USB_REQ_TYPE_STANDARD | USB_REQ_RECIP_DEVICE;
    ControlRequest.bRequest = USB_REQ_SET_ADDRESS;
    ControlRequest.wValue = DeviceAddress;
    ControlRequest.wIndex = 0;
    ControlRequest.wLength = 0;
    
    HCD_SendCtrlSetupPacket(0, USB_REQ_DIR_OUT, ControlRequest, NULL, 0, USBD_SetAddressEnd);
}

USBD_SetAddressEnd prints a message about completion of address setup and initiate next configuration step which is setting up configuration (see Delay function section).

Delay function

After device's address has been set, 2 ms must be given to the device as a recovery interval [2, p.246]. At the end of this interval, the device must start accepting further requests that are directed to newly assigned address.

I will create a function that handles this situation:

void HCD_StartDelayed(void (*TransferStart)(void), uint16_t Pause)
{
    hcd_ControlStructure.TransferStart = TransferStart;
    hcd_ControlStructure.Pause = Pause;
}

So the the next enumeration operation (specifying configuration) will start in 2ms after end of address setup:

void USBD_SetAddressEnd(uint16_t ByteReceived)
{
    PrintStr("Device address has been set.\r\n");
    PrintStr("Setting device configuration.\r\n");
    HCD_StartDelayed(USBD_SetConfigurationBegin, 2);
}

Note: as I use Print functions, this 2ms pause is not necessary as sending via RS232 makes even bigger delay than 2ms.

Setting configuration

USB device has one or more configurations, one of them can be active at a time. My device has only one configuration (see previous article about descriptors):

To specify a configuration, its number should be passed as a parameter. I get this number from configuration descriptor (see previous article):

So parameter is 1. Transfer that sets configuration is similar to one that sets address - they are both OUT transfers and have no data stage, thus no alteration to USB interrupt handler is required. Function that starts transfer:

void USBD_SetConfigurationBegin()
{
    usb_setup_req_t ControlRequest;
    
    ControlRequest.bmRequestType = USB_REQ_DIR_OUT | USB_REQ_TYPE_STANDARD | USB_REQ_RECIP_DEVICE;
    ControlRequest.bRequest = USB_REQ_SET_CONFIGURATION;
    ControlRequest.wValue = DeviceConfiguration;
    ControlRequest.wIndex = 0;
    ControlRequest.wLength = 0;
    
    if(HCD_InitiatePipeZero(DeviceAddress))
    {
        HCD_SendCtrlSetupPacket(DeviceAddress, USB_REQ_DIR_OUT, ControlRequest, 
                                                                NULL, 0, USBD_SetConfigurationEnd);
    }
}

DeviceConfiguration is defined as 1 in my case. Note bRequest constant and how it is different from one used in Set Address function. Pipe 0 must be reinitialized as address has been changed. Function USBD_SetConfigurationEnd will be called upon completion of Set Configuration transfer:

void USBD_SetConfigurationEnd(uint16_t ByteReceived)
{
    PrintStr("Device configuration has been set.\r\n");
    PrintStr("Setting video probe control.\r\n");    
    USBD_SetVideoProbeControlBegin();
}

This function prints some messages and starts the next enumeration step.

Stream negotiation

Before stream can be started, its parameters such as format, frame, payload size, etc. must be negotiated. Host proposes needed values and device confirms the values or lowers those it cannot support. If value is specified incorrect, device would return correct one.

UVC specification has special picture for that process:

So, first host proposes values - PROBE_CONTROL(SET_CUR), then it reads them from device - PROBE_CONTROL(GET_CUR), lastly it sets (commits) those values - COMMIT_CONTROL(SET_CUR).

SET_CUR and GET_CUR are just constants and are used the same way a USB_REQ_SET_ADDRESS or USB_REQ_SET_CONFIGURATION in bRequest field. I defined them in USB_Specification.h file:

//Class-specific UVC requests
#define  USB_REQ_SET_CUR                    0x01
#define  USB_REQ_GET_CUR                    0x81

Special structure is used for negotiation [3, p.97]:

and in C:

typedef struct
{
    uint16_t bmHint;
    uint8_t bFormatIndex;
    uint8_t bFrameIndex;
    uint32_t dwFrameInterval;
    uint16_t wKeyFrameRate;
    uint16_t wPFrameRate;
    uint16_t wCompQuality;
    uint16_t wCompQualitySize;
    uint16_t wDelay;
    uint16_t Padding;
    uint32_t dwMaxVideoFrameSize;
    uint32_t dwMaxPayloadTransferSize;
    //uint32_t dwClockFrequency;
    //uint8_t bmFramingInfo;
    //uint8_t bPreferedVersion;
    //uint8_t bMinVersion;
    //uint8_t bMaxVersion;
} uvc_video_probe_and_commit_controls_t;

Commented out fields are not present in 1.0 version of UVC specification, they started to appear from 1.1 version. Fields from wKeyFramRate to wCompQualitySize are related to compressed formats which my camera is not, so those will be zero.

OUT transfer with data stage

First action during negotiation process is to send proposed values to a device. This transfer includes sending of filled up uvc_video_probe_and_commit_controls_t structure to device inside data stage. Graphically such transfer looks as following:

and because it has not happened before, HCD interrupt handler should be altered to correctly handle OUT transactions with data stage. First, in "SETUP sent interrupts" section:

//...

if(hcd_ControlStructure.Direction == USB_REQ_DIR_OUT)
{
    if(hcd_ControlStructure.ByteCount == 0)             //Empty data-stage
    {
        hcd_ControlStructure.ControlRequestStage = USB_CTRL_REQ_STAGE_ZLP_IN;
        HCD_SendCtrlInPacket();
        return;
    }
    else                                                //Sending OUT-packet
    {
        hcd_ControlStructure.ControlRequestStage = USB_CTRL_REQ_STAGE_DATA_OUT;
        HCD_SendCtrlOutData();
    }
}

//...

When byte count is not zero means that OUT transfer has data to send and data stage is required. Transfer stage is specified and function HCD_SendCtrlOutData is called that will provide data and initiate data stage:

void HCD_SendCtrlOutData(void)
{
    PrintStr("Sending DATA0/1 to device.\r\n");
    
    uint8_t ControlPipeSize = (UOTGHS->UOTGHS_HSTPIPCFG[0] & UOTGHS_HSTPIPCFG_PSIZE_Msk) 
                                                                   >> UOTGHS_HSTPIPCFG_PSIZE_Pos;
    uint8_t volatile *ptrSendBuffer = 
                                 (uint8_t *)&(((volatile uint8_t(*)[0x8000])UOTGHS_RAM_ADDR)[0]);
    
    if (hcd_ControlStructure.Index == hcd_ControlStructure.ByteCount)
    {
        //Everything has been sent, start HANDSHAKE stage by changing direction
        hcd_ControlStructure.ControlRequestStage = USB_CTRL_REQ_STAGE_ZLP_IN;
        HCD_SendCtrlInPacket();
        return;
    }
    
    //Moving data to pipe 0 FIFO buffer
    while ((hcd_ControlStructure.Index < hcd_ControlStructure.ByteCount) && ControlPipeSize)
    {
        *ptrSendBuffer++ = hcd_ControlStructure.ptrBuffer[hcd_ControlStructure.Index];
        hcd_ControlStructure.Index++;
        ControlPipeSize--;
    }
    
    //Start OUT transaction by sending OUT packet
    HCD_SentCtrlOutPacket();
}

Firstly, it takes pipe max transfer size to not specify more data than device can handle. Then it gets pointer to pipe's FIFO buffer that should be populated with data. Then it checks if all data has been sent and if yes - starts IN packet to initiate HANDSHAKE stage. If there is data to send, it simply copies byte-by-byte to FIFO buffer and checks if end of data or maximum pipe size is reached. Once portion of data is copied - OUT transaction is initiated by sending OUT packet (SAM3X automatically handles sending DATA packet of OUT transaction and getting ACK form device).

After this function, there can be two further actions:

1. To start next OUT transaction if more data to send (stage is still DATA_OUT).

2. To catch end of transfer (HANDSHAKE stage ZLP_IN is specified here) and notify a caller about it.

Both actions are recognized in control endpoint interrupt handling method HCD_HandleControlPipeInterrupt. Once OUT transaction is finished, "OUT data interrupt" is fired that leads to same name section of the function:

//...

//Received OUT Data interrupt
if(UOTGHS->UOTGHS_HSTPIPISR[0] & UOTGHS_HSTPIPISR_TXOUTI)
{
    UOTGHS->UOTGHS_HSTPIPICR[0] = UOTGHS_HSTPIPICR_TXOUTIC;    //Ack interrupt
      
    if(hcd_ControlStructure.ControlRequestStage == USB_CTRL_REQ_STAGE_DATA_OUT)
    {
        HCD_SendCtrlOutData();                                //Sending more data
        return;
    }
        
    //HANDSHAKE stage, ACK for empty OUT is received
    if(hcd_ControlStructure.ControlRequestStage == USB_CTRL_REQ_STAGE_ZLP_OUT)
    {
        //Notifies about transaction completion
        (*hcd_ControlStructure.TransferEnd)(hcd_ControlStructure.Index);    
    }
    return;
}

//...

If stages stayed DATA_OUT, HCD_SendCtrlOutData is called again and again until there is no data to be sent.

Second action is handled in following section:

//...

//Received IN Data interrupt
if(UOTGHS->UOTGHS_HSTPIPISR[0] & UOTGHS_HSTPIPISR_RXINI)    
{
    UOTGHS->UOTGHS_HSTPIPICR[0] = UOTGHS_HSTPIPICR_RXINIC;    //Ack interrupt    
        
    //Data stage: IN token has been send and DATA0/1 came in    
    if(hcd_ControlStructure.ControlRequestStage == USB_CTRL_REQ_STAGE_DATA_IN)
    {
        HCD_GetCtrlInData();                                  //Getting data
        return;
    }
    if(hcd_ControlStructure.ControlRequestStage == USB_CTRL_REQ_STAGE_ZLP_IN)
    {
        (*hcd_ControlStructure.TransferEnd)(0);               //Notifies about completion
        return;
    }        
    return;
}

//...

A function, pointer to which is specified in TransferEnd function pointer, is called to notify about the whole transfer completion.

Proposing parameters to a device - PROBE_CONTROL(SET_CUR)

At this moment all low level (HCD) functions are done and we can carry on with enumeration process. Here is a diagram from previous article with necessary numbers:

Note that the only Format number is 1 and the lowest frame number is 5 - 160x120 pixels. Number of lowest speed alternate setting is 1 - 128 bytes per transaction.

Function that starts the transfer:

void USBD_SetVideoProbeControlBegin(void)
{
    uvc_video_probe_and_commit_controls_t Parameters;
    Parameters.bmHint = 0x0001;
    Parameters.bFormatIndex = 1;                //format #1
    Parameters.bFrameIndex = 5;                 //160x120 frame size
    Parameters.dwFrameInterval = 333333;        //in 100ns units (from frame descriptor)
    Parameters.wKeyFrameRate = 0;               //unused
    Parameters.wPFrameRate = 0;                 //unused
    Parameters.wCompQuality = 0;                //unused
    Parameters.wCompQualitySize = 0;            //unused
    Parameters.wDelay = 0;                      //set by device
    Parameters.dwMaxVideoFrameSize = 0;         //set by device
    Parameters.dwMaxPayloadTransferSize = 0;    //set by device    
    
    USBD_CopyVideoProbeControl(Buffer, &Parameters);
    PrintVideoProbeAndCommitControls(Buffer);
    
    usb_setup_req_t ControlRequest;    
    ControlRequest.bmRequestType = USB_REQ_DIR_OUT | USB_REQ_TYPE_CLASS | USB_REQ_RECIP_INTERFACE;
    ControlRequest.bRequest = USB_REQ_SET_CUR;
    ControlRequest.wValue = 0x0100;            //Probe control selector (01)
    ControlRequest.wIndex = 1;                 //Video Streaming interface number is 1
    ControlRequest.wLength = 26;               //Length of the returned block (parameter block)
    
    HCD_SendCtrlSetupPacket(DeviceAddress, USB_REQ_DIR_OUT, ControlRequest, 
                               Buffer, ControlRequest.wLength, USBD_SetVideoProbeControlEnd);
}

Note PrintVideoProbeAndCommitControls function, I call it here before sending parameters and I'll call it right after I got parameters back from a device. This is for easy comparison. Value 01 in higher byte of wValue means Probe controls, 02 - Commit controls.

I used function USBD_CopyVideoProbeControl to copy Parameters to buffer and I explain why. It took me some time to understand that I cannot simply cast Parameters like this:

(uint8_t*)&Parameters

to pass it into HCD_SendCtrlSetupPacket function. The reason is padding, members are aligned in structures and sometimes paddings are added to fill up to 4 bytes. I'll illustrate this with a picture:

so as you can see the structure actually occupies 28 (not 26) bytes and has padding (marked blue) inside because 4 bytes field dwMaxVideoFrameSize cannot be split in half in 32-bit ARM processor memory. That is why I cannot just cast and pass a variable of this structure as it inserts those 2 un-useful bytes. Therefore I created special function to make this copy operation:

void USBD_CopyVideoProbeControl(uint8_t *ptrBuffer, uint8_t *ptrParameters)
{
    for(uint8_t i = 0; i < 18; i++)
        ptrBuffer[i] = ptrParameters[i];
    
    for(uint8_t i = 18; i < 26; i++)
        ptrBuffer[i] = ptrParameters[i+2];
}

Function USBD_SetVideoProbeControlEnd is called upon this transfer completion:

void USBD_SetVideoProbeControlEnd(uint16_t ByteReceived)
{
    PrintStr("Proposed parameter values have been sent.");
    PrintStr("\r\nGETTING VIDEO PROBE CONTROL.\r\n");
    USBD_GetVideoProbeControlBegin();
}

It prints some debug messages and initiates next enumeration step.

Device answer on proposed parameters - PROBE_CONTROL(GET_CUR)

Following function starts the transfer that gets the answer from a device:

void USBD_GetVideoProbeControlBegin(void)
{
    usb_setup_req_t ControlRequest;
    
    ControlRequest.bmRequestType = USB_REQ_DIR_IN | USB_REQ_TYPE_CLASS | USB_REQ_RECIP_INTERFACE;
    ControlRequest.bRequest = USB_REQ_GET_CUR;
    ControlRequest.wValue = 0x0100;            //Probe control selector (01)
    ControlRequest.wIndex = 1;                 //Video Streaming interface number is 1
    ControlRequest.wLength = 26;               //Length of the returned block (parameter block)
    
    HCD_SendCtrlSetupPacket(DeviceAddress, USB_REQ_DIR_IN, ControlRequest, Buffer, 
                                           ControlRequest.wLength, USBD_GetVideoProbeControlEnd);
}

bRequest value is now GET_GUR and direction is IN.

USBD_GetVideoProbeControlEnd gets the answer, prints it and initiate next enumeration step:

void USBD_GetVideoProbeControlEnd(uint16_t ByteReceived)
{
    PrintStr("\r\nDEVICE ANSWER:\r\n");
    PrintVideoProbeAndCommitControls(Buffer);    
    
    PrintStr("COMMITTING VIDEO PROBE CONTROL.\r\n");    
    USBD_SetVideoCommitControlBegin();
}

Program output of this and previous enumeration steps:

Device changed two last fields only. Frame size can be checked easily as I already know that YUY format has 2 bytes per pixel and frame size is 160x120: 160*120*2 = 38400 bytes. Max transfer size is much bigger than I'm going to use.

Setting stream parameters

Once host and device negotiated, agreed parameters can be set:

void USBD_SetVideoCommitControlBegin(void)
{
    usb_setup_req_t ControlRequest;
    
    ControlRequest.bmRequestType = USB_REQ_DIR_OUT | USB_REQ_TYPE_CLASS | USB_REQ_RECIP_INTERFACE;
    ControlRequest.bRequest = USB_REQ_SET_CUR;
    ControlRequest.wValue = 0x0200;            //Commit control selector (02)
    ControlRequest.wIndex = 1;                 //Video Streaming interface number is 1
    ControlRequest.wLength = 26;               //Length of the returned block (parameter block)
    
    HCD_SendCtrlSetupPacket(DeviceAddress, USB_REQ_DIR_OUT, ControlRequest, 
                                    Buffer, ControlRequest.wLength, USBD_SetVideoCommitControlEnd);
}

bRequest is SET_CUR again but wValue's high byte is 0x02 which indicates COMMIT (not PROBE as before), Buffer still holds response from previous step.

USBD_SetVideoCommitControlEnd is called upon this transfer completion:

void USBD_SetVideoCommitControlEnd(uint16_t ByteReceived)
{
    PrintStr("Video Commit Control has been set.\r\n");
    
    PrintStr("SELECTING ALTERNATE SETTING.\r\n");    
    USBD_SetInterfaceAlternativeSettingBegin();
}

This function prints debug messages and initiates next enumeration step.

Starting the stream

In previous article we saw that there was no endpoint right under streaming interface descriptor, but there were under streaming interface's alternate setting descriptors. So, the last thing to do is to select alternate setting with appropriate endpoint data size and as I said before I would start with lowest #1 that has 126 bytes data size:

void USBD_SetInterfaceAlternativeSettingBegin(void)
{
    usb_setup_req_t ControlRequest;
    
    ControlRequest.bmRequestType = USB_REQ_DIR_OUT |USB_REQ_TYPE_STANDARD |USB_REQ_RECIP_INTERFACE;
    ControlRequest.bRequest = USB_REQ_SET_INTERFACE;
    ControlRequest.wValue = 1;                //Alternative setting #1: 128 bytes endpoint
    ControlRequest.wIndex = 1;                //Video Streaming interface number is #1
    ControlRequest.wLength = 0;               //No data
    
    HCD_SendCtrlSetupPacket(DeviceAddress, USB_REQ_DIR_OUT, ControlRequest, 
                                                  NULL, 0, USBD_SetInterfaceAlternateSettingEnd);
}

bRequest is SET_INTERFACE as this request is directed to interface. wValue holds alternate setting number, wIndex is number of interface - 0 is Video Control, 1 is Video Streaming. No data stage is in this transfer (wLength is zero).

USBD_SetInterfaceAlternateSettingEnd is called upon this transfer completion:

void USBD_SetInterfaceAlternateSettingEnd(uint16_t ByteReceived)
{
    PrintStr("Alternate setting has been set.\r\n");
    
    PrintStr("PROCESSING THE STREAM.\r\n");
}

Here stream processing will be initiated (in the next article).

Conclusion

I showed how to enumerate (initialize) USB web-camera. The steps discussed here fit any USB camera and give you a good understanding of what commands (requests) to use, a sequence and how to apply parameters that were obtained from device's descriptors.

At this stage stream is ready, the camera produces video and if I start sending IN packets to streaming endpoint (which is #2 in my case), I'll get 128 bytes DATAx packets with video data. Video format (Uncompressed in my case) specification helps to understand how to handle incoming data because not all of it is video data, there is meta-data too that helps putting pieces of a video frame together. All that will be explained in the next article and I'll try to make/upload a video that demonstrates the end result. Also I'll try to make stream in at least two frame sizes as Arduino Due processing power allows it.

Source code is here.

Part 8 is here.