I need a queue with variable size items without massive heap frag

See more:

C++

I have a bit of embedded code. It sits and monitors for incoming serial data on a thread.

It then decodes that serial data into data structures that it puts in a queue for retrieval by another thread.

This is so serial data does not overwhelm the application. It's quite important because the application can run on devices with slow screens, even e-ink.

This works flawlessly in my current rendition of my PC GPU/CPU monitor widget.

However, I want to make a version where I can configure what's displayed on the screen.

There can be zero or more items on the screen, and the user can change what those items are - basically on the fly.

This creates an issue where I need to have variable sized data in a queue, or break up the data into fixed length structures (might be possible) and use multiple queue entries for a screen.

I could just store pointers in a queue, but then I have to malloc and free quite a bit, and I don't want to have to test how many days that will run without crashing due to heap frag.

I'm not necessarily looking for code at all, and i didn't post much code here. I'm looking for ideas on how to structure this data.

Several of these (a variable amount selected by the user) make up one screen.

typedef struct screen_packet {
    char format[16]; // the format string
    float value_max; // max value
    uint8_t icon[16*16*2]; // icon bitmap
    uint16_t colors[2]; // the gradient colors
    bool hsv_color; // true if colors are HSV
    float value; // the current value
} screen_packet_t;

This only works for fixed length structures, but I basically need something like this - replace read_status_t with screen_packet_t here because that's what I am getting at.

//Waiting for UART event.
if (xQueueReceive(serial_queue, (void *)&event, portMAX_DELAY)) {
    switch (event.type) {
    //Event of UART receving data
    /*We'd better handler data event fast, there would be much more data events than
    other types of events. If we take too much time on data event, the queue might
    be full.*/
    case UART_DATA:
        p=dtmp+size; 
        s = uart_read_bytes(UART_NUM_0, p, event.size, portMAX_DELAY);
        if(s>0) {
            size+=s;
            //ESP_LOGI(TAG, "[SERIAL DATA RECEIVED]");
            //puts("[Serial data received]");
            if(size>=sizeof(read_status_t)) {
                memcpy(&status,p,sizeof(read_status_t));
                if(size>sizeof(read_status_t)) {
                    memmove(dtmp,dtmp+sizeof(read_status_t),size-sizeof(read_status_t));
                    size-=sizeof(read_status_t);
                    --size;
                }
                xQueueSend(serial_out_queue,&status,portMAX_DELAY);
                //ESP_LOGI(TAG, "[PACKET POSTED]");
                //puts("[Packet posted]");
            }
        }
        break;

What I have tried:

I was receiving a count of data structures as one serial byte. Then I would take that count and read "count" data structures off that serial line. That works, until I involve a queue with variable sized items. Necessary because the approach I just mentioned doesn't handle slow screens. It gets overwhelmed.

It looks kinda like this:

count = serial_getch();
if(count==-1) {
    break;
}
if(connected ==false || incoming_packets.size()!=count) {
    refresh = true;
}
incoming_packets.clear(true);
incoming_packets.reserve(count);
entry = ui_screen_entries;
for(int i = 0;i<count;++i) {
    if(sizeof(screen_packet_t)!=serial_read_bytes((uint8_t*)&packet,sizeof(packet))) {
        while(serial_getch()!=-1);
        goto done;
    }
    if(!refresh && entry!=nullptr) {
        if(0!=strcmp(entry->value_format,packet.format)) {
            refresh = true;
        }
        entry=entry->next;
    }
    incoming_packets.push_back(packet);
}