Use QNetworkAccessManager for synchronous downloads

• Download demo - 30.2 KB
• Download source - 17.7 KB

Introduction

Sometimes help forums frankly turn out to not being helpful at all. Using Google on the topic, the question can be found in many places: "How can I use QNetworkManager for synchronous downloads?". Several times, the first hint is "Use it in asynchronous mode." This tends to make me angry, not only because the reply is useless. It also sounds to me like "Don't you know how to use it asynchronously, stupid?".

Of course I know how to do that, as probably does whoever posed the question before. But the problem prevails: we do have a reason why we want to download synchronously. Why the heck would we ask otherwise?

The reason in my case is, I want to stream a video or sound file via http from a web server and start playing it immediately without having to complete the download in the first place. Also, I want to play Shoutcast streams which are simply impossible to download first, as they have no end-of-file. And finally I want to play DVDs directly via http, which involves concatenating the 1GB file parts to a virtual single large image and handle them as such. It is as simple as that.

Qt offers the QNetworkManager class which has proxy server support, can authenticate against proxies if required (it even supports Microsoft NTLM), can send sign on credentials to remote servers, implements the http put/get methods including passing of parameters or data, and can manipulate the http headers as well as evaluating the server reply.

In other words: everything you need when talking to web servers.

So: Why write a new class? Reinvent the wheel just because it seems that the Qt approach cannot handle synchronous downloads?

The answer is: no. It can. And it's easy.

The Basics

Up front, after sending out the request, the code has to wait for the reply. This is clear as we want to know what happened – did we get a connect, have we been able to access the file we asked for? There is a whole bunch of things that could go wrong at that stage, so we need to add a check:

QNetworkAccessManager *manager = new QNetworkAccessManager(this);

connect(manager, SIGNAL(proxyAuthenticationRequired(const QNetworkProxy &, QAuthenticator *)), 
  SLOT(slotProxyAuthenticationRequired(const QNetworkProxy &, QAuthenticator *)));
connect(manager, SIGNAL(authenticationRequired(QNetworkReply *, QAuthenticator *)), 
  SLOT(slotAuthenticationRequired(QNetworkReply *, QAuthenticator *)));

QNetworkRequest request;

request.setUrl(m_url);
request.setRawHeader("User-Agent", "Qt NetworkAccess 1.3");

m_pReply = manager->get(request);

After sending out the request with get(), we enter an event loop and wait for the finished() signal from the network access manager. When it arrives, the request has completed - either successfully or it has failed. Next we should call reply->error() to see if there was a problem and report it, or if all went well subsequently call reply->readAll() or similar to get our data.

If we want to be able to start reading at an arbitrary offset, we could throw in these lines just before the get(request) call:

if (m_nPos)
{
    QByteArray data;
    QString strData("bytes=" + QString::number(m_nPos) + "-");

    data = strData.toLatin1();
    request.setRawHeader("Range", data);
}

Up to this point, one can find internet sources that already describe the steps above. Alas, there are drawbacks which prove to be quite nasty showstoppers:

1. The file will be downloaded completely into memory before finished() fires. For large files, this can take quite a while and even fill up all memory (causing your application to crash).

2. Using a different signal that fires earlier like readyRead() by the QNetworkRequest reply object will leave you with the first portion of a large file, but you are unable to get the rest.

3. There is no way to display a progress to the user whilst downloading.

4. Streams cannot be handled this way, as finish() would never be called – the app will lock up, eventually run out of memory and crash.

5. When using this code in non GUI threads the QEventLoop can, and will, cause deadlocks.

The solution

Getting this to work is a combination of several little tricks:

1. Using a QThread to handle the incoming data.

2. Using different approaches for thread synchonisation depending on whether running in GUI or non GUI mode.

3. For GUI threads, using QEventLoop ensures that we are maintaining a snappy user interface.

Unfortunately, a few small glitches remain:

1. If the data processing thread is not fast enough, the memory can still run full.

2. The application complains that “QTimers must be created from QThread”. This is a Qt flaw: Our threads are QThreads and the timers actually do work.

The first problem is likely to come up when playing a compressed file from the web, e.g., a 128Kbit MP4 will almost certainly be pulled at a slower speed from the server as the internet connection could sustain. The server can send data faster as required, gradually filling up memory at the receiving end.

The code

How to solve the problem is shown here with a few code snippets. Please download the complete code and the example for the complete solution.

First of all we need to create and start a QThread to handle incoming data while still keeping our user interface happy and alive:

webfile::webfile(QObject *parent /*= 0*/) :
    QObject(parent),
    m_pNetworkProxy(NULL)
{
    // QThread is required, otherwise QEventLoop will block
    m_pSocketThread = new QThread(this);
    moveToThread(m_pSocketThread);
    m_pSocketThread->start(QThread::HighestPriority);
}

In open and read operations the thread will be used. Please note that we have to check if we are running as a GUI thread or not and act differently. Calling the exec() function of QEventLoop can cause a deadlock in non GUI threads – signals sent by QNetworkRequest and such would not be accepted and so the quit() slot would never be activated.

bool webfile::open(qint64 offset /*= 0*/)
{
    bool bSuccess = false;

    if (isGuiThread())
    {
        // For GUI threads, we use the non-blocking call and use QEventLoop to wait and yet keep the GUI alive
        QMetaObject::invokeMethod(this, "slotOpen", Qt::QueuedConnection,
                                  Q_ARG(void *, &bSuccess),
                                  Q_ARG(void *, &m_loop),
                                  Q_ARG(qint64, offset));
        m_loop.exec();
    }
    else
    {
        // For non-GUI threads, QEventLoop would cause a deadlock, so we simply use a blocking call.
        // (Does not hurt as no messages need to be processed either during the open operation).
        QMetaObject::invokeMethod(this, "slotOpen", Qt::BlockingQueuedConnection,
                                  Q_ARG(void *, &bSuccess),
                                  Q_ARG(void *, NULL),
                                  Q_ARG(qint64, offset));
    }

    return bSuccess;
    // Please note that it's perfectly safe to wait on the return Q_ARG,
    // as we wait for the invokeMethod call to complete.
}

void webfile::slotOpen(void *pReturnSuccess, void *pLoop, qint64 offset /*= 0*/)
{
    *(bool*)pReturnSuccess = workerOpen(offset);

    if (pLoop != NULL)
    {
        QMetaObject::invokeMethod((QEventLoop*)pLoop, "quit", Qt::QueuedConnection);
    }
}

The parameters are passed conveniently as pointers via Q_ARG(). This is perfectly legal in this case, because we wait until the slot completes its task, so they will never inadvertedly be pulled off while the thread is still active. This way we can get the results as well (success or the data in case of the read() call).

Here is the complete code for connecting to the server, and waiting for the result:

bool webfile::workerOpen(qint64 offset /*= 0*/)
{
    qDebug() << "webfile::open(): start offset =" << offset;

    clear();

    resetReadFails();

    close();

    QNetworkAccessManager *manager = new QNetworkAccessManager(this);

    if (m_pNetworkProxy != NULL)
        manager->setProxy(*m_pNetworkProxy);

    connect(manager, SIGNAL(proxyAuthenticationRequired(const QNetworkProxy &, QAuthenticator *)), 
      SLOT(slotProxyAuthenticationRequired(const QNetworkProxy &, QAuthenticator *)));
    connect(manager, SIGNAL(authenticationRequired(QNetworkReply *, QAuthenticator *)), 
      SLOT(slotAuthenticationRequired(QNetworkReply *, QAuthenticator *)));

    QNetworkRequest request;

    request.setUrl(m_url);
    request.setRawHeader("User-Agent", "Qt NetworkAccess 1.3");

    m_nPos = offset;
    if (m_nPos)
    {
        QByteArray data;
        QString strData("bytes=" + QString::number(m_nPos) + "-");

        data = strData.toLatin1();
        request.setRawHeader("Range", data);
    }

    m_pReply = manager->get(request);

    if (m_pReply == NULL)
    {
        qDebug() << "webfile::open(): network error";
        m_NetworkError = QNetworkReply::UnknownNetworkError;
        return false;
    }

    // Set the read buffer size
    m_pReply->setReadBufferSize(m_nBufferSize);

    connect(m_pReply, SIGNAL(error(QNetworkReply::NetworkError)), SLOT(slotError(QNetworkReply::NetworkError)));
    connect(m_pReply, SIGNAL(sslErrors(QList<QSslError>)), SLOT(slotSslErrors(QList<QSslError>)));

    if (!waitForConnect(m_nOpenTimeOutms, manager))
    {
        qDebug() << "webfile::open(): timeout";
        m_NetworkError = QNetworkReply::TimeoutError;
        return false;
    }

    if (m_pReply == NULL)
    {
        qDebug() << "webfile::open(): cancelled";
        m_NetworkError = QNetworkReply::OperationCanceledError;
        return false;
    }

    if (m_pReply->error() != QNetworkReply::NoError)
    {
        qDebug() << "webfile::open(): error" << m_pReply->errorString();
        m_NetworkError = m_pReply->error();
        return false;
    }

    m_NetworkError      = m_pReply->error();

    m_strContentType    = m_pReply->header(QNetworkRequest::ContentTypeHeader).toString();
    m_LastModified      = m_pReply->header(QNetworkRequest::LastModifiedHeader).toDateTime();
    m_nSize             = m_pReply->header(QNetworkRequest::ContentLengthHeader).toULongLong();

    m_nResponse         = m_pReply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt();
    m_strResponse       = m_pReply->attribute(QNetworkRequest::HttpReasonPhraseAttribute).toString();
    m_bHaveSize =  (m_nSize ? true : false);

    m_nSize += m_nPos;

    if (error() != QNetworkReply::NoError)
    {
        qDebug() << "webfile::open(): error" << error();
        return false;
    }

    m_NetworkError = response2error(m_nResponse);

    qDebug() << "webfile::open(): end response" << response() 
      << "error" << error() << "size" << m_nSize;

    return (response() == 200 || response() == 206);
}

And not to forget the wait function:

bool webfile::waitForConnect(int nTimeOutms, QNetworkAccessManager *manager)
{
    QTimer *timer = NULL;
    QEventLoop eventLoop;
    bool bReadTimeOut = false;

    m_bReadTimeOut = false;

    if (nTimeOutms > 0)
    {
        timer = new QTimer(this);

        connect(timer, SIGNAL(timeout()), this, SLOT(slotWaitTimeout()));
        timer->setSingleShot(true);
        timer->start(nTimeOutms);

        connect(this, SIGNAL(signalReadTimeout()), &eventLoop, SLOT(quit()));
    }

    // Wait on QNetworkManager reply here
    connect(manager, SIGNAL(finished(QNetworkReply *)), &eventLoop, SLOT(quit()));

    if (m_pReply != NULL)
    {
        // Preferrably we wait for the first reply which comes faster than the finished signal
        connect(m_pReply, SIGNAL(readyRead()), &eventLoop, SLOT(quit()));
    }
    eventLoop.exec();

    if (timer != NULL)
    {
        timer->stop();
        delete timer;
        timer = NULL;
    }

    bReadTimeOut = m_bReadTimeOut;
    m_bReadTimeOut = false;

 return !bReadTimeOut;
}

Now it is your turn!

This is how that can be done. The rest is up to you. I would appreciate if you sent me patches or bug reports so I can improve my code.

Besides of that, hopefully the code will save you from some trouble I had, so: lean back and enjoy.

History

Released v1.0 (27.10.2012)

Released v1.1 (25.03.2013):

• Updated code for Qt5
• added two extra error messages (for HTTP 401 and 403)

Released v1.2 (27.03.2013):

• read() could return less than the number of requested bytes (even 0 if not enough received)
• Webfile monitor calculated astronomically high average download rates if seek was used
• for v1.1 I screwed up the download files, sorry for that one.

This is getting us quite near a "factory strength" release.

Released v1.3 (24.10.2013):

• Finally added readLine() and readAll().
• New httpfileinfo and httpdir classes provide some baisc functionality similar to QFileInfo and QDir.
• Code has been tested a lot can now be seen as stable or even "factory strength".