(untagged)

Beginners Intro to HTTP calls

Michael A. Barnhart

0.00/5 (No votes)

26 Nov 2002

A beginers explaination to the segements of a HTTP call.

Download source files and demo project - 109 Kb

Sample Image - http_calls.jpg

Introduction

Several recent articles have lead me to believe a beginners intro to what is included in a basic HTTP call is justified. Here is my explanation of the basic parts of both the text line seen and the actual messages that are sent, along with some sample code to experiment.

The sample(s) is provided to give one a starting point to look through. The code submitted is extracted from what I use to test SOAP calls to a service. It utilizes the MFC CInternetSession class.

The Server side code in the article (not in the down load) is generic C code. The server code has been run with no modifications as a CGI application under both the Apache Server (1.3.19) and MS's IIS server.

URL's and the Basic Segments to make a Connection.

Before we get into the code, let's spend a little time getting on the same page. We must understand what is used by the client to make a server connection and what is actually sent. The following line was typed into my browser:

http://localhost:680/cgi/cgimfc1?MyInfo=data1+AndMoreInfo=data2

Following completion of the server handling the request my browser displayed this:

Hello Me
Your First Line
This is more text There are 1 Arguments here 
The Query String is MyInfo=data1 AndMoreInfo=data2
The Server Soft is Apache/1.3.19 (Win32)
The SERVER_PORT is 680
The REQUEST_METHOD is GET
The REMOTE_USER is 
The CONTENT_TYPE is 
The CONTENT_LENGTH is 
OR The CONTENT_LENGTH is 0
The windir is C:\WINNT
The DeCode length is 0

Now what happened and what parts were involved. First let's look at the line we typed.

Segment Description Transmission

http	Tells the browser what kind of connection to make. This is not info that is transmitted.
://	Separator Ends Connection Type Definition
localhost	Tell the browser the address of the machine to talk to. This is not info that is transmitted other than to make the connection.
:	Separator Optional if the port number is given. Default http port number is 80
680	Tells the browser which port on the server to make the connection request to.This is not info that is transmitted other than to make the connection.
/	Separator Terminates Connection Information
Cgi/cgimfc1	Tells the Bowser the process to run (or page to return) This is part of the transmitted data
?	Separator Optional Ends Process definition
MyInfo=data1+AndMoreInfo=data2	Query Info that the server may use to handle the request. This is part of the transmitted data. This is called the Query

What was sent:

GET /cgi/cgimfc1?MyInfo=data1+AndMoreInfo=data2 HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, 
image/pjpeg, application/vnd.ms-excel, application/msword, 
application/vnd.ms-powerpoint, application/pdf, */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0)
Host: localhost:680
Connection: Keep-Alive

Comments on what was sent

The first line is going to start with either GET or POST. The lines following this are available via the environment calls. This a major service that is being provide by the Web Server Software. The last line in what is sent as the basic block of data is "Content Length" (not shown since no attached data) if an attached set of data is included with the transmission. The length of the transmitted additional data is the value on this line. This line is then terminated with two CR/LF pairs. Following this is the attached data. Since we just sent a request from the browser, this additional data is not there. It would be created by either scripting or a program generated http call.

What was returned:

HTTP/1.1 200 OK
Date: Thu, 14 Jun 2001 18:40:09 GMT
Server: Apache/1.3.19 (Win32)
Transfer-Encoding: chunked
Content-Type: text/html

1eb
<HTML><BODY>Hello Me<br><br>Your First Line<br><br>
This is more text
There are 2 Arguments here <br><br>
The Query String is MyQuery=QData<br><br>
The Server Soft is Apache/1.3.19 (Win32)<br><br>
The SERVER_PORT is 980<br><br>
The REQUEST_METHOD is POST<br><br>
The REMOTE_USER is <br><br>
The CONTENT_TYPE is <br><br>
The CONTENT_LENGTH is 23<br><br>
OR The CONTENT_LENGTH is 23<br><br>
The windir is C:\WINNT<br><br>
The DeCode length is 21<br><br>
This is Data to Send!<br><br>
</BODY></HTML>

Comments on what was returned.

Note the header data is read by the receiving program and contains the information on how the request was handled and any error message that may apply. Following this information is the HTML data that is displayed by the browser.

The Server CGI Program.

Both Apache and IIS support the HTTP server running a command line program under the server as a CGI app. (For those not on a NT machine Apache can be installed as an application and run to provide a learning environment.) In doing this the servers set up environment variables that the CGI program may ask for and the "standard in" points to the data stream of the attached data sent with the HTTP call (this is usually used with POST.) This may be done with the GET call also, but many servers have a fairly small buffer. The "standard out" points to the data stream of the data to be sent back. So all a CGI program needs to be is a command line application that reads the required environment variables and if needed the attached data stream and performs the operations required. This includes returning data if a faulty call was made!

Our first CGI program will be best done if we experiment with what data is available to get an understanding of the environment variables and reading attached data. We are learning, right.

We start the Visual Studio environment and ask to create a new Win32 Application.

We will call it Cgi1. I will elect to include MFC because I like the CString class. Care is needed here not use unhandled portions that cause user messages that would put the server in a wait state if no operators are available! I.E. This is not good practice for production code!

You have the choice of leaving the root entry point to being the int _tmain(int argc, TCHAR* argv[], TCHAR* envp[]) function or just using a standard main function. I have chosen to use the standard main function on the belief it would be more portable. Of course the MFC usage would have to be removed. My entire program (less the helper encoding functions) is:

// Cgi1.cpp : Defines the entry point for the console 

// application.

//

#include <stdio.h>


#include "stdafx.h"

#include "Cgi1.h"


#ifdef _DEBUG
#define new DEBUG_NEW
#undef THIS_FILE
static char THIS_FILE[] = __FILE__;
#endif

///////////////////////////////////////////////////////////////////

// The one and only application object


CWinApp theApp;

using namespace std;

int main(int argc, TCHAR* argv[])
{
    int nRetCode = 0;
    printf("Content-type: text/html\n\n");

    FILE *stream;
    stream = fopen( "CGITest.txt", "w" );
       printf("<HTML><BODY>");

    printf("Hello Me<br><br>");
    printf("Your First Line<br><br>\n");

    cout << "This is more text<br><br>" << endl;

    CString Line;
    Line.Format("There are %d Arguments here <br><br>\n",argc);
    printf((LPCSTR)Line);
    fprintf( stream, "%s\n",Line);

// Enviroment Variables

    CString path;
    path = DeCode(getenv("QUERY_STRING"));
    Line.Format("The Query String is %s<br><br>\n",path);
    printf((LPCSTR)Line);
    fprintf( stream, "%s\n",Line);

    path = getenv("SERVER_SOFTWARE");
    Line.Format("The Server Soft is %s<br><br>\n",path);
    printf((LPCSTR)Line);
    fprintf( stream, "%s\n",Line);

    path = getenv("SERVER_PORT");
    Line.Format("The SERVER_PORT is %s<br><br>\n",path);
    printf((LPCSTR)Line);
    fprintf( stream, "%s\n",Line);

    path = getenv("REQUEST_METHOD");
    Line.Format("The REQUEST_METHOD is %s<br><br>\n",path);
    printf((LPCSTR)Line);
    fprintf( stream, "%s\n",Line);

    path = getenv("REMOTE_USER");
    Line.Format("The REMOTE_USER is %s<br><br>\n",path);
    printf((LPCSTR)Line);
    fprintf( stream, "%s\n",Line);

    path = getenv("CONTENT_TYPE");
    Line.Format("The CONTENT_TYPE is %s<br><br>\n",path);
    printf((LPCSTR)Line);
    fprintf( stream, "%s\n",Line);

    path = getenv("CONTENT_LENGTH");
    Line.Format("The CONTENT_LENGTH is %s<br><br>\n",path);
    printf((LPCSTR)Line);
    fprintf( stream, "%s\n",Line);

    long ldata = atol((LPCSTR)path);
    Line.Format("OR The CONTENT_LENGTH is %d<br><br>\n",ldata);
    printf((LPCSTR)Line);
    fprintf( stream, "%s\n",Line);

    path = getenv("windir");
    Line.Format("The windir is %s<br><br>\n",path);
    printf((LPCSTR)Line);

    // Now Read the Data Block in from stdin if data has been 

    // labeled as existing.

    Line="";
    CString xstr;

    if(ldata>0)
    {
        fread(xstr.GetBuffer(ldata),sizeof(char),ldata,stdin);
        xstr.ReleaseBuffer();
        Line = DeCode(xstr);
    }
    xstr.Format("The DeCode length is %d<br><br>\n",Line.GetLength());
    printf((LPCSTR)xstr);
    fprintf( stream, "%s\n",xstr);

    Line+="<br><br>\n";
    printf((LPCSTR)Line);
    fprintf( stream, "%s\n",Line);

       printf("</BODY></HTML>");
    fclose( stream );
    return nRetCode;
}

I have used both the "printf" function and the cout << operator in the above.

Actual text returned by this program is:

HTTP/1.1 200 OK
Date: Thu, 14 Jun 2001 18:40:09 GMT
Server: Apache/1.3.19 (Win32)
Transfer-Encoding: chunked
Content-Type: text/html

1eb
<HTML><BODY>Hello Me<br><br>Your First Line<br><br>
This is more text<br><br>
There are 2 Arguments here <br><br>
The Query String is MyQuery=QData<br><br>
The Server Soft is Apache/1.3.19 (Win32)<br><br>
The SERVER_PORT is 980<br><br>
The REQUEST_METHOD is POST<br><br>
The REMOTE_USER is <br><br>
The CONTENT_TYPE is <br><br>
The CONTENT_LENGTH is 23<br><br>
OR The CONTENT_LENGTH is 23<br><br>
The windir is C:\WINNT<br><br>
The DeCode length is 21<br><br>
This is Data to Send!<br><br>
</BODY></HTML>

The header is supplied by the server. It then adds double cr/lf followed by length of return buffer on next line. Which is then followed by the text your program wrote out. It is required that your program output the content type for the server to use as the first output from the program. Following this you need to bound the remaining text with in the <HTML> and <BODY> tags if you wish to use a browser to display the results for testing. I have found this useful. When you are happy with the data you can then uncomment these lines and proceed.

Critical items to note here are the calls to get environment variables and the names of variables you may need. Is the call being a GET or POST important? Logging the Port important? What was the "Query String"? How long (if any) is the attached data. Then go read the data and handle it.

The Client Program (the download)

To get started we just need some basic functionality that all other usage can be implemented on. We also must be able to view our data to understand what is really being sent. The web browser is a starting point but we also need to understand how much processing the browser is doing to create the call to the server. Several items in the standard URL are never sent but are used to create the connection to the server. To help clarify this I created a C function which accepts the components needed and then does the processing of the request. This function can then be used throughout any application we choose.

To show this usage I will just create a simple Dialog Application with edit fields for each meaningful item.

The server address and port are needed to make the connection. That data is never actually sent to the server. Once the connection is made you must package the data to be sent. The actual data sent is as follows:

POST /cgi-bin/CgiMfc1?MyQuery%3dQData HTTP/1.1
Accept: text/*
User-Agent: HttpCall
Accept-Language: en-us
Host: localhost:680
Content-Length: 98

000629EF2656+vs+%0d%0a143%2e114%2e37%2e22+vs+%0d%0aDEKGKIEG+vs+
%0d%0a000629ef2656+vs+%0d%0a2051586

The Code to make the call in the demo is:

void CHttpAccessDemoDlg::OnSubmit() 
{
    CWnd* pWndRet = GetDlgItem(IDC_ReturnData);
    UpdateData(TRUE);
    m_ReturnedData="";
    CMemFile MemFile;
    int RetVal = HttpCall((LPCSTR)m_ServerAddress,m_ServerPort,
        (LPCSTR)m_ServerProcess,(LPCSTR)m_QueryString,
        (LPCSTR)m_DataBlock,&MemFile,pWndRet);
    int DataLen = MemFile.GetLength();
    MemFile.SeekToBegin();
    MemFile.Read(m_ReturnedData.GetBuffer(DataLen),DataLen);
    m_ReturnedData.ReleaseBuffer();
    UpdateData(FALSE);
}

Which calls the general a purpose function "HttpCall" which for this example uses the MFC CHttpSession and related classes.

int HttpCall(const char* ServerAddress,
             int ServerPort,
             const char* ServerProcess,
             const char* QueryString,
             const char* DataBlock,
             CFile* DataFile,
             CWnd* pStatusWnd)
{
    CString strUser;
    strUser = GetTheUserName();
    long lread,max;
    // No Need to parse URL we create entities themselves.

    DWORD ServiceType = INTERNET_SERVICE_HTTP;
    CString UserName="anonymous";
    CString UserPassWord="";

    CString lf;
    lf.Format("%c%c",char(13),char(10)); // need unix line feed


    CString EnCodeData = EnCodeStr(DataBlock);

    //

     // Setup Server

    DWORD AccessType = PRE_CONFIG_INTERNET_ACCESS;
    CString UserDef = "HttpCall";    // User Application


    CHttpSession* pHttpSession = new CHttpSession(UserDef,1,AccessType);
    pHttpSession->SetStatusWnd(pStatusWnd);

    CHttpConnection* pHttpConnection = NULL;
    pHttpConnection = pHttpSession->
                          GetHttpConnection(ServerAddress,ServerPort,
                                            UserName,UserPassWord);
    CHttpFile* pHttpFile = NULL;
    DWORD HttpRequestFlags;
    //

    CString Process,Query;
    Process = ServerProcess;
    Query = QueryString;
    if(Query.GetLength()>0) Process += '?'+EnCodeStr(Query);
    // Open the request and send it;

    // if the request has an invalid port it fails. 

        // Need to look for option for error test.

         TRY    
    {
        HttpRequestFlags = INTERNET_FLAG_EXISTING_CONNECT | 
                                   INTERNET_FLAG_RELOAD | INTERNET_FLAG_DONT_CACHE;

        DWORD TotalLen;
        TotalLen = EnCodeData.GetLength();
        if(TotalLen>0)
        {
            pHttpFile = pHttpConnection->OpenRequest(CHttpConnection::HTTP_VERB_POST,
                Process, NULL, 1, NULL, (LPCTSTR)"1.0", HttpRequestFlags);
        }
        else
        {
            pHttpFile = pHttpConnection->OpenRequest(CHttpConnection::HTTP_VERB_GET,
                Process, NULL, 1, NULL, (LPCTSTR)"1.0", HttpRequestFlags);
        }
        // Use direct write to posting field!

        CString strHeaders = "Accept: text/*\r\n";
        strHeaders += "User-Agent: HttpCall\r\n";
        strHeaders += "Accept-Language: en-us\r\n";

        //    strHeaders += "Content-type: application/x-www-form-urlencoded\r\n";

        //    strHeaders += "REMOTE_USER: "+strUser+"\r\n";

        //    strHeaders += "Accept-Language: en-us\r\n";


        pHttpFile->AddRequestHeaders((LPCSTR)strHeaders);


        if(TotalLen>0)
        {
            pHttpFile->SendRequestEx(TotalLen,HSR_INITIATE,1);
            pHttpFile->WriteString((LPCTSTR)EnCodeData);
            pHttpFile->EndRequest();
        }
        else
        {
            pHttpFile->SendRequest();
        }
        max = pHttpFile->GetLength();
    }
    CATCH_ALL(e)
    {
        AfxMessageBox("Connection to Server Failed");
    }
    END_CATCH_ALL
    DWORD dwRet;
    pHttpFile->QueryInfoStatusCode(dwRet);
    //

    //Check Return Code.

    DataFile->SeekToBegin();
    // Read Data

    CString strRetBufLen;
    pHttpFile->QueryInfo(HTTP_QUERY_CONTENT_LENGTH, strRetBufLen);
    max = atol((LPCSTR)strRetBufLen);
    if(max<=0)
    {
        max = pHttpFile->GetLength();
    }
    // Read Data

    lread=1000;
    char c[1000];
    while(lread>0)
    {
            lread = pHttpFile->Read(c,1000);
            if(lread>0) DataFile->Write(c,lread);
    }

    pHttpFile->Close();
    pHttpConnection->Close();
    pHttpSession->Close();
    delete pHttpFile;
    delete pHttpConnection;
    delete pHttpSession;
    return dwRet;
}

General purpose helper functions:

CString EnCodeStr(CString ToCode)
{
    CString RetStr,AddStr;
    int i,max;
    unsigned short asc;
    unsigned char c;
    max = (unsigned int)ToCode.GetLength();
    for(i=0;i<max;i++)
    {
        c = ToCode[i];
        asc = c;//(unsigned int)c;

        if(asc&gt47 && asc&lt58)
        {
            RetStr+=c;
        }
        else if(asc&gt64 && asc&lt91)
        {
            RetStr+=c;
        }
        else if(asc&gt96 && asc&lt123)
        {
            RetStr+=c;
        }
        else if(asc==32)
        {
            RetStr+="+";
        }
        else
        {
            AddStr.Format("%%%2x",asc);
            int iv = (int)AddStr.GetAt(1);
            if((int)AddStr.GetAt(1)==32)
            {
                AddStr.SetAt(1,'0');
            }
            RetStr+=AddStr;
        }
    }
    return RetStr;
}

CString DeCodeStr(CString ToCode)
{
    CString RetStr,AddStr;
    int i,max;
    unsigned short asc;
    unsigned char c;
    max = (unsigned int)ToCode.GetLength();
    for(i=0;i&ltmax;)
    {
        c = ToCode[i];
        asc = c;//(unsigned int)c;

        if(asc==37)
        {
            AddStr=ToCode.Mid(i+1,2);
            i+=3;
            sscanf((LPCTSTR)AddStr,"%2x",&asc);
            RetStr+=(char)asc;
        }
        else if(asc==43)
        {
            RetStr += ' ';
            i++;
        }
        else
        {
            RetStr += c;
            i++;
        }
    }
    return RetStr;
}

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here