Introduction
It is well known that HTTP is the protocol of the Web, what is probably less known by many people is that the base idea of HTTP is indeed quite simple and can be used for many other applications even not related with the internet.
An example of such application and very useful nowadays would be communicating devices that are attached to a home WLAN. For instance, a PC with a smart phone, a tablet, etc.
In this article, we show how to do it and for that, we will implement some basic but quite demonstrative functionalities like getting a screenshot from the PC, interchanging files (download and upload) and save notes into the PC written from a device.
The demos use a convenient and tiny HTTP server but they differ from typical Web applications in several ways. Thought to be ran in a secured network like a home WLAN, there is no need to deal with security issues. Also because of this, we can make the server play any role we want thus having full control of it which cannot be the case in a usual web application.
HTTP Very Briefly
Although the HTTP specification is quite long and contains a lot of rules and features, we only need to know a small but fundamental part of it. Partly because many things like transferring files are already implemented in the server and client that we are going to use and partly because in "home" applications like these, there is more freedom and flexibility for the design than in a generic product thought to have many users. Finally, it is probably just the basis of this protocol, its simplicity and inherent flexibility is the most powerful part of HTTP.
TCP/IP Base Communication
HTTP has on its base TCP/IP which is a point to point and a client-server communication protocol. Each point is identified by an IP address which is unique for each device (i.e., a PC, smart phone, etc.) in a network (i.e., home WLAN or internet) and a port number where each device has more than 65000 port numbers available.
For example, two TCP/IP points in a home WLAN could be:
Device IP TCP port
----------- --------------- ----------
PC 192.168.178.22 8080
Smart phone 192.168.178.36 1727
In TCP/IP, client-server means the server is listening to its port for incoming requests from clients, when a request is received, it comes along with the IP and port number of the client, so the server can respond to the request. By now, think of REQUEST and RESPONSE just as a set of an arbitrary number of bytes.
Server Client
(PC) (Smart phone)
192.168.178.22:8080 192.168.178.36:1727
| |
[]<------------- REQUEST ------------|
[] |
[] |
[]------------- RESPONSE ----------->|
| |
This request-response principle is simple and flexible but also has some restrictions that has to be taken into account when designing an application using it. Among other things, what we can say about this kind of communication is:
- A server has to be started first and be awake (listening) always
- A client needs to know the IP and port of the server to communicate with it
- The communication is started by the client with a request and the server can just respond to that request.
- The server does know anything about clients until it receives requests from them.
The TCP/IP address (IP and port) is not shareable, which means we cannot have two applications located in the same machine (same IP address) opening the same port for HTTP communication. Still multiple clients, each one having its own unique TCP/IP address, might send requests to the same server simultaneously.
HTTP Messages
Above the TCP/IP protocol, HTTP defines how the REQUEST and RESPONSE has to be formed. For that, it uses a simple structure called HTTP message which may have up to three parts separated by CRLF (carriage return and line feed characters): START line, HEADER part and BODY part.
General Syntax Request example Response Example
----------------- ----------------------------- ------------------------------
START LINE<CRLF> GET /myIndex.html HTTP/1.1 HTTP/1.1 200 OK
MESSAGE HEADER<CRLF> Content-Type: text/html
... Content-Length: 170
MESSAGE HEADER<CRLF> ...
<CRLF>
MESSAGE BODY <html><body> ...
... ....
All these parts can be used to send information from client to server and vice versa. Let's take a closer look at them.
Start Line
The start line is different for REQUEST and RESPONSE messages. In the REQUEST, the start line has the function of telling the server what we want to do: get a resource, store something, etc. and for that giving some kind of detail is needed, like a path of a file, an id or more things. The general syntax is:
METHOD URI HTTP_VERSION
where
METHOD : GET, POST, ...
URI : path or path?query
HTTP_VERSION : HTTP/1.1
For METHOD, there are more defined values in the specification but in many cases, just GET and POST are used as we will do here.
"path" might correspond or not to a physical file in the server. In many cases of GET requests, it does but in the end it is the server who decides the meaning of "path" and in general of the whole request. For instance, taking together METHOD and "path", it is possible to describe exactly the same actions using more or less number of methods. Just as an example, we could express the operations of retrieving, storing and removing an event as:
using GET,PUT,DELETE using only POST
----------------------- ----------------------
GET /event POST /get/event
PUT /event POST /store/event
DELETE /event POST /delete/event
In general, the URI in an HTTP message does not need host and port information as in a URL in the browser because these values are already known by the TCP/IP connection itself. Some examples showing this:
Typical URL in browser Request received by the server
---------------------------------- ---------------------------------
http://www.wakeupthecat.com/info GET /info HTTP/1.1
http://192.168.178.22:1811/info GET /info HTTP/1.1
Since the browser always performs a GET in these cases.
Finally, "query" is an optional string with "variable=value" units separated by the character '&'. This is a good place to put some parameters to complete the request information. An example of the query part.
?id=7172&name=Sonia
In the RESPONSE, the start line is:
HTTP_VERSION STATUS_ID STATUS_TEXT
It contains much less information since only a set of predefined status can be returned, for example "200 OK" or the well known "404 Not Found". We will use only:
HTTP/1.1 200 OK
any other feedback to the client still can be sent by using response headers or body.
Header Part
The header part is composed for an arbitrary number of lines, each one consisting of two strings separated by a colon, the first string is the header name and the second its value.
In a HTTP application, custom headers can be set from client and server side to pass information. There are as well standard headers defined in the specification, in particular one is not just important but critical for the protocol itself: the Content-Length. Its value has to be exactly the number of bytes of the body (if any) in order for the communication to work. Fortunately, both server and client (browser) will fill this header automatically, so we don't have to worry about it.
Also automatically the header "Content-Type" will be set in requests and responses, this is needed for instance by the browser in order to treat properly CSS, HTML, etc. contents.
Other headers even standard ones send from browsers can be ignored as they are irrelevant for the applications, although they could be handled if desired.
Body
The body is the most free part, it can transmit everything, from text to binary data. When the browser request a file using the GET method, if it is an actual file the body contains just all bytes of that file. Content for the body in form of readable text can be used to transmit HTML, CSS, JavaScript or any other structured data like XML, JSON or what we use here EVA (Estructura Variable de Archivo).
HTTP Client and Servers
A communication application using HTTP can be implemented in almost any language. Nowadays, it makes a lot of sense to use modern browsers for the client side. They have tons of features and are already installed on most devices. Of course, by the use of browsers, we have to accept the limitations imposed for security reasons on the access and control of the host device itself, but for many application purposes, it should not be an issue.
Browsers have to be programmed using HTML, CSS and JavaScript. In our examples, we will use a JavaScript library called jGastona that will make all this a little bit easier.
For the server side, we will use an open source written by myself called gastona (gastona.jar) which is general purpose application builder and can implement an http server in a very straighforward way.
HTTP with Browsers
There are basically two requests, actually the most frequently used and the ones that we are going to use in this demo, GET and POST.
The GET method is used to retrieve resources that in general are going to be rendered or used for rendering purposes in the browser. For example, when we click on a link in a HTML page, the content (attribute href of "a" tag) is retrieved using a GET request. The URI to retrieve can be directly a file name or any other text and the result is send by the server using the BODY. The very first request of the browser to a server is using the uri "/" the servers use to interpret as "/index.html" as default page.
Server Client
(PC) (Smart phone)
192.168.178.22:8080 192.168.178.36:1727
| |
[]<------------- "GET /" ---------------------|
[] |
[] |
[]------------- RESPONSE BODY ---------------->|
| <html> |
| <a href="nextPage.html">Next</a> |
| </html> |
| |
| |
[]<------------- "GET /nextPage.html" --------| User click on "Next"
[] |
[] |
[]------------- RESPONSE BODY ---------------->|
| <html>
| .... etc
| </html>
Appart from the requests that the browser do automatically, it is possible to perform requests using JavaScript for any purpose: set or retrieve data or any communication with the server. This mechanism is called AJAX and, in general, the method POST is used for it. Also uploading a file is done using a POST request.
0. A First HTTP Server
The samples explained in this article build both the server and client side of the application. In both cases, the GUI (graphic user interface) and the logic are separated in two big groups or units called javaj and listix. Adding a third unit "data" which is common to both GUI and logic, we finally have the general structure:
#javaj#
variables to define GUI elements and its layout
#data#
variables to hold data of GUI elements and being accessible by listix
#listix#
variables to define the logic containing text and commands
In this first very simple HTTP server, we use pure HTML for the client, so the focus will be only on the server side.
The purpose is just serve a initial page containing a link to a second one which in turn has also a link to the first one. Although the simplicity, it is actually a multi page application (MPA) because the browser is loading a new page each time. Next examples will be Single Page Application (SPA) which requires a little bit more programing in server and client sides.
The command needed to start the server is called MICOHTTP (or simply MICO) and has the syntax:
MICO, START, serverNamne, portNumber
if we don't specify a portNumber, one available is chosen by the system, generally between ~ 50000 to 65000. Also in this case, MICO will automatically open a browser pointing to the port so a first request to the server is immediately sent by the browser. This is very convenient for developing and testing phases, but of course, the last intention is to communicate from a browser of a device.
So if we start our server with the command:
MICO, START, myFirstServer
We will have the following communication scenarios:
Server Client
(PC) (Web Browser in PC)
localhost:50130 localhost:56380
| |
[]<------------- "GET /" ---------------------| Web browser opened by the MICO command
[] | pointing to localhost:50130 (the server)
[] |
Notes: localhost is the name equivalent to the IP 127.0.0.1
ports numbers 50130 and 56380 are in this case chosen by the system and browser
On giving an address to the browser this sends immediately a first request to the server
To connect to the server from another device in the network (e.g. via WiFi)
we need to know the IP address of the server, localhost or 127.0.0.1 will not work!
The way to program the server to respond to a request is to put the response body in a variable named like the specific request (METHOD and path). In this case, both pages will be static texts, which can be written as:
<GET />
//<html>
// First page! <a href="/nextPage.html">Go to a second page</a>
//</html>
<GET /nextPage.html>
//<html>
// 2nd page <a href="/">Go to the first one</a>
//</html>
Finally, let's put a console as GUI for the server. Although real servers have actually no GUI at all, this will help us for testing as well as being a way of stopping the server when we close the window.
Putting all together in a gastona script (firstServer.gast):
#javaj#
<frames> oConsola
#listix#
<main>
MICO, START, myFirstServer
<GET />
//<html>
// First page! <a href="/nextPage.html">Go to a second page</a>
//</html>
<GET /nextPage.html>
//<html>
// 2nd page <a href="/">Go to the first one</a>
//</html>
now by typing in the command line:
java -jar gastona.jar firstServer.gast
We have our first server running.
In this example, we are serving the pages directly from variables, but the pages can be placed in physical files as well. Supposing we name index.html the first page and nextPage.html the second one, the script itself will be reduced then to:
#javaj#
<frames> oConsola
#listix#
<main>
MICO, START, myFirstServer
<GET />
@<:infile index.html>
Now in response to the first request, we load the contents of index.html but nothing is needed for further requests referred to physical files that the server might find, since these will be served by MICO directly from the file. This means that if index.html has links to other pages, images, etc. that are to be found in the file system, the above script would be enough to act as a server.
Sites based only on files (HTML, CSS, JS, GIF, etc.) that not change are referred as static web pages, now we know how to achieve it using MICO HTTTP server.
The next three examples explained will be SPA or Single Page Applications.
Loading JAST Script for the Client Part
For the client - the browser in this case - we will use of course HTML, CSS and JavaScript, but the JavaScript library jGastona will make things a little bit easier with them.
First, the library has to be loaded in the browser, then we initialize jGastona with a special script, let's call it JAST, which will define the client application in each case.
As we will see the JAST script, used for the client, and the gast script used for the server are very similar in form but actually they are completely different implementations where one runs as a desktop application and the other just in the browser.
The following code for the server does all the needed loads in one step just in the first response.
<GET />
//<html><style> * { font-family: Tahoma } </style> <body><script>
where META-GASTONA/js/jGastonaEva-min.js is contained in gastona.jar so it will be loaded from there and MAIN_JAST is the name of a variable where we have to put the JAST script for a particular application.
Note that the HTML body that we provide contains no HTML element at all except the script tag. The magic happens in jGastona that based on the JAST we provide will generate all the elements on the fly directly in the browser.
Now we have to program on each case the JAST script and the server logic by defining the responses to the rest of http requests that can be sent.
1. Screenshot from Server
This utility has the purpose of getting a screenshot of the PC where the server is running and send it back to the client. So, for example, we can check the progress of some task running on the PC from the kitchen using a smartphone.
Client side (JAST)
For the GUI of the client, we just need a button and an image element. We give them a name for further reference, the first character or characters indicate the type of element, in this case "b" for button and "m" for image. We dispose them in a special grid layout called EVALAYOUT as follows:
#javaj#
<layout of main>
EVA, 12, 12, 8, 8
---, X
, bScreenshot
X , mImagen
when the user presses the button, the server has to do a new screenshot and send the new image to be updated in mImagen. Also from the client point of view, this can be described simply as: on pressing the button, update the content of mImagen.
Given that the button event is captured in the variable "-- bScreenshot", the following code will do the job.
#listix#
<-- bScreenshot>
//AJAXgetIdContent ("mImagen");
so when the button "bScreenshot" is pressed, we call the jGastona function AJAXgetIdContent which asks the server for updating the image "mImagen" or better said, the source of the image.
Putting all together in the variable MAIN_JAST:
<MAIN_JGAST>
//#javaj#
//
// <layout of main>
// EVA, 12, 12, 8, 8
//
// ---, X
// , bScreenshot
// X , mImagen
//
//#listix#
//
// <-- bScreenshot>
// //AJAXgetIdContent ("mImagen");
Server Side
The server has to respond to the call done by the client through the function AJAXgetIdContent which is nothing but the request:
POST /getIdContent?id=mImagen HTTP/1.1
A first (but naive) try to do so would be:
<POST /getIdContent>
SCREEN, myScreenshot.png, png
myScreenshot.png
where the first line is the command SCREEN that makes a screenshot over the file myScreenshot.png of png type. Note that it is executed but it does not generate actually any text to be added to the body, while the second line is a text (only one element in the row) and will form the response body of the getIdContent request.
This will work, but only the first time! Why?
After each screenshot, the browser should perform the request "GET /myScreenshot.png" so the server can deliver the new content of the file. But this only happens the first time, next requests are taken from the browser cache instead, and the server does not receive the GET anymore.
To avoid this caching mechanism, we will rotate the name and remove the old screenshot to keep the directory clean. It is not the only solution but it will work with any browser and also illustrates how to call system functions.
#data#
<NN> 0
<SHOT_NN_PNG> //cache/screenshot_@<NN>.png
#listix#
<POST /getIdContent>
CALL, //CMD /C del @<:path SHOT_NN_PNG>
NUM=, NN, NN + 1
SCREEN, @<SHOT_NN_PNG>, png
@<SHOT_NN_PNG>
The first command is a CALL for Windows to delete the old screenshot file, for Linux or MacOS, we should write:
CALL, //rm @<SHOT_NN_PNG>
Note that for windows, we use "@<:path ..." that changes the "/" to the native separator (in windows "\") in order for the "del" command to work. We haven't written directly "\" in the variable SHOT_NN_PNG because for the browser, we need the separator "/" even if the server runs in Windows.
Listix distinguish commands from text because commands and its parameters and options are always expressed in more than one column separated by commas, therefore a row containing only one value or column is a text.
Note also the use of "//" this time not at the beginning of the row but within a command. The sequence "//" has the special meaning of "take the value from here to the end of the line". It is usually needed when the value contains commas or other special characters like < or quotes but in general, it can be always used for the value of the last column of a row.
Complete sweet_Screenshot.gast
#javaj#
<frames> oConsola
#data#
<NN> 0
<SHOT_NN_PNG> //cache/screenshot_@<NN>.png
#listix#
<main>
micohttp, start, monoMico
<GET />
//<html><style> * { font-family: Tahoma } </style> <body><script>
//
//@<:infile META-GASTONA/js/jGastonaEva-min.js>
//
// var jgas = jGastona (evaFileUTF82obj ("@<:encode-utf8 MAIN_JGAST>"));
//
//</script></body></html>
<MAIN_JGAST>
//#javaj#
//
// <layout of main>
// EVA, 12, 12, 8, 8
//
// ---, X
// , bScreenshot
// X , mImagen
//
//#listix#
//
// <-- bScreenshot>
// //AJAXgetIdContent ("mImagen");
<POST /getIdContent>
CALL, //CMD /C del @<:path SHOT_NN_PNG>
NUM=, NN, NN + 1
SCREEN, @<SHOT_NN_PNG>, png
@<SHOT_NN_PNG>
2. A File Server With Upload Ability
In this example, we allow downloading files located in a specific directory on the server and also upload one file at a time from a device. The directory where the files to download are placed can be configured in the script while the upload is done always in a fixed directory named "filesUpload" that has to be created or ensured on starting the script.
For the files to be downloaded, we just have to generate am HTML table with links to the final files, then when clicking on them in the browser, they will be requested and downloaded automatically.
For the upload feature, we need two buttons - one to select the file to upload and one to start the download.
Uploading will happen automatically in the server side, we still will code something in the client side for a basic feedback of the upload progress.
Client Side (JAST)
In this GUI, we also use a single list of component, so our grid will be:
, X
, lAvailable files
X , dFileList
, lUpload a file
, uSelectUpFile
, bUpload
where the widget type used are:
l Label
d div (to place the html table later)
u special button for selecting file to upload
b normal button
For the logic, we request initially the list of the files to download with:
<main>
//AJAXgetIdContent ("dFileList");
For the button to start the upload of the file, we use jGastona functions to make some checks, start the upload and once started, we change the text of the button to "uploading ..." to provide some feedback and on receiving the server response, restore the initial button label.
<-- bUpload>
//if (canUploadFile ("uSelectUpFile", 500)) {
// if (AJAXUploadFile ("uSelectUpFile", "uploadFile")) {
// setData ("bUpload", "uploading ...");
// }
//}
<-- ajaxResponse uploadFile>
//setData ("bUpload", "Upload file");
Server Side
First, the server will receive the post:
POST /getIdContent?id=dFileList HTTP/1.1
in order to serve the table with the links, we use the command LOOP (LOOP, FILES) in all loop commands the default option is BODY (body of the loop) and it can be omitted as we will do in further examples:
<FIDIR> filesForDownload
<POST /getIdContent>
// <table>
//
LOOP, FILES, @<FIDIR>
, RECURSIVE, 0
, BODY , // <tr>
, BODY , // <td> <a href="@<FIDIR>/@<fileName>"> @<FIDIR>/@<fileName> </a>
, BODY , // </tr>
//
// </table>
Note within the loop, the use of the variable "FIDIR" which is declared before but more interestingly, the use of the variable "fileName" which is specific for this loop and changes its value with each iteration.
Regarding the upload, if a file has been uploaded successfully, the server sets the variable _uploadedFile0 with the name of the physical file on the server side. So we can check it and return the appropriate response (ok or nok):
<POST /uploadFile>
CHECK, VAR, _uploadedFile0, nok
CHECK, FILE, @<_uploadedFile0>, nok
ok
although for the sake of simplicity, we actually don't use this information in the client side.
3. A Simple Bulletin Stored in the Server
The purpose of this utility is to have a bulletin of events stored in the server with date, title and description that the client can handle remotely.
For this script, we will learn more advanced features from gastona and jGastona, namely:
- Dynamic GUI layout switching
- Passing structured data (a form)
- Using SQL databases
Client Side
In this example, we have separated the JAST in its own file Boletin.JAST.
Let's take a look at the JAST code combined with the result in the browser:
In the javaj unit, using very few lines, the GUI consisting of two different layouts is created, HTML elements and its positions, all except the widget dBoletin which table content is filled by the server.
jGastona will load initially the layout main. On pressing the "new Event" button, the jGastona method mask("main", "layInput") is called and it makes "layInput" to be shown instead of "main". The contrary occurs ("main" layout is restored) when pressing the button "bCancel". All this happens only in the browser, no request or response from the server is needed to switch layouts.
Also important to note is that on creating HTML components, variables with their names are created in the data unit. So for example when entering date, title and description on a new event, the variables eEventDate, eEventTitle and xEventDesc are filled with the input values and when calling AJAXSend ("insertEvent") are sent to the server.
Server Side
The server is as well quite compact as shown with functionalities grouped:
Note that now the loop is over a SQL select getting the first 50 records. The variables "date", "title", "evendesc" and "id", last used in DEL_COMMAND, are filled on each iteration automatically by the LOOP command.
Finally, for those who don't have experience with databases and SQL:
We create the table with:
CREATE TABLE IF NOT EXISTS boletin (
id INTEGER PRIMARY KEY AUTOINCREMENT,
date,
title,
eventdesc);
where we specify the long type to the "id" column in order to be filled automatically by the database engine on each insertion. The rest of columns do not require data type in sqlite. By not providing types to columns when possible, we keep our schema flexible and efficient.
For insert a new event, the SQL is:
INSERT INTO boletin (date, title, eventdesc)
VALUES ('@<:encode eEventDate>',
'@<:encode eEventTitle>',
'@<:encode xEventDesc>');
Here is very important the pre-process ":encode" on every field which escapes several characters among others the quote and double quotes (' and "). You can experiment yourself what happens if for example :encode is not used in eEventTitle and we try to save an entry with title ' (one quote). The SQL will be then malformed and the insert will fail. Actually, the well known attack named SQL-Injection is based on this kind of wrong programmed SQLs.
Conclusion
Through three small scripts, we have developed some non trivial and actually useful functionalities using http protocol.
Hopefully, someone after reading the article, even without any previous knowledge about HTTP, HTML, JavaScript and gastona, can make some changes on the scripts for customization or even create a new application from them. Or at least be able to figure out how it could be done, which is actually the most important step when developing.
The samples of this article will be maintained in the github repository HttpSweetHttp for serving as reference of http applications using gastona MICO http command (server) and jGastona js library (client) either together or separately.
History
- 21st May, 2017: Initial version
