Rethinking the Web - Part 1

Preface

In the mid 1990s, software development was reasonably predictable. The Client-Server model was well established, methodologies were solid, and applications were fast and (in an industrial way) pretty. Like the peak of the Roman Empire, things were good. Then, the web swept in like the barbarian hordes, with its primitive scripts and HTML. While users wept about performance and quality (just Wayback Machine to web "apps" in the early 90's), management did nothing. Why?

The initial explosion of the web in the enterprise was really based on a single factor. Distribution. At the time, every update to every application required a trip to the users desk with a CD.

As the web was taking form and approaches being solidified, enterprises were consumed with Y2K Remediation. Everyone else was trying to get on the web, be found there, and keeping users from clicking away. Tools and methodologies supported this, and the technology core of the web was built around CSS and SEO.

Things have changed. SEO is mostly pointless now. Crawlers are going to index you pretty well anyway, but search results are now Pay to Play for your site to show up. CSS has become pervasive and invasive. It could be the Jeopardy response to, "It laughs at the Single Responsibility Principle".

It has taken over so much that there are people out there who actually think the `class` attribute on an element means CSS Class!

In spite of all of this (and more), implementors and developers are doing some awesome things. The Document based approach works well for Informational, Promotional, and Catalog type sites - they have paper document analogs. But the model isn't getting enterprise apps back to their former glory.

Add to this our Object Model. During the 60s and 70s, two general approaches to Object Orientation emerged. One thought of objects as "things", the other thought of objects as "beings". The mainstream way we see object today is neither. It is a variation of the "things" approach, where things are like RDBMS tables. The Association in OO Theory is implemented as a DB style relationship.

And like many things, the software is packed with dogma and dogma spouting parrots. Things like DRY, SOLID and a host of others are just accepted. Are they still valid in the 20 years later world?

I started a "back to basics" project, with a focus on the needs of the enterprise. I took the tools, techs, and knowledge we have today back to the 60s and started over.

These articles share what I've learned and created.

Introduction

This article provides an overview of WebSockets and shows how to remote control browsers over sockets. The example uses a Windows Console to send and execute code on the browser.

The Parts

Browsers work in a Request-Response mode. But they can also work in a Solicit-Response mode. A basic difference between these is direction. Clients initiate Requests, Servers initiate Solicits. Another difference is control. In an RR application, the browser/client controls the workflow. In an SR application, that is inverted - the server is in control. Using Solicit-Response, we can remote control our browsers.

HTTP doesn't support solicits, but WebSockets can.

WebSockets

Creating a socket is easy: (script)

new WebSocket(location.origin.replace("http", "ws"));

Sockets are same-origin, different protocol, so we can use a little replace hack.
This sends an HTTP GET with some socket related information in the header .

On the server side, it's just as easy: (C# concept code - don't.)

MapGet("/", async (HttpContext http) => await http.WebSockets.AcceptWebSocketAsync( ));

The server (in C#) creates an object (HttpContext) with a WebSockets interface.
Note: I will be using interface in the common way. If I mean a C# Type (IWhatever), I will specify.
The Accept method sends a transparent response to the browser that triggers the onopen event of the socket.

An open socket can't really do much. It needs to be wired into the system. If the server started sending messages right away, they would essentially go nowhere. No one is listening. We need listeners. And, we need to know we have listeners.

self.socket = new WebSocket(location.origin.replace("http", "ws"));
socket.onopen = () => {
  socket.onmessage = () => {...});
  // Other stuff
  socket.send("ready");
  }

In the onopen handler, we can attach an onmessage handler, and do any other initialization work. When finished, we let the server know all is well - "ready".

Sockets on the server are a little different than we might expect from C# objects. While many messaging components "push" (e.g., Events/Delegates). We need to explicitly wait on sockets.
Note: A current issue with sockets is you can't stop waiting. Canceling kills the connection.

WebSocket socket = await http.WebSockets.AcceptWebSocketAsync();
var buffer = new byte[1024];
WebSocketReceiveResult ack = await socket.ReceiveAsync(buffer, CancellationToken.None);

The "ack" object is a status object. The data/message is on the buffer, and easy to get as text.

string msg = Encoding.UTF8.GetString(buffer[...ack.Count]);

There is one thing left to do. Sockets are like nested connections - a socket connection inside an HTTP connection. If we don't do anything, the connections will fall though or time out. The infrastructure handles HTTP with built in "keep-alive". The HTTP part is basically unused, we tell it to wait for a response and never send one. The "keep-alive" extends the lie (over and over) - "Just 30 sec more..." (Cruel).

While the HTTP connection is open, a WebSocket connection can live inside. By default, a socket will close on its own after sending a message. We need to keep it open too. (e.g., By looping over the receive waiter).

That's really about 90% of WebSockets. The rest is variations of this (i.e., binary messages) or plumbing.

Browser Tasks

The browser "JavaScript Virtual Machine" (JVM) that runs JavaScript is an interesting beast. While it has been improved and augmented, the basic architecture hasn't changed since V1. Internally, it appears much like a Motorola 6800/68000 CPU from the 1970/80s. Programming takes me back to BASIC (not in syntax, but approach) and batch files in DOS or Mainframe schedulers. Not bashing. Those are all good techs, but slightly outdated in this age of distributed asynchronous computing. This topic is explored in other articles.

What is relevant now is that browser code is interpreted. Ignoring any optimizations, an Interpreter knows nothing about interpreted code until it sees it. It doesn't matter when or how code gets there, as long as it is there when needed. A perfect environment for a Just In Time approach.

Script supports Functions (function(){ }), which can be asynchronous. The async function constructor isn't surfaced on its own. This code will build and run an async or "standard" function:

self.AsyncFunction = Object.getPrototypeOf(async function () { }).constructor;
self.Execute = async (cmd) => await (new AsyncFunction(cmd))() ?? "Success"

This runs any valid JavaScript code. If the code return a value, Execute returns that value. "Success" is returned if it completes with no errors and no return value.

Plugged in the socket code:

self.AsyncFunction = Object.getPrototypeOf(async function () { }).constructor; 
self.Execute = async (cmd) => await (new AsyncFunction(cmd))() ?? "Success"

self.socket = new WebSocket(location.origin.replace("http", "ws"));
socket.onopen = async () => {
    socket.send("ack");
    socket.onmessage = async (msg) => {
        try { socket.send(await Execute(msg.data)); }
        catch (e) { socket.send(`Fail: ${e}`); }
        }
    };

This builds a function from a message "Just in Time", executes it, and sends the result back through the socket. It operates (ignoring optimizations) exactly like a function created on page load.

The server side uses the send and wait lines:

string cmd = "return 4+5;"
var buffer = new byte[1024];
_ = socket.SendAsync(new(UTF8.GetBytes(cmd)), Text, true, _ct.None);
var result = await socket.ReceiveAsync(buffer, _ct.None);
string data = Encoding.UTF8.GetString(buffer[..result.Count]);

Variable data will equal "9". Leaving off the return ("4+5") will put "Success" in data.

With this approach, we can send tasks (in Script) to the browser from the server.

Server

To support this stuff, we need a web server. The relevant server code right now is:

_ = app.MapGet("/", (HttpContext http) => http.WebSockets.IsWebSocketRequest switch {
    false => http.Response.WriteAsync("<!DOCTYPE html><html>%SocketCode%</html>"),
    true => Task.Run(async () => {
        WebSocket socket = await http.WebSockets.AcceptWebSocketAsync();
           ....

HTTP and WebSocket requests can map to the same url. The socket interface on HttpContext provides an IsWebSocketRequest property to differentiate. We can simply switch on this.
When true, we start the socket process. When false, a standard HTML Document. Or something else.

Browsers are pretty tolerant. If we don't provide a doc, it will create one. All we care about from the HTTP request is that we get the socket code run. We can put the document in with:

self.SetDocument = (content) => {
       const doc = document.open(); doc.write(content); doc.close(); }

We can just send (made easy with raw string literals):

http.Response.WriteAsync($$"""
  <script>
    self.AsyncFunction = Object.getPrototypeOf(async function () { }).constructor;
    self.Execute = async (cmd) => await (new AsyncFunction(cmd))() ?? "Success"
    self.SetDocument = (content) => { const doc = document.open(); 
         doc.write(content); doc.close(); }
    self.SetSheet = (content) => { const s = new CSSStyleSheet(); s.replace(content);
         document.adoptedStyleSheets = [s]; }
    self.socket = new WebSocket(location.origin.replace("http", "ws"));
    socket.onopen = async () => {
      socket.send("ack");
      socket.onmessage = async (msg) => {
        try { socket.send(await Execute(msg.data)); }
        catch (e) { socket.send(`Fail: ${e}`); }
        }
    };
  </script>
  """);

Replacing the document through code isn't like an HTTP GET. It doesn't reset the window or execution context. It only replaces what is between the HTML tags. Everything we mount in the EC self.Thing = ... stays, and is available for the new document.

Not only can we replace documents and parts dynamically, we can do this with CSS. Notice the SetSheet function. We can take "full control" of a browser putting in or removing and CSS, Markup, or Script in realtime.

Important Concept Point

With this approach, we can control the browser execution context and content in real time.

There never needs to be "just in case" code, markup, or pages of CSS. The only code that needs to be in the browser is the currently visible artifacts. Code can be "fabricated" (another article) in real-time to fit specific conditions and states.

All Together Now...

Here is a working application. It uses a Windows Console to control a browser.
If your IDE starts, a browser turn that off. The code is .net7 + raw string literals.

In VS, Project File and Launch:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net7.0</TargetFramework>
    <Nullable>enable</Nullable>
    <LangVersion>preview</LangVersion>
  </PropertyGroup>
</Project>

{
  "profiles": {
    "http": {
      "commandName": "Project",
      "launchBrowser": false,
      "applicationUrl": "http://localhost:5150"
    }
  }
}

The code is as follows:

using System.Threading;
namespace CP;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using System;
using System.Diagnostics;
using System.Net.WebSockets;
using System.Threading.Tasks;
using static System.Net.WebSockets.WebSocketMessageType;
using static System.Text.Encoding;
using _ct = CancellationToken;

public class Program {
    public static void Main(string[] args) {

        int port = 5085;
        Barrier barrier = new(2);
        WebApplicationBuilder builder = WebApplication.CreateBuilder();
        builder.WebHost.PreferHostingUrls(true).ConfigureKestrel
                             (s => s.ListenLocalhost(port));
        WebApplication app = (WebApplication)builder.Build().UseWebSockets();

        _ = app.MapGet("/", (HttpContext http) => 
                             http.WebSockets.IsWebSocketRequest switch {

            false => http.Response.WriteAsync($$"""
              <script>
                self.AsyncFunction = Object.getPrototypeOf
                                     (async function () { }).constructor;
                self.Execute = async (cmd) => 
                               await (new AsyncFunction(cmd))() ?? "Success"
                self.SetDocument = (content) => { const doc = document.open();
                  doc.write(content); doc.close(); }
                self.SetSheet = (content) => 
                     { const s = new CSSStyleSheet(); s.replace(content);
                  document.adoptedStyleSheets = [s]; }
                self.Read = (facet, field) => document.querySelector
                            (`[facet="${facet}"][field="${field}"]`).value;
                self.Write = (facet, field, v) => 
                document.querySelector
                         (`[facet="${facet}"][field="${field}"]`).value = v;
                self.socket = new WebSocket(location.origin.replace("http", "ws"));
                socket.onopen = async () => {
                  socket.send("ack");
                  socket.onmessage = async (msg) => {
                    try { socket.send(await Execute(msg.data)); }
                    catch (e) { socket.send(`Fail: ${e}`); }
                    }
                  };

                self.NicheNode = class NicheNode extends HTMLElement 
                                 { #content = null; #name = null;
                  static Replace(niche, content) { 
                    const n = document.querySelector(`layout-niche[niche="${niche}"]`)
                    .replaceWith(new NicheNode(niche, content)); }

                  constructor(name, content) { super(); 
                     if (content) this.#content = content; 
                                  if (name) this.#name = name; }

                  connectedCallback() {
                    if (this.#content != null) this.innerHTML = this.#content;
                    if (this.#name != null) this.setAttribute("niche", this.#name);
                    }
                  };
                customElements.define("layout-niche", NicheNode);

              </script>
              """),

            true => Task.Run(async () => {
                WebSocket socket = await http.WebSockets.AcceptWebSocketAsync();
                var buffer = new byte[1024];
                WebSocketReceiveResult ack = 
                         await socket.ReceiveAsync(buffer, _ct.None);
                barrier.SignalAndWait();

                _ = socket.SendAsync(new(UTF8.GetBytes($$"""
                  SetDocument(`
                  <!DOCTYPE html><html lang="en">
                    <head><title>Wisp</title>
                    </head>
                    <body>
                      <area-left>
                        <layout-niche niche="left"></layout-niche>
                      </area-left>
                      <area-right>
                        <layout-niche niche="left">
                          <input type="text" facet="person" 
                          field="name" value="" />
                        </layout-niche>
                      </area-right>
                    </body>
                  </html>
                  `);
                  """)), Text, true, _ct.None);
                _ = await socket.ReceiveAsync(buffer, _ct.None);

                _ = socket.SendAsync(new(UTF8.GetBytes($$"""
                  SetSheet(`
                  body { display:flex; gap:10px; }
                  area-left, area-right { display:flex; flex-direction:column; 
                      border: 1px solid black; min-height: 200px; }
                  area-left { flex:1; }
                  area-right { flex:2; }
                  `);
                  """)), Text, true, _ct.None);
                _ = await socket.ReceiveAsync(buffer, _ct.None);

            Loop:
                string cmd = "";
                Console.Write(">");
                if( (cmd = Console.ReadLine() ?? throw new()) == "" ) goto Loop;
                await socket.SendAsync(new(UTF8.GetBytes(cmd)), Text, true, _ct.None);
                var status = await socket.ReceiveAsync(buffer, _ct.None);
                Console.WriteLine(UTF8.GetString(buffer[..status.Count]));
                goto Loop;
            })
            });

        Task.Factory.StartNew(async () => { await app.StartAsync(); });
        _ = Process.Start("explorer", $"http://localhost:{port}");
        barrier.SignalAndWait();
        _ = new AutoResetEvent(false).WaitOne();
        }
    }

You should see a ">" prompt in the Windows Console.

Try these commands:

1. a return 4+5;
   b 4+5
2. NicheNode.Replace("left", `<h3>Replacement</h3>`);
3. SetSheet(`body { background-color: blue; } `);

   This blows away the old styling. In a future article, I cover Sheet Management.
   Just restore it.

4. SetSheet(` body { display:flex; gap:10px; } area-left, area-right { display:flex; flex-direction:column; border: 1px solid black; min-height: 200px; } area-left { flex:1; } area-right { flex:2; } `);

Solicit

In a Request-Response model, we wait for a user to post/submit and they send some predefined data. In a Solict-Response model, we just go get what we want, when we want.

5. Write("person", "name", "Gwyll");
6. Read("person", "name");

Closing

In the next article, I will expand on this foundation.
I hope you found this interesting.

History

• 17^th September, 2022: Initial version