Introduction
Rendering graphics for Virtual Reality (VR) games on the spectrum of modern devices with a single code file is easy! Sound far-fetched? I’ve provided a few explanations and justifications, plus the exciting block game Tetro 2D/3D with fully commented source code as proof of concept.
Purpose
I’m pleased your advanced coding skills and inquisitive nature have brought you past the introduction. We’re here now because I've been enamored with the resurgence of what I grew up knowing as "Virtual Reality" or VR. Long rotting wayside with Alf, Max Headroom and other esoteric blips from the 80s, VR became the unfilled promise of a dawning computer age. Well, time passed and now that we're living in the future, powerful and cheap computing has prompted VR to regain footing. The Oculus Rift1 has brought awareness and appeal to the technology, and Google’s Cardboard2 has contributed innovation and accessibility. This article is meant to inspire others and drive interest for the continued revival of VR by unraveling some mystery surrounding VR graphics.
Using the Code
If you’re impatient or just don’t care about explanations or justifications, the source code exists as a single HTML file, “Tetro.htm” with CSS and JavaScript embedded. To play in 2D mode, open the file with any currently-versioned browser (Chrome recommended). To play in 3D mode, you may additionally use Cardboard or a 3D capable monitor in Side-By-Side mode. If you’d rather read or edit the code, open the file in any text editor. Easy Peasy.
Why Tetro?
Tetris creator Alexey Pajitnov undoubtedly ranks as a programming Hall-of-Famer. The simplicity of his game makes it ideal as an illustrative tool, but copying it directly would be infringement. This article instead uses a semantically different game called Tetro. This article is not intended as a programming tutorial for the game itself as there are already other Code Project articles that explore game logic. I’m assuming you already understand tetromino games3 or could unravel the mystery with existing tutorials. Instead, I chose Tetro for its slim game logic, allowing for increased visibility of the actual rendering.
Native vs. Web
Ok, so you already know that Windows, Android, Apple, etc. devices execute differently with distinct paths to native processing. This means that you need to write an application many times over in order to distribute your software to the spectrum of devices. Web applications aim to fix that by allowing programmers a universal and device-independent runtime (much like Java and the JRE). While the history shows that browsers have implemented some W3C standards with soupy compatibility and/or willful incompatibility, the overall prowess of any single web file to render equally among different browsers is excellent. Because I’m here to demonstrate the power of HTML as a viable platform for VR gaming, I’ve selected to use only pure HTML, CSS, and JavaScript. No JQuery or other wrappers used, just free-range, organic web.
Vector vs. Raster (Drawing vs. Painting)
As your experience has taught you, there are roughly two styles of graphics rendering, vector and raster. Raster (bitmapped) graphics harness 2D grids of color objects mashed in succession to create the game’s display. If the game’s resolution doesn’t change, then raster graphics may look stunningly photographic. Unfortunately, storing, transmitting, and transforming colored grids is computationally expensive and requires pre-existing assets (e.g. backgrounds, items, characters and other sprites). This expense is multiplied by the number of playable resolutions which may be large considering the range of device resolutions to market.
Valuing simplicity and portable source code, I have alternatively chosen to implement Tetro exclusively using vector graphics. Vector images are not color grids, but rather little rendering recipes (think connect-the-dots from coloring books). Essentially, a series of derivatives or points are stored with draw instructions. Taking very few bytes to express these geometries and instructions, vector processing is insanely fast and can champion even the most limited hardware (provided the device has adequate software). The geometric nature of vector processing aligns well with Tetro's blocky rendering.
A final consideration is that vector graphics scale to any resolution without the loss or mutation that occurs when resizing bitmaps. The dynamic and responsive sizing of vector graphics allows maximum compatibility without added expense. On a side note, learning to work effectively with vectors and raster cooperatively facilitates meaningful GIS/Mapping, graphing and charting within reporting systems, modern gaming and other dynamic rendering solutions.
3D vs. 3D
With new 3D technologies introduced in recent years, it’s sometimes difficult to distinguish context, especially true with graphics rendering. About a decade after the initial VR, 3D graphics came to consumer popularity with gaming consoles such as the Nintendo64 and Playstation. In addition to important depth effects as lighting and shadows from the 2D gaming realm, we were seeing objects modeled with an additional axis. Instead of confined movement along the X and Y axes by traditional side and top-scrolling games, a third axis, the Z was added. The Z axis allowed new dimensionality by adding the “near and far” component. Storing and processing game objects in 3D space had created unparalleled realism, but still lacked the other 3D.
The other 3D is what may be called stereoscopic or parallax 3D. Parallax is the angular difference between what each of your two eyeballs see when looking at something. Instead of analyzing each eyeball’s image separately, your brain streams them together and rationalizes the difference as a perception of depth. If you had more eyes, then your depth perception would become more resolute, while if you had fewer eyes, then this whole concept would be lost on you. Remember it takes at least two points of reference in order to subtract one and get a difference, which is why it takes at least two eyes to triangulate depth.
Despite use of the Z axis, 3D console gaming is analogous to how a one-eyed person perceives a 3D world because the screen itself is actually 2D. To render a true 3D simulation when actual depth is not present we require one image stream for each eyeball and a two-eyeball minimum, re-enter VR technology. With VR creating a different image stream for each eye, the technology can utilize gaming 3D and parallax 3D concurrently to create the most realistic visual experience.
VR Output
If you have a 3D television then you likely have used it to watch 3D Blu-ray discs. 3D Blu-rays (and 3D theatres) create their parallax effect by rapidly alternating left and right frames at full resolution. That is why a 3D monitor or projector must operate at a doubled refresh rate. It draws what the left eye sees and then draws what the right sees in succession, but indicating a single frame of motion.
VR employs a slightly different strategy called Side-by-Side or SBS 3D. SBS has both the left and right images drawn together in the same frame. A VR device puts a thin physical barrier between your eyes that converges on the split line. If you’ve handled a View-Master toy, then you already understand how this method works. The advantage is that smooth 3D is possible with a normal refresh rate, widening the range of capable devices.
Please note that 3D monitors and televisions commonly support SBS rendering. However, unlike current VR devices, the television hardware cuts the image in half and renders each stretched half in alternating frames. Like Blu-rays, this method also requires a doubled refresh rate because it renders left and right images in succession. An issue with this method of SBS rendering is that the final graphic is stretched at exactly twice the horizontal resolution. Unless you pre-squish the graphics to compensate, the game will appear skewed.
Calculating Parallax
The first thing to point out is that when calculating depth, we are not concerned with Y axis values. We would need eyes on our chin or forehead in order for height parallax to factor, so Y values in the left and right views are the same. Another thing to note is that all Z axis values need to be translated to X axis offset. This is necessary because VR devices and monitors in reality only have length and width (remember depth is an illusion). This means that while we can have the Z axis in our object model, the final drawing must translate Z values to the X axis.
SBS requires that you render an image twice into the same view, once for each eye. If you use two identical images for the left and right halves, then all the objects will have the same relative positioning and appear to be visually level with the device screen. To illustrate a level object, the below image shows a SBS render of a palm tree having its X coordinate (distance from left border) set to 146 pixels for both the left and right frames. As the palm tree has equal X coordinates, it will be level with the device when your brain combines the images.
However, if you render the right frame’s palm in the positive X direction, then your brain will resolve the difference as a more distant tree.
Conversely, if you render the right frame’s palm in the negative X direction, then your brain will resolve the difference as a nearer palm tree.
That’s all there is to it. The calculation of parallax is just a reasonable correlation between the X axis offset and desired perception of depth. In real life, additional factors (e.g. temperature, atmosphere, gravity, etc.) affect the calculation. However, when writing for games, a more practical matter in formulating depth is visual comfort. Ultimately, testing with an artistic rather than mathematical lens may be required to render a vertigo-free environment.
Rendering Parallax
To actually render SBS to the screen, we just loop through the drawing twice, once for each eye’s frame. During the first iteration, the left frame is drawn without consideration for 3D. When Tetro runs in 2D mode, it is actually just a rendering of the left frame centered to the screen. When in 3D mode, the right frame is also rendered and carries all the parallax offsets, generating depth. To get those offsets, we modify any X coordinate by subtracting or adding, depending on whether we want the object closer or farther. The Z coordinate in the object model should dictate the actual offset value.
Tetro demonstrates the spectrum of depth by rendering near, level, and far objects. In order to provide an example of far-away rendering, the background stars are drawn at varying degrees of distance. The game pieces, board, title and game statistics are all drawn level, and to demonstrate shallow rendering, a starburst appears when lines are cleared from the board.
Additional Considerations
There were a number of optimizations not taken to keep maintain code simplicity. For example, there is absolutely no reason to redraw the starry background every time. Each star’s position is unnecessarily recalculated and redrawn when no change has occurred. Efficiency would be served to use a second canvas to render the game’s static elements, and then layer it under the canvas with highly dynamic elements. Additionally, you’ll see a large number of calculations inside the Draw procedure used to support dynamic sizing and layout. In your own simulations, you might consider storing calculations supporting a responsive UI (e.g. after a page resize) in a more global scope and only calculated once after the summoning event.
While the article focus is rendering, I wanted to quickly address how audio was incorporated into this project. While browsers are fully capable of supporting all the auditory bells and whistles to keep your game humming along, you’ll discover adding them may require unexpected fiddling and cursing. There is not a single audio format that translates to all browsers, and any one format may have mixed compatibility within the same browser depending on encoding and codec versioning. To help with compatibility issues, the Audio object in JavaScript specifies a function which facilitates compatibility testing of audio formats with the client’s browser. To test for mp3-ability, you would pass “audio/mpeg” as the type and any non-compatible browser will return an empty string. However, if the browser is not-not capable, then either “probably” or “maybe” is returned, neither of which is definitive. Anyways, despite squishy audio support among browsers, Tetro demonstrates use of embedded WAV and MP3 types along with JavaScript to serve sound effects while keeping resource dependencies at zero.
A last consideration is that programming games for VR carries tactics not explored in this article. Control by position sensing and collision detection under parallax are two such examples. Perhaps additional VR programming strategies can be addressed in a future article. Thanks for reading and good luck with your projects!
Additional Descriptions
History
