Basic concepts on Endianness

Introduction

A long time ago, in a very remote island known as Lilliput, society was split into two factions: Big-Endians who opened their soft-boiled eggs at the larger end ("the primitive way") and Little-Endians who broke their eggs at the smaller end. As the Emperor commanded all his subjects to break the smaller end, this resulted in a civil war with dramatic consequences: 11.000 people have, at several times, suffered death rather than submitting to breaking their eggs at the smaller end. Eventually, the 'Little-Endian' vs. 'Big-Endian' feud carried over into the world of computing as well, where it refers to the order in which bytes in multi-byte numbers should be stored, most-significant first (Big-Endian) or least-significant first (Little-Endian) to be more precise [1]

• Big-Endian means that the most significant byte of any multibyte data field is stored at the lowest memory address, which is also the address of the larger field.
• Little-Endian means that the least significant byte of any multibyte data field is stored at the lowest memory address, which is also the address of the larger field.

For example, consider the 32-bit number, 0xDEADBEEF. Following the Big-Endian convention, a computer will store it as follows:

Figure 1. Big-Endian: The most significant byte is stored at the lowest byte address.

Whereas architectures that follow the Little-Endian rules will store it as depicted in Figure 2:

Figure 2. Little-endian: Least significant byte is stored at the lowest byte address.

The Intel x86 family and Digital Equipment Corporation architectures (PDP-11, VAX, Alpha) are representatives of Little-Endian, while the Sun SPARC, IBM 360/370, and Motorola 68000 and 88000 architectures are Big-Endians. Still, other architectures such as PowerPC, MIPS, and Intel�s 64 IA-64 are Bi-Endian, i.e. they are capable of operating in either Big-Endian or Little-Endian mode. [1].

Endianess is also referred to as the NUXI problem. Imagine the word UNIX stored in two 2-byte words. In a Big-Endian system, it would be stored as UNIX. In a little-endian system, it would be stored as NUXI.

Which format is better?

Like the egg debate described in the Gulliver's Travels, the Big- .vs. Little-Endian computer dispute has much more to do with political issues than with technological merits. In practice, both systems perform equally well in most applications. There is however a significant difference in performance when using Little-Endian processors instead of Big-Endian ones in network devices (more details below).

How to switch from one format to the other?

It is very easy to reverse a multi-byte number if you need the other format, it is simply a matter of swapping bytes and the conversion is the same in both directions. The following example shows how an Endian conversion function could be implemented using simple C unions:

unsigned long ByteSwap1 (unsigned long nLongNumber)
{
   union u {unsigned long vi; unsigned char c[sizeof(unsigned long)];}; 
   union v {unsigned long ni; unsigned char d[sizeof(unsigned long)];};
   union u un; 
   union v vn; 
   un.vi = nLongNumber; 
   vn.d[0]=un.c[3]; 
   vn.d[1]=un.c[2]; 
   vn.d[2]=un.c[1]; 
   vn.d[3]=un.c[0]; 
   return (vn.ni); 
}

Note that this function is intented to work with 32-bit integers.

A more efficient function can be implemented using bitwise operations as shown below:

unsigned long ByteSwap2 (unsigned long nLongNumber)
{
   return (((nLongNumber&0x000000FF)<<24)+((nLongNumber&0x0000FF00)<<8)+
   ((nLongNumber&0x00FF0000)>>8)+((nLongNumber&0xFF000000)>>24));
}

And this is a version in assembly language:

unsigned long ByteSwap3 (unsigned long nLongNumber)
{
   unsigned long nRetNumber ;

   __asm
   {
      mov eax, nLongNumber
      xchg ah, al
      ror eax, 16
      xchg ah, al
      mov nRetNumber, eax
   }

   return nRetNumber;
}

A 16-bit version of a byte swap function is really straightforward:

unsigned short ByteSwap4 (unsigned short nValue)
{
   return (((nValue>> 8)) | (nValue << 8));

}

Finally, we can write a more general function that can deal with any atomic data type (e.g. int, float, double, etc) with automatic size detection:

#include <algorithm> //required for std::swap


#define ByteSwap5(x) ByteSwap((unsigned char *) &x,sizeof(x))

void ByteSwap(unsigned char * b, int n)
{
   register int i = 0;
   register int j = n-1;
   while (i<j)
   {
      std::swap(b[i], b[j]);
      i++, j--;
   }
}

For example, the next code snippet shows how to convert a data array of doubles from one format (e.g. Big-Endian) to the other (e.g. Little-Endian):

double* dArray; //array in big-endian format

int n; //Number of elements


for (register int i = 0; i <n; i++) 
   ByteSwap5(dArray[i]);

Actually, in most cases, you won't need to implement any of the above functions since there are a set of socket functions (see Table I), declared in winsock2.h, which are defined for TCP/IP, so all machines that support TCP/IP networking have them available. They store the data in 'network byte order' which is standard and endianness independent.

Table I: Windows Sockets Byte-Order Conversion Functions [2]

The socket interface specifies a standard byte ordering called network-byte order, which happens to be Big-Endian. Consequently, all network communication should be Big-Endian, irrespective of the client or server architecture.

Suppose your machine uses Little Endian order. To transmit the 32-bit value 0x0a0b0c0d over a TCP/IP connection, you have to call htonl() and transmit the result:

TransmitNum(htonl(0x0a0b0c0d)); 

Likewise, to convert an incoming 32-bit value, use ntohl():

int n = ntohl(GetNumberFromNetwork()); 

If the processor on which the TCP/IP stack is to be run is itself also Big-Endian, each of the four macros (i.e. ntohs, ntohl, htons, htonl) will be defined to do nothing and there will be no run-time performance impact. If, however, the processor is Little-Endian, the macros will reorder the bytes appropriately. These macros are routinely called when building and parsing network packets and when socket connections are created. Serious run-time performance penalties occur when using TCP/IP on a Little-Endian processor. For that reason, it may be unwise to select a Little-Endian processor for use in a device, such as a router or gateway, with an abundance of network functionality. (Excerpt from reference [1]).

One additional problem with the host-to-network APIs is that they are unable to manipulate 64-bit data elements. However, you can write your own ntohll() and htonll() corresponding functions:

• ntohll: converts a 64-bit integer to host byte order.
• ntonll: converts a 64-bit integer to network byte order.

The implementation is simple enough:

#define ntohll(x) (((_int64)(ntohl((int)((x << 32) >> 32))) << 32) | 
                     (unsigned int)ntohl(((int)(x >> 32)))) //By Runner

#define htonll(x) ntohll(x)

How to dynamically test for the Endian type at run time?

As explained in Computer Animation FAQ, you can use the following function to see if your code is running on a Little- or Big-Endian system:

#define BIG_ENDIAN      0
#define LITTLE_ENDIAN   1

int TestByteOrder()
{
   short int word = 0x0001;
   char *byte = (char *) &word;
   return(byte[0] ? LITTLE_ENDIAN : BIG_ENDIAN);
}

This code assigns the value 0001h to a 16-bit integer. A char pointer is then assigned to point at the first (least-significant) byte of the integer value. If the first byte of the integer is 0x01h, then the system is Little-Endian (the 0x01h is in the lowest, or least-significant, address). If it is 0x00h then the system is Big-Endian.

Similarly,

bool IsBigEndian()
{
   short word = 0x4321;
   if((*(char *)& word) != 0x21 )
     return true;
   else 
     return false;
}

which is just the reverse of the same coin.

You can also use the standard byte order API�s to determine the byte-order of a system at run-time. For example:

bool IsBigEndian() { return( htonl(1)==1 ); }

Auto detecting the correct Endian format of a data file

Suppose you are developing a Windows application that imports Nuclear Magnetic Resonance (NMR) spectra. High resolution NMR files are generally recorded in Silicon or Sun Workstations but recently Windows or Linux based spectrometers are emerging as practical substitutes. It turns out that you will need to know in advance the Endian format of the file to parse correctly all the information. Here are some practical guidelines you can follow to decipher the correct Endianness of a data file:

1. Typically, the binary file includes a header with the information about the Endian format.
2. If the header is not present , you can guess the Endian format if you know the native format of the computer from which the file comes from. For instance, if the file was created in a Sun Workstation, the Endian format will most likely be Big-Endian. 
3. If none of the above points apply, the Endian format can be determined by trial and error. For example, if after reading the file assuming one format, the spectrum does not make sense, you will know that you have to use the other format.

If the data points in the file are in floating point format (double), then the _isnan() function can be of some help to determine the Endian format. For example:

double dValue;
FILE* fp;
(...)
fread(&nValue, 1, sizeof(double), fp);
bool bByteSwap = _isnan(dValue) ? true : false

Note that this method does only guarantee that the byte swap operation is required if _isnan() returns a nonzero value (TRUE); if it returns 0 (FALSE), then it is not possible to know the correct Endian format without further information.

Acknowledgments

Thanks to Santiago Dom�nguez, Ehsan Akhgari, Santiago Fraga and Ignacio Sordo for their helpful suggestions.

References

1. Introducction to Endianness, by Michael Barr, Embedded Systems Programming.
2. Visual C++ Concepts: Adding Functionality. Windows Sockets: Byte Ordering

Further reading

• ON HOLY WARS AND A PLEA FOR PEACE by Danny Cohen.
• For a full technical discussion of software Endianness issues, refer to the Sun white paper, "Endianness in Solaris".
• A list of graphics common file formats and their Endian order can be found here.