Trailing Commas in JavaScript

The Scenario

You’re working on a JavaScript project, and your project includes the following object definition:

var deckOfCards = {
    deal : function() {
        // TODO: add dealing logic
    }
};

You go to add another method to your object.

var deckOfCards = {
    deal : function () {
        // TODO: add dealing logic
    }

    shuffle : function () {
        // TODO: add shuffling logic
    }
};

You return to your test page and get mysteriously unexpected behavior. What happened?

You’ve probably recognized the mistake in the above code, but then if you write much JavaScript, there’s a good chance you’ve made the same mistake numerous times, and you’ll probably make it again. Maybe it’s because of our experience with other languages, or maybe it’s just one of those small details, but we often forget to separate JavaScript object members with commas, especially when adding a new method to the end of a class.

So maybe you think to yourself, “You know, most browsers will tolerate a comma after the last member in the object.” You might train yourself to always put a comma after each member in an object, so that you won’t have this problem going forward. Then you would write:

var deckOfCards = {
    deal : function () {
        // TODO: add dealing logic
    },

    shuffle : function () {
        // TODO: add shuffling logic
    },
};

Then you might go on happily building your application, with its ever-growing code base, until one day your client says “Oh, by the way, it’s vitally important that this run in IE8.”

You think back on your observation that most browsers will accept a comma after the last member of a JavaScript object, only to reflect that IE8 is not in that particular majority. Now what?

Now you have a bunch of tiny, hard-to-find bugs scattered throughout a large code base, and you need to find and remove them. To make matters worse, they’re going to blend into the background. They look like valid syntax, especially if you’ve been training yourself to put them in the code on purpose. If I had a worse sense of humor, I’d call them comma chameleons. (Oh, look, I do have a worse sense of humor. Must be a byproduct of the long debugging sessions.)

Going forward, you might adopt a different convention in lieu of trailing commas. You could use the SQL convention of putting commas at the beginning of the new line, on the theory that you edit the end of a list more often than the beginning. Maybe you could always put a dummy element at the end of an object definition so that every “real” element can have a trailing comma with no issues. But right now, you just need a way to fix a mountain of existing code.

Our Solution

When our team encountered this problem, we felt that Perl might hold a solution. Indeed, a quick Google search reveals that others have fallen into the “trailing comma” trap, and some have provided RE strings that will find some of the trailing commas, some of the time. What follows is a somewhat more comprehensive script.

You can download the script here; if you’d like a step-by-step explanation of what the script will do, read on! (Note: the code as it appears below has been modified for readability; only the original version, as found via the previous link, has been tested.)

Before I get into the nuts and bolts, two caveats:

• First, I said “somewhat more comprehensive.” What we’re looking for here is a good return on investment; it didn’t make sense to write a full-fledged JavaScript parser, which means some evil genius might find a way to hide a trailing comma that this script won’t catch. It should catch the cases that are likely to show up in real code.
• Second, the error handling in this script is nearly nonexistent. Feel free to beef it up if you’d like.

With those caveats in mind, the script only finds the problems and tells you where they are; it does not attempt to repair the code for you.

Step 1: Find JavaScript files to process

Somehow we have to tell the script what file(s) to process. We could accept a filename on the command line and then rely on OS utilities to traverse our project tree and execute the script for each source file it locates. But directory traversal isn’t very hard in Perl anyway, so let’s allow our command line arguments to include directories to be recursively searched.

sub process
{
    my ($item) = @_;
    print "$item…\n";

    # directory to be searched recursively?
    search($item) if –d $item;

    # …or file to be scanned?
    scan($item) if –f $item;
}

sub search
{
    my ($item) = @_;
    my $d, @f;

    # Get the list of files/directories in the directory
    opendir($d, $item);
    @f = readdir $d;
    closedir $d;

    # Process each one except the . and .. links
    foreach $f (@f) {
        next if $f eq "." || $f eq "..";
        proc("$item/$f");
    }
}

sub scan
{
    # TODO
}

# Functions declared; now process each command-line arg
foreach $nxt (@ARGV)
{
    proc($nxt);
}

Now the scan routine will be called for each file named on the command line, or found in a directory named on the command line (or a subdirectory thereof, recursively). So what should the scan function do?

Step 2: Strip Comments

One easy way to get a trailing comma is to comment out the last method in an object. We want our script to handle that case, and we suspect there might be other times when comments get in the way.

We expect to find two kinds of comment: single-line comments (//…) and block comments (/*…*/). Single-line comments alone would be pretty simple. Multi-line comments are somewhat more complex. The possible combinations of the two can get downright ugly. We’ll make some simplifying assumptions for now, so if your code is likely to say things like:

/* single line comments start with // */ ,

…then you may need to add some more robust parsing logic. At a minimum, we’ll handle cases without nested comment markers correctly.

We’ll declare $comment and set it to 0; this will be a boolean indicator of whether we’re in the middle of a multi-line comment; any time a line contains a /* with no matching */, we’ll set $comment to 1. Then we ignore subsequent lines until we find one that does contain a */.

For those lines we’re not ignoring, we just apply a little substring and/or regex logic to remove the comments that are present in the line. We don’t modify $_ itself because we want to preserve the original file contents for our output; so we use the $nocomment variable.

We start the body of the scan function like this:

my ($item) = @_;

my $comment=0;  #is this line part of a multi-line comment?
my $comma=0;  #did the last nonblank line end with a comma?
my $block;      #potential output
my $nocomment;  #input line with comments stripped out
my $f, $i;

open($f, "<$item");
while(<$f>)
{
    $nocomment = $_;

   if ($comment) {
       #skip lines until we see an end-of-comment marker
       #then remove everything up to the marker (inclusive)
       $i = index $nocomment, '*/';
       next unless $i > -1;
       $nocomment = substr $nocomment, $i+2;
       $comment = 0;
   }

   # remove comments that are entirely on this line
   $nocomment =~ s/\/\/.*//;
   $nocomment =~ s/\/\*.*?\*\///g;

   # check if a comment spills over onto the next line
   $i = index $nocomment, '/*';
   if ($i > -1) {
       $comment = 1;
       $nocomment = substr $nocomment, 0, $i;
   }

   # TODO: detect trailing commas
}

close $f;

Note the importance of order here: We remove all instances of /*…*/ from the line, and after that is done if we still see a /* we know that a comment will carry over to subsequent lines.

Step 3: Detect Trailing Commas

We might think of a trailing comma as the string “,}” or “,]”. Of course, there could be whitespace (likely including newlines) between the comma and the closing brace. (There could be comments, too, but we’ve removed those.)

If we find a comma followed by a closing brace, we’ll output the line number and line contents. (We output the filename when we started processing the file.) If we find a comma at the end of a line, then we’ll scan subsequent lines until we know whether it is or is not a trailing comma. (The $comma variable will serve much the same purpose here as the $comment variable did for multi-line comments.)

So the last #TODO is replaced as follows:

if( $comma )
{
    # We’re resolving a , from the end of a previous line.
    # Add this line to the block of potential output.
    $block .= "  $.: $_";

    # If this isn’t a blank line, we’ll resolve the comma
    if ($nocomment !~ /^\s*$/)
    {
        $comma = 0;
    }
    # If the line starts with ] or }, it’s a trailing comma
    if ($nocomment =~ /^\s*[\]}]/)
    {
        print "$block\n";
    }
}

# look for trailing commas on this line
if( $nocomment =~ /,\s*[\]}]/ )
{
    print "  $.: $_\n" ;
}

# see if this line ends with a ,
if( $nocomment =~ /,\s*$/ )
{
    $comma = 1;
    $block = "  $.: $_";
}

And that’s should do it. Again, there are many ways to improve this script – add error handling, parse the JavaScript more thoroughly, etc. But this is a good “bang for your buck” implementation that shouldn’t produce false positives and will likely catch trailing commas unless someone’s intentionally trying to hide them.

In addition to after-the-fact bug hunting, a tool like this can be used proactively. For example, our team is considering putting a pre-commit hook in place so that code with trailing commas won’t make it into source control in the first place.

Happy Hunting!

– Mark Adelsberger, asktheteam@keyholesoftware.com