Pellucid

Tips and Tricks for Faster Front-End Builds

Written by Michael Martin-Smucker on September 23, 2014

Background

Let's be honest, for most of us in the front-end world, setting up a build tool consists of finding a previous project with a similar structure, copying the Gruntfile or Gulpfile, and tweaking some minor details, such as folder names, to better suit the new project.

Even migrating from a task runner like Grunt to a newer build tool like Gulp is often an exercise in searching NPM for gulp- equivalents of grunt-contrib- plugins. Previously used grunt-contrib-stylus? Try gulp-stylus. Looking for grunt-contrib-jshint? gulp-jshint probably does what you want. Beyond that, it's just a matter of converting syntax.

Setting up a build system this way is quick, requires almost no thought, and works well enough for small projects. But after working at Pellucid for the past several months — where our app is easily the largest JavaScript project I've worked on, and our incremental build time is creeping closer and closer to 10s — it seems like an appropriate time to explore optimizations.

Introducing our Sample Project

As a reduced test case for experimentation, I set up a sample repository. The project uses jQuery, Lo-dash, and Handlebars, along with 50-or-so custom CommonJS modules (bundling to half a megabyte) to give Browserify something to work with.

In this repository, our ideal build system will do the following (doubly ideal would be doing these things quickly... we'll get there).

  • Preprocess, prefix, and minify CSS
  • Lint/Hint, Browserify, and Uglify our JavaScript
  • Watch for changes and re-run any appropriate steps

Our build process at Pellucid has a few more requirements than this, but these should cover some of the longest operations, and this sample project should be generic enough to apply to many front-end projects.

Speeding Things Up

A naïve approach might result in a Gulpfile that looks something like this. Pretty straightforward; we've taken our laundry list of compilation tasks and broken them down into Gulp tasks. Easy to write, easy to read, and it gets the job done.

However, even in our relatively small sample app, this takes just over 4 seconds on my laptop (timed with time gulp). That's acceptable for an initial compile, but with our watch task recompiling after every change, this adds noticeable latency to our make-a-change/test-that-change feedback loop.

Let's hit some of slowest operations and see what we can do to speed things up:

Browserify with Watchify

Clocking in at nearly 3s, bundling our modules with Browserify is by far the most time-consuming operation in our build process. Luckily it's also one of the simplest to improve.

The easiest win involves switching our watch task to use Watchify. Written by the same author as Browserify, Watchify has a built-in watch mechanism that only re-bundles the files that need to be rebundled. Gulp already has a recipe for bundling with watchify so let's use that as our starting point.

Our js task will still use Browserify (so that we can compile our JavaScript without starting any watching), but let's switch from gulp-browserify, which has been blacklisted by the Gulp developers, to pure Browserify.

Note: gulp-browserify locks down the Browserify version at 3.x. By using Browserify directly, we're free to use the latest 5.x release, which handles configuration differently.

gulp.task('js', function () {
  var browserify = require('browserify'),
      source     = require('vinyl-source-stream');

  return browserify('./js/main.js', {debug: true})
    .bundle()
    .pipe(source()) // convert to a stream gulp understands
    // ... continue with uglify
});

And for our watch task, instead of using Gulp's watching, we'll use Watchify, which handles file system monitoring:

gulp.task('watch', ['stylus'], function () {
    var watchify   = require('watchify'),
        browserify = require('browserify'),
        bundler    = watchify(browserify('./js/main.js', {
            cache: {},
            packageCache: {},
            fullPaths: true,
            transform: ['hbsfy'],
            debug: true
        }));

    function rebundle() {
        return bundler.bundle()
            .pipe(source('main.js'))
            .pipe(gulp.dest('./dist'));
    }
    bundler.on('update', rebundle);
    // run any other gulp.watch tasks

    return rebundle();
});

Most of the new options that we're passing to browserify() are required by Watchify. You may notice we've stopped Uglifying in our watch task. Since we'll only be producing development builds with this task, there's no point slowing things down with minification.

In our sample project, the incremental build triggered by saving JS files has dropped from 3-4s down to roughly 200ms with Watchify. At Pellucid, this change reduced bundling time from 7 seconds to 1. This improvement alone would probably be enough to quit optimizing and feel proud of ourselves, but there's still more we can do.

Additional Browserify Optimizations

If your project depends on big libraries that don't require() any other modules, Browserify doesn't know this, and will check the entire file for require statements anyway, which can be time-consuming. You can speed things up by passing an array of modules that Browserify shouldn't parse:

browserify('./js/main.js', {
    noparse: ['jquery', 'lodash', 'q']
    // other options
});

While this won't do much for the incremental builds, it shaves 200-300ms off our initial build. Do note, however, that if you use browserify-shim and you're shimming a plugin, you can't use noparse for the plugin's dependencies.

Finally, if you use Browserify to generate a standalone build -- one that can be used outside of a CommonJS context -- make sure you are using the latest version of Browserify, as previous versions used derequire, which can cause your build to be much slower. If you're stuck with a pre-5.x version of Browserify, make sure you aren't generating standalone builds as part of your watch task

Filter Unchanged Files

Another task that can take a long time is hinting. While our sample repository only spends about 500ms in the jshint task, larger projects can take several seconds.

The Gulp project recommends tools for incremental builds, and one of the best is gulp-cached. To use it, you inject gulp-cached into your stream, and it will keep a cached copy of the file contents in memory. The next time you run the task, Gulp will first compare the files against the cache, and it will only pass changed files through to the rest of the operations.

At Pellucid, this technique cut several seconds off of our JSHint operation. Even in the sample application, incremental hinting was cut from 500ms down to below 100ms.

gulp.task('hint', function () {
    var cached  = require('gulp-cached'),
        jshint  = require('gulp-jshint'),
        stylish = require('jshint-stylish');

    return gulp.src('./js/**/*.js')
      .pipe(cached('hinting'))
        .pipe(jshint())
        .pipe(jshint.reporter(stylish));
});

This same technique will also work with other files that only need to be processed when they've changed. For example, if you're minifying many images, only operating on changed images is a big time saver.

If you're worried about the amount of memory this will take, especially with lots of large images, you can pass an optimizeMemory flag that will store an md5 hash instead of the full file contents. Alternatively, there are options such as gulp-changed and gulp-newer that compare timestamps instead of saving contents in memory.

If you need to bring files back into the stream — for example, if you hint only changed files, but you want to concatenate all of them — you can do so with gulp-remember.

Separate Production Tasks

We've already stopped Uglifying our development JS build, but there are a few other steps we can take to handle production builds differently than development builds. This brings some minor speed improvements by only running the operations we care about for the current build target.

To start, we'll allow passing a --prod flag to our Gulp build. We'll read this with gulp-util and store it in a variable for easy access.

var gulp  = require('gulp'),
    gutil = require('gulp-util'),
    prod  = gutil.env.prod;

Now, inside our various tasks, we can check this prod flag and conditionally run certain operations or fall back to gutil.noop() which simply passes on the stream.

For example, in our Browserifying task, we'll create sourcemaps for dev, and we'll Uglify for production:

gulp.task('js', function () {
  var browserify = require('browserify'),
      source     = require('vinyl-source-stream'),
      streamify  = require('gulp-streamify');

  return browserify('./js/main.js', {
      debug: !prod
    })
    .bundle()
    .pipe(source()) // convert to a stream gulp understands
    .pipe(prod ? stream(uglify()) : gutil.noop())
    .pipe(gulp.dest('./build'));
});

Similarly in our Stylus task, line numbers (or if you so desire, the newly enabled sourcemaps) are for dev, and minifcation is for production:

gulp.task('stylus', ['cleancss'], function () {
    var stylus = require('gulp-stylus'),
        prefix = require('gulp-autoprefixer'),
        minify = require('gulp-minify-css');

    gulp.src('./styl/main.styl')
        .pipe(stylus({linenos: !prod}))
        .pipe(prefix())
        .pipe(prod ? minify() : gutil.noop())
        .pipe(gulp.dest('./dist/css'));
});

If you're more comfortable with the idea of running a gulp prod task instead of passing a --prod argument, that can work too, with the help of run-sequence.

gulp.task('setProduction', function () {
  // global prod variable that we added above
  prod = true;
});

gulp.task('prod', function () {
  var sequence = require('run-sequence');

  // first set our prod flag, then run other tasks
  sequence('setProduction', ['stylus', 'js']);
});

Putting it Together

The changes we made to our Gulpfile tooke quite a bit of time off of our builds — particularly the incremental builds, which matter the most. Watchify was the biggest win, taking our bundling from around 3 seconds down to 200ms. Caching file contents also saved several hundred milliseconds in our hint task.

We also separated our development and production concerns, allowing us to perform only the operations we care about at the appoopriate times. This allows us to add source maps and debugging hints for dev builds, and minify everything for production builds.

As a final comparison, here was our original Gulpfile, building (and incrementally building) in around 4 seconds. And here is our improved Gulpfile, building slightly faster, and incrementally building in less than 1/10th of the time.

But wait! What about...

Broccoli?

Broccoli is a newer build tool that specializes in incremental builds. If you read through the blog post for Broccoli's initial release, it's clear that Broccoli was created with the intention of solving many of the same problems that we've been addressing above.

With Broccoli, once you fire up broccoli serve, it will figure out by itself which files to watch, and only rebuild those that need rebuilding. ... I’m aiming for under 200 ms per rebuild with a typical build stack...

Sounds pretty great, with one exception. The community-maintained Browserify plugin for Broccoli doesn't seem to use watchify. Broccoli's ability to "only rebuild [files] that need rebuilding" doesn't apply in the context of a full Browserify build. Since Browserify was one of our biggest initial pain points, missing out on Watchify makes Broccoli a non-starter, for now.

CSS preprocessing?

Admittedly, we glossed over this a bit. Gulp's recommended incremental build tools, such as gulp-cached, won't do much to help us in cases where we need to handle lots of input files, combining them into a single target file. For incremental CSS builds, we would need a dedicated tool, similar to what Watchify does for Browserify, and I'm not aware of any such tools for Stylus.

With Stylus at Pellucid, our CSS compilation combines more than 300 Stylus files, containing more than 2300 selectors, in 2-3 seconds. Changes to the styles aren't nearly as frequent as JavaScript changes, so this is tolerable.

I have heard horror stories of Ruby Sass compilation taking 20-30 seconds or more, in which case, you may want to consider changing to the dramatically faster C port of Sass. The Treehouse Blog also has a great article on how they got their 50-second Sass build down to 3.

All the other interesting things we can automate?

It's true, there's no shortage of ways an automated build can speed up your workflow. From popular options such as Live Reload and running automated tests to the more obscure, like bumping package versions and deploying to Heroku, there are plenty of ways to build out your Gulpfile.

Hopefully we've tackled some of the most common pain points in this article: the mundane tasks that read lots of files and tend to be run and re-run each time files are saved. If you've found additional tasks are slowing you down — and especially if you've found clever ways to speed these tasks up — be sure to let us know in the comments!