Improving Drupal Performance After the Initial Page Request

When investigating performance problems of a web site, its helpful to split the problem up into two areas:

  • initial page request
  • loading assets after initial page request

This article is all about what to do if a majority of your overall page load time is from loading assets after the initial page request.  For information on how to improve performance of that initial page request, see the other article I wrote about Improving the Performance of Drupal's Initial Page Request.

To further illustrate what part we are optimizing by using Firebug, we are optimizing what happens AFTER the first line in the Net panel, as shown in this image:

Performance problems after the initial page request can be broken down into a couple categories:

  • too many files
  • files are too big

Too many files can make it hard for your web server to serve all the requests and your web browser also restricts how many you can request at once, which can leave the user waiting for the server to respond.  Having too many large files can cause disk I/O problems, and highlight bandwidth issues where the files just can't be downloaded in a quick enough time to allow your website to load fast enough.

Too Many Files

The files that get fetched after the initial page load include javascript, CSS, images, and other external resources that are not on your domain, such as if you use Google Analytics, or a Facebook Like button, etc.  So some of these are on your domain ( above), and some are on an external domain.

Having too many files poses a problem because a web browser can only send a limited number of requests in parallel at a time.  For desktop web browsers, this is typically 4-6, but for a mobile browser, this is typically smaller, like 2.  This limitation is per domain.  So if you have 24 assets to load, and all are on, you can only load 4 in parallel at a time until they are all loaded.  While waiting for requests to free up, your browser just sits there waiting.

In addition to the web browser only being able to send 4-6 requests at a time, your server can only handle a finite number of requests at a time.  In Apache, I'm talking about the MaxClients configuration variable.  This is the limit of how many total requests you can handle at a single time.  Let's say the web browser limits to 4 requests at a time, and the Apache web server is configured to have MaxClients of 16.  That means you can only have 4 people viewing your web site at a time before things start to slow down for them as they would need to wait for other requests to be completed.

Why not just increase MaxClients?  Well, your server also has a finite amount of RAM.  Each client request takes a certain amount of RAM, with Drupal this can be in the order of 32MB but greatly depend on what modules you have enabled.  So if you have MaxClients set to 16, and each request takes 32MB, then you need 512MB of RAM dedicated to run Apache.  Its important to realize this limitation, and realize that going over your RAM capacity and being forced to use Virtual Memory will absolutely crater your performance.  You must try and guarantee that you NEVER run out of RAM as the performance impact by using Virtual Memory (on the hard disk) is among the worst that you will ever experience.

Luckily, there are some easy ways to reduce the number of files

See in the Firebug screenshot above, there are multiple javascript .js files, one of the basic ways is to concatenate all those .js files together into one big .js file.  This doesn't save any size, but does reduce it into one request.  We can do the same thing with all the CSS files.  Why are there so many JS and CSS files in the first place?  Well, Drupal is a flexible system, and many modules provide their own CSS files, so by default you may have 20 or more CSS files that are provided by other modules.  With flexibility you get with Drupal and CMS's in general, there are always other tradeoffs, having all these files is one of them.  Luckily, Drupal has a built in way to combine all these JS files together, and combine the CSS files together.

Under Configuration | Development | Performance, you see the Bandwidth Optimization section:

The two checkboxes for "Aggregate and compress CSS files" and "Aggregate JavaScript files" are what we are looking for.

So before checking these, on my site, I had 30 JS files and 30 CSS files.  Afterwards, here is what I see in Firebug:

The ones that are on my domain and done by Drupal are the css_ and the js_ files.  It does not pull in external CSS and JS files, so it leaves those widgets.js, etc., alone because those are from other external sites.  So the CSS files are reduced from 30 down to 11, and the javascript files are reduced from 30 down to 8. (there is a lot more to the Net panel that I haven't shown here to be concise).

Because all of this is done with a simple click of a button, it's a must when looking at performance issues.

One other obvious way to reduce files is to disable modules that you don't need to be using.

Another less obvious way that requires a bit more work is when you have a lot of small images, like icons or hover/state backgrounds.  If you have separate images for each icon or hover/state, you can combine these images into a single tiled image that you use CSS to shift around.  The proper terminology for this Image Sprites, read more about CSS Image Sprites on W3Schools.

Another alternative, move static files to another server

One common strategy is to use a CDN (content delivery network) to host some of your static files, with images and videos being one of the easiest ways to implement this, especially if they are a part of your theme.  Because the 4-6 request parallelism max is per domain, if you move some of your assets to another domain, you can request another 4-6 in parallel from that domain, so you can run something like 8-12 in parallel then.  Always a good idea if you can afford it, as CDN's usually do cost a little bit of money, the most common as of today is Amazon S3.

To enable this a little easier, there is a AmazonS3 module for Drupal to move the file system to S3 on Drupal 7, and other modules like the Video module integrate with the AmazonS3 module to allow videos to be auto-stored for you.

Second problem: Files are Too Big

One of the quickest ways to reduce file size that we have already done is by enabling CSS aggregation, you may have also noticed in the screenshot that it also mentioned CSS compression.  That involves stripping out all unnecessary whitespace and newlines to have as compact of a file as possible.  If you develop your own modules and have javascript, I would highly suggest using a Javascript compressor to compress/minify your code.  A couple options are the YUI Compressor or the Google Closure compiler.

Let's look into Image size using the Firebug Net Panel:

Notice that I have clicked on the "Image" button at the top so it only shows image, and I scrolled all the way down to the bottom to see the total size.  You can also click on the Size header to sort by size, you can't see it on the screenshot because its up at the top.

Moving big image files, like backgrounds or pictures of people, off to a CDN is a good idea, so they can start downloading right away, and typically with super-shared web hosting companies, the bandwidth will be unpredictable, so adding a little predictability by moving the biggest images to a CDN will help smooth out the bumps.

If image files are too big, there are a couple ways to help that.  You can try and compress them a little bit more.  This works well with JPEGs, but not well with PNGs in the traditional way of changing compression which is nice with JPEGs because they just have a 1 - 10 compression factor you can play around with and see what is acceptable.  This works great if you have the originals, but be careful compressing an already compressed image, it can look really bad.

To reduce the size of a PNG image, you can try PNG optimization tools like pngcrush or TinyPNG, however for PNGs created with Photoshop, I've found these tools to not do anything helpful.  I've gotten the best results by the techniques in this article on Smashing Magazine about Clever PNG Optimization Techniques.  This stuff is rather complex, but Posterization works amazingly well and is pretty easy to do, basically you reduce the amount of colors so that the compression can work better, compressing a smaller number of colors is easier than a lot of different colors.  That does reduce image quality, so compare before/after and compare that to the file size you save and the impact that load time will have on the success of your site.

Bigger Changes that can help too

In this article, I mainly stick with small changes that can be implemented in minutes.  But there are some bigger solutions that can help.

If you are having problems will all the files causing too much disk I/O, or are running out of RAM to increase your Apache MaxClients, consider moving your database to a different server if it is not already.  A MySQL database can add a lot of I/O load, and depending on how its configured, can take up quite a bit of RAM.  Moving it to a separate server allows you to optimize each server for its specific role.

Encourage caching of static files in client web browsers by using Apache mod_expires in your .htaccess file, here is a great article on HowToForge.  Do this just for image files, Drupal handles the JS and CSS files.

You can use a system like Varnish, Squid, HAProxy to cache static files on the Apache side so they can be requested faster, think like serving the files from RAM instead of from disk.