Drupal Performance Tuning Presentation Slides From Boise Code Camp 2013

As requested and promised, here are the slides from my Drupal Performance Tuning presentation at Boise Code Camp 2013.

Definitely appreciated all of the questions and great discussions and ideas from everyone in the presentation. Hope that everybody learned something to take back and solve any Drupal performance problems that you have.

Drupal Performance Tuning Presentation Slides

10 Ways to Improve Drupal 7 Performance on Shared Web Hosting

When you look to improve performance of your Drupal 7 site when it is on a shared web hosting plan, you are limited in options due to the nature of shared hosting.  Many of the performance optimization options you would have if you were on a Virtual Private Server (VPS) or on your own dedicated hosting box are not available.

Some of the reasons why you are limited in a shared web hosting plan:

  • No configuring the web server software
  • No configuring the relational DB
  • No extra services like APC (Alternative PHP Cache) or Memcached that allow you fetch objects from fast memory instead of slow hard disks
  • No consistency with service, you are sharing with lots of people so you typically don't have dedicated hardware resources for running your site
  • No easy path to upgrade hardware resources within shared hosting, typically need to move to a different service like a VPS or dedicated hosting as next step

Yes, these are pretty limiting, and if you can afford it, I definitely recommend going with at least a VPS if your web presence is a large part of your business.  However, if you just have a blog or a brochure-type site, shared hosting can be just fine.  I've tested out several shared web hosting setups and my current recommendation is Site5's Shared Cloud Hosting plan for $20/month, I have several clients running Drupal sites on that.  For myself, I currently am running my Drupal sites using VPS servers from mainly Linode, but also Amazon EC2.

With all these limitations, you still have a lot you can do to performance tune your Drupal site on shared web hosting plans.  The options below are listed in order of the amount of performance impact for the implementation time they typically have on a shared hosting Drupal site.

1) Turn Page Caching On

What page caching does is that instead of using a bunch of database queries to get the data used in making a typical web page, the rendered contents of the web page are stored in a separate database cache table so that it can be recalled quicker.  If you have 10 people visiting the site from different computers, Drupal first looks into the database cache table to see if the page is there, if it is, it just gives them the page.  Think of saving the output of 50 separate queries so that is accessible with a single query.  You obviously are reducing the SQL queries required by a lot.  What the page cache table actually stores is HTML content.

One gotcha with this Page Caching is that it only works to optimize the page load time for Anonymous users.  This is because when you are logged in, you might have blocks that show up on the page that are customized for you, if it served everybody the same page, they would see your customized information (think of a My Recent Posts block), so Drupal does not use the Page Cache for Authenticated users automatically.  This allows you to turn Page Caching on and still get the benefit for Anonymous user page load times, but does not break the site for Authenticated users.  There are other caching options that will help with Authenticated user page performance, we will talk about those later.

To enable Page Caching, you go to Configuration | Development and select the checkbox next to "Cache pages for anonymous users".  See screenshot:

An alternative to the page caching built into Drupal 7 is to use Boost to cache your pages as static HTML files instead of in the database, which can be served very efficiently especially in shared hosting environments.  If you use Boost, you would not want the Drupal page caching enabled like mentioned above.  Boost would be a preferred option except for the fact that currently, Boost is still in beta for Drupal 7 even though Drupal 7 has been out for quite a while, and I don't often recommend using beta modules unless you have tested them thoroughly.  Also, like the Drupal page caching, this still only works for Anonymous users.

2) Aggregate and Compress CSS Files

Drupal is very flexible, but with all that flexibility, you will have some performance tradeoffs.  One of the ways these tradeoffs are easy to see is in the number of CSS files you see loaded in each Drupal page.  There can be 30 or more CSS files even for pretty basic Drupal sites, the reason for this is that Drupal operates using modules, and whenever a module has some CSS files that it wants to add, it is a separate file.  Your theme is also going to provide several CSS files.

This can cause performance problems because you have to make 30 separate requests to the web server, one for each file.  We would like to reduce the number of requests to the web server as a general rule of web performance optimization.

So by enabling "Aggregate and compress CSS files", Drupal will take your CSS files and reduce the number of them and size of them.  The Aggregate part of that is that instead of having 30 separate files, Drupal will concatenate all the files together and separate them into just a few files.  It doesn't reduce them to 1, but reduces it down to 8 files or so.  The Compression part of that setting will remove all whitespace in the CSS files, those whitespaces each contribute to the size but have no functional purpose.  So this setting reduces the number and total size, which will increase the speed of loading CSS.

3) Aggregate JS

Just like Aggregate and Compress CSS files, you can aggregate Javascript files with this setting.  For the same reason why there are many CSS files, there can be many JS files, each module can provide one or many of its own Javascript files.  Aggregation reduces the number of files down to around 8.  So by Aggregating both CSS and Javascript files, you can reduce the requests from 60+ to less than 20, this can help to deal with the shared resources so you can get as much from the server with each request as possible.

For items 2), and 3), the Settings can be enabled in Drupal 7 on the Configuration | Development | Performance age, you see the Bandwidth Optimization section by checking all three checkboxes:

4) Turn Views caching on

As mentioned when talking about Page Caching only working for anonymous users above, there are other caching options for helping with Authenticated user page performance.  One of those options is to turn on caching for blocks and pages that you create using the Views module.  This allows you to cache the output of the query used to generate the view, or the end HTML output of your View, and you can tune the cache for them separately.  And realize too that this means you can cache portions of a page if you are using one or several Views blocks in the page, it will just cache that block in the page, not the whole page.

You can find the settings for caching a View when you are in the Views UI and edit your view, then under Advanced | Other | Caching after you click Time-based to enable caching, you can see the settings below where you can specify the maximum age of a cached query or page:

The way that specifying the maximum age works is that when the cached copy is stored in the database, every time that a user visits the page with the View in it, it will check to see if the cached copy is too old and will fetch a fresh copy if it is expired.  This is the nature of caching, you have to determine for the typical use of your site what kind of cache usage you can have.  If changes are made to your site infrequently, then the cache doesn't have to be cleared often.  If changes are made often, it may be best to keep the maximum age times pretty low so that people see a View that represents the actual contents of the site that will be out of date by no more than 1 hour (using the settings above as an example).

The difference between caching the Query results and the Rendered output applies more when you are doing some Theme altering, you may not want to cache the rendered output, but could instead just cache the query results.  The Rendered output is achieved using the Query results, so you can likely save a bulk of the time using Query results alone.  Bottom line, if you can use Rendered output caching, it will give you the best caching results.

5) Turn block caching on

This is another type of caching to help out with Authenticated user performance.  This will cache individual blocks, not pages, but since many blocks can be used to build up a page, doing block caching can still be a significant performance improvement.  This block caching is not the same as caching a Views block, in fact, it does not apply to Views blocks.  This block caching only applies to custom blocks that you create, or blocks that modules provide.  Modules themselves provide what type of caching is allowed for blocks to help do it appropriately, as some blocks should be cached per user, some cached per role, all meaning that for a single role, a block might be the same for all users in that role, etc.

To enable Block Caching, you go to Configuration | Development and select the checkbox next to "Cache blocks". See screenshot:

To read in a little more detail about caching, see my article on Improving the Performance of Drupal's Initial Page Request.

6) Disable Modules You Are Not Using

Having unused modules that are enabled adds to overhead for additional PHP code to execute on each page load, extra CSS and Javascript files included with each page load, even if you are not using the module for anything.

Going one step further, if you have a module that you are only using one small part of, and can achieve the same results using another method utilizing an existing module or another method in core Drupal, then that's always a good idea for optimizing performance.

7) Move Images, Videos, and Static Files to a CDN (Content Delivery Network)


Offloading static files to another server is a way to conserve resources on your shared hosting.  This helps with avoiding disk I/O and reduces the number of requests that your shared hosting has to handle.  One common strategy is to use a CDN (content delivery network) to host some of your static files, with images and videos being one of the easiest ways to implement this, especially if they are a part of your theme.

Since web browsers restrict the number of requests a single domain name can handle in parallel at any given moment to 4-6, if you move some of your assets to another domain, you can request another 4-6 in parallel from that domain, so it can increase speed in yet another way.  Always a good idea if you can afford it, as CDN's usually do cost a little bit of money, the most common as of today is Amazon S3.  The cost can be really small though, like less than $5/month, check out the Amazon S3 price calculators to make sure you understand what costs you might have.

To enable this a little easier, there is a AmazonS3 module for Drupal to move the file system to S3 on Drupal 7, and other modules like the Video module integrate with the AmazonS3 module to allow videos to be auto-stored for you.

8) Implement Caching in Your Own Modules

If you are running a Drupal site where you have created your own modules, make sure to implement caching so you can take advantage of the caching system Drupal has in place.  If your module will be caching a lot of different types of data, consider creating your own caching table, then you can control the maintenance of that cache table easier.  Also helps if at some point in the future you move onto a non-shared hosting system where you can take your own caching tables and move them to use memcached, but leave the other cache types to use the database cache table.

A good overview of how to include caching in your own modules is here: Beginners Guide to Caching Data in Drupal 7

9) Make Sure Your Own Modules Use Indexes Properly

If you have database queries in modules that you have developed yourself, verify that you have set up indexes properly to help those queries execute efficiently.  You can use the EXPLAIN command in MySQL to see if your query is using an index, and how it is using that index.  It's very helpful to use the Drupal Devel module to figure out which database queries are taking a long time.  Indexes are one thing that you can control in a MySQL database even when you are on a shared hosting platform.  I show the basics of using Devel in my article on Improving the Performance of Drupal's Initial Page Request.

10) Reduce Image Sizes and Number of Images

Reducing image sizes and/or number of images works to improve performance in a similar way as the CSS and JS aggregation did, you are reducing the number of requests to the server, and reducing the overall size that needs to be transferred from the server to your user's computer.

You can always work on reducing the size of images that you have by using different levels of compression.  You can use software tools to try and compress pngs, or just use settings when you save an image within Photoshop for JPEG images, or you can try some advanced techniques that I've used before at Smashing Magazine about Clever PNG Optimization Techniques.

Reducing the number of images can be accomplished by using CSS image sprites to combine multiple images into a single image.  This works especially well if you are using images as backgrounds for hover/active states for tabs or buttons.

Google Chart Tools Presentation

Here is a link and embed of a presentation I gave to the Boise Google Developers Group a couple weeks ago.  I presented about Google Chart Tools, which is a web data visualization package provided by Google to use.  It is a Javascript API to create interactive, nice looking charts to dress up a data-driven website.  I have used Google Chart Tools in several projects for clients and myself, and find them to be easy to use and stable.

Knowing that any solution to a problem first needs a problem, I begin the presentation with a list of the requirements that a particular project for a client had regarding Data, Tools, and Development environment.  I then proceed through the many web-based data visualization solutions out there and show why Google Chart Tools fit my requirements best.  Then, I have a quick, general overview of the Google Chart Tools features then leave the rest of the details for you to explore on the site which includes a Code Playground and Chart Gallery.

Link: http://prezi.com/hacdlbxeoroe/choosing-a-web-data-visualization-solution...

Improving Drupal Performance After the Initial Page Request

When investigating performance problems of a web site, its helpful to split the problem up into two areas:

  • initial page request
  • loading assets after initial page request

This article is all about what to do if a majority of your overall page load time is from loading assets after the initial page request.  For information on how to improve performance of that initial page request, see the other article I wrote about Improving the Performance of Drupal's Initial Page Request.

To further illustrate what part we are optimizing by using Firebug, we are optimizing what happens AFTER the first line in the Net panel, as shown in this image:

Performance problems after the initial page request can be broken down into a couple categories:

  • too many files
  • files are too big

Too many files can make it hard for your web server to serve all the requests and your web browser also restricts how many you can request at once, which can leave the user waiting for the server to respond.  Having too many large files can cause disk I/O problems, and highlight bandwidth issues where the files just can't be downloaded in a quick enough time to allow your website to load fast enough.

Too Many Files

The files that get fetched after the initial page load include javascript, CSS, images, and other external resources that are not on your domain, such as if you use Google Analytics, or a Facebook Like button, etc.  So some of these are on your domain (jeremyzerr.com above), and some are on an external domain.

Having too many files poses a problem because a web browser can only send a limited number of requests in parallel at a time.  For desktop web browsers, this is typically 4-6, but for a mobile browser, this is typically smaller, like 2.  This limitation is per domain.  So if you have 24 assets to load, and all are on example.com, you can only load 4 in parallel at a time until they are all loaded.  While waiting for requests to free up, your browser just sits there waiting.

In addition to the web browser only being able to send 4-6 requests at a time, your server can only handle a finite number of requests at a time.  In Apache, I'm talking about the MaxClients configuration variable.  This is the limit of how many total requests you can handle at a single time.  Let's say the web browser limits to 4 requests at a time, and the Apache web server is configured to have MaxClients of 16.  That means you can only have 4 people viewing your web site at a time before things start to slow down for them as they would need to wait for other requests to be completed.

Why not just increase MaxClients?  Well, your server also has a finite amount of RAM.  Each client request takes a certain amount of RAM, with Drupal this can be in the order of 32MB but greatly depend on what modules you have enabled.  So if you have MaxClients set to 16, and each request takes 32MB, then you need 512MB of RAM dedicated to run Apache.  Its important to realize this limitation, and realize that going over your RAM capacity and being forced to use Virtual Memory will absolutely crater your performance.  You must try and guarantee that you NEVER run out of RAM as the performance impact by using Virtual Memory (on the hard disk) is among the worst that you will ever experience.

Luckily, there are some easy ways to reduce the number of files

See in the Firebug screenshot above, there are multiple javascript .js files, one of the basic ways is to concatenate all those .js files together into one big .js file.  This doesn't save any size, but does reduce it into one request.  We can do the same thing with all the CSS files.  Why are there so many JS and CSS files in the first place?  Well, Drupal is a flexible system, and many modules provide their own CSS files, so by default you may have 20 or more CSS files that are provided by other modules.  With flexibility you get with Drupal and CMS's in general, there are always other tradeoffs, having all these files is one of them.  Luckily, Drupal has a built in way to combine all these JS files together, and combine the CSS files together.

Under Configuration | Development | Performance, you see the Bandwidth Optimization section:

The two checkboxes for "Aggregate and compress CSS files" and "Aggregate JavaScript files" are what we are looking for.

So before checking these, on my jeremyzerr.com site, I had 30 JS files and 30 CSS files.  Afterwards, here is what I see in Firebug:

The ones that are on my domain and done by Drupal are the css_ and the js_ files.  It does not pull in external CSS and JS files, so it leaves those widgets.js, etc., alone because those are from other external sites.  So the CSS files are reduced from 30 down to 11, and the javascript files are reduced from 30 down to 8. (there is a lot more to the Net panel that I haven't shown here to be concise).

Because all of this is done with a simple click of a button, it's a must when looking at performance issues.

One other obvious way to reduce files is to disable modules that you don't need to be using.

Another less obvious way that requires a bit more work is when you have a lot of small images, like icons or hover/state backgrounds.  If you have separate images for each icon or hover/state, you can combine these images into a single tiled image that you use CSS to shift around.  The proper terminology for this Image Sprites, read more about CSS Image Sprites on W3Schools.

Another alternative, move static files to another server

One common strategy is to use a CDN (content delivery network) to host some of your static files, with images and videos being one of the easiest ways to implement this, especially if they are a part of your theme.  Because the 4-6 request parallelism max is per domain, if you move some of your assets to another domain, you can request another 4-6 in parallel from that domain, so you can run something like 8-12 in parallel then.  Always a good idea if you can afford it, as CDN's usually do cost a little bit of money, the most common as of today is Amazon S3.

To enable this a little easier, there is a AmazonS3 module for Drupal to move the file system to S3 on Drupal 7, and other modules like the Video module integrate with the AmazonS3 module to allow videos to be auto-stored for you.

Second problem: Files are Too Big

One of the quickest ways to reduce file size that we have already done is by enabling CSS aggregation, you may have also noticed in the screenshot that it also mentioned CSS compression.  That involves stripping out all unnecessary whitespace and newlines to have as compact of a file as possible.  If you develop your own modules and have javascript, I would highly suggest using a Javascript compressor to compress/minify your code.  A couple options are the YUI Compressor or the Google Closure compiler.

Let's look into Image size using the Firebug Net Panel:

Notice that I have clicked on the "Image" button at the top so it only shows image, and I scrolled all the way down to the bottom to see the total size.  You can also click on the Size header to sort by size, you can't see it on the screenshot because its up at the top.

Moving big image files, like backgrounds or pictures of people, off to a CDN is a good idea, so they can start downloading right away, and typically with super-shared web hosting companies, the bandwidth will be unpredictable, so adding a little predictability by moving the biggest images to a CDN will help smooth out the bumps.

If image files are too big, there are a couple ways to help that.  You can try and compress them a little bit more.  This works well with JPEGs, but not well with PNGs in the traditional way of changing compression which is nice with JPEGs because they just have a 1 - 10 compression factor you can play around with and see what is acceptable.  This works great if you have the originals, but be careful compressing an already compressed image, it can look really bad.

To reduce the size of a PNG image, you can try PNG optimization tools like pngcrush or TinyPNG, however for PNGs created with Photoshop, I've found these tools to not do anything helpful.  I've gotten the best results by the techniques in this article on Smashing Magazine about Clever PNG Optimization Techniques.  This stuff is rather complex, but Posterization works amazingly well and is pretty easy to do, basically you reduce the amount of colors so that the compression can work better, compressing a smaller number of colors is easier than a lot of different colors.  That does reduce image quality, so compare before/after and compare that to the file size you save and the impact that load time will have on the success of your site.

Bigger Changes that can help too

In this article, I mainly stick with small changes that can be implemented in minutes.  But there are some bigger solutions that can help.

If you are having problems will all the files causing too much disk I/O, or are running out of RAM to increase your Apache MaxClients, consider moving your database to a different server if it is not already.  A MySQL database can add a lot of I/O load, and depending on how its configured, can take up quite a bit of RAM.  Moving it to a separate server allows you to optimize each server for its specific role.

Encourage caching of static files in client web browsers by using Apache mod_expires in your .htaccess file, here is a great article on HowToForge.  Do this just for image files, Drupal handles the JS and CSS files.

You can use a system like Varnish, Squid, HAProxy to cache static files on the Apache side so they can be requested faster, think like serving the files from RAM instead of from disk.

Improving the Performance of Drupal's Initial Page Request

When troubleshooting performance problems with any web site, I first look at the problem to try and break it down into pieces.  Where is the biggest problem?  One tool I use to do that is Firebug.  If you don't know much about Firebug, read my article Using Firebug to Figure out Why Your Website is Running Slow.

The first attempt to break down performance problems in to pieces results in finding out if the problem is with the initial page request, or the subsequent assets that are loaded.  See this Firebug screenshot of the Net panel:

When I talk about the "initial page request", I'm talking about the very first line, the www.jeremyzerr.com one.  That first request is requesting the entire HTML source of the page.  That entire source needs to be downloaded, then the web browser knows to make a bunch more requests for assets.  This would be all the lines from the 2nd one on, includes CSS, Javascript, Images.

This article will talk about improving the performance of this intial page request, I will cover improving the subsequent asset loading in an upcoming blog post.

I will list out what to look at in the order of likelihood of being the cause of your problem in a typical Drupal site.  I typically follow a strategy like this to save time and getting right to the problem.

its important to note the time above from the Firebug output.  It is 516ms.  When looking at these options, you want to be able to figure out how to break down that time into different components, and also remember what it was initially so you can change one thing, test, repeat.

Too Many Database Queries Needed to Generate the Page

To see how many database queries are needed, and how long they take, use the Devel module.  Go to its configuration at Configuration | Development | Devel Settings.  Select the checkbox for "Display Query log", and also select to Sort By duration, it is more useful than sorting by source.  Save those settings, then visit the page that is slow.  At the bottom of the page, you will see some extra output that looks like this:

There are a few important items here.  You see the overal number of queries, the total time it took them to execute, and for each query you see the time and number of executions.

We want this information not only to look at it and see if any queries are taking too long to run, but we also want to use it as a base for our benchmarking when we start making changes.

There are a few ways we can reduce the number and overall time of access to the database.

Page and Block Caching

What page caching does is that instead of using all those queries to each get information used in making the web page, the contents of the web page are stored in a separate database cache table so that it can be recalled quicker.  If you have 10 people visiting the site from different computers, Drupal first looks into the database cache table to see if the page is there, if it is, it just gives them the page.  Think of saving the output of 50 separate queries so that is accessible with a single query.  You obviously are reducing the SQL queries required by a lot.  What the page cache table actually stores is HTML content.

One gotcha with this Page Caching is that it only works to optimize the page load time for Anonymous users.  This is because when you are logged in, you might have blocks that show up on the page that are customized for you, if it served everybody the same page, they would see your customized information (think of My Recent Posts), so Drupal does not use the Page Cache for Authenticated users automatically.  This allows you to turn Page Caching on and still get the benefit for Anonymous user page load times, but does not break the site for Authenticated users.

There is another caching setting for "Cache blocks" that helps to optimize the site for Authenticated users, it fills in the gaps by caching those blocks that might be different per user.  Takes up more space in the cache, but it is better than nothing.

To enable Page Caching, you go to Configuration | Development and select the checkbox next to "Cache pages for anonymous users".  Also select the "Cache blocks" checkbox, this is what works for Authenticated users, blocks are cached, which means they can be assembled into an entire page easier.  See screenshot:

To show you how big of an impact this can have, take a look at a Firebug output before turning on caching and viewing a page as an Anonymous user:

And after turning on caching:

You can see it shows the page loading in about 1/3 of the time.  Remember when benchmarking that the first request to a page will fill the cache, the 2nd request will use the cache.

To take this one step further, you can change out your Drupal caching engine to not store to a database table, but to instead store it in memory using Memcached.  So if you have plenty of RAM on your server, you can eliminate a LOT of database queries and operate even faster.

Remember what caching is though, a whole page is stored in the database and always served up.  What if you actually change that page, like you make a new blog post and want your home page to show it.  With page caching on, it will not show up until the cache is cleared.  By default, with Drupal, the cache is cleared every time cron is run.  So as always, its a tradeoff.  You can clear the cache manually each time you post something new, or if its OK that a new post doesn't show up for a few hours (based on your Drupal cron period setting), then you can take advantage of some serious performance improvements.

Some of the extra settings on the Performance page also deal with this, there are settings for Minimum Cache Lifetime, and Expiration of Cached pages.  See this screenshot:

So you can even further fine tune how long you would like your cached entries to live.  Just be realistic about the need to show the latest content.  If you are posting a few times a day, can do with hours, if you only post a few times a week, a day is just fine.  But also if you want to post and then push links out to social sites about your content, you might want to clear cache manually.

As a final note, along with page and block caching, if you have a lot of Drupal Views that have blocks or pages showing slow performance, you have to make a special trip into the Views settings and change the cache settings there.  You can fine tune the cache settings per View, and even separately for Blocks and Pages within a single View.  Hopefully if you have a Views block or page causing you performance issues, you see that in the Devel query log and can pinpoint it back to which View it is coming from.

More Caching, MySQL Query Caching

A quick, easy thing to do to reduce the burden of lots of really small, simple database queries is to use the MySQL Query Cache.  This sets aside a portion of memory to save the output of a database query so that it can be looked up if the output of the query has not changed since the last time you issued the query.  Basically, if a particular database table has not had any changes to it, then any queries on that table can go in the query cache and stay there.  As soon as a change happens to a table, all queries on that table that are in the query cache are invalidated.

Long story short, just give a small amount of memory to it, like 8M or 16M.  By default, it is not enabled at all in the MySQL distributions that I use most, which is why its worthwhile to at least turn it on.  Because of its low-level interaction with the database, it does not require any application changes.

Is your Drupal Page Using Too Much Memory?

Another slowdown possibility once you have taken care of all the database stuff is that you just have too much traffic, and your Apache web server is configured to accept a lot of connections, but your web server does not have enough memory to serve all those connections.  Or it isn't configured to handle the connections you are getting, and they have to wait until another connection frees up.

Both situations cause web performance to grind to a halt, either from running out of RAM, which causes Virtual Memory disk I/O in the first case, or in the second case your requests have to pause until other requests are handled.

Another setting in the Drupal devel module helps you figure out how much memory it takes to load a page.  Select the checkbox for "Display memory usage":

Then go to your slow page and look at the bottom for output that looks like this:

Your page will likely have a lot more, I'm showing this memory usage from a demo site that does not have much being loaded on the page.

The setting we look at in Apache to see how many connections we can handle is the MaxClients setting in apache.conf.  If you look in your Apache error log, you can see if you have been hitting this limit, it will say that you have reached your MaxClients limit.  If you look at your utilized swap space using the Linux command "top" and see that its increasing, or is non-zero, inspect the processes and see how much RAM they are using.  The solution to these issues is increase your MaxClients if you are running into the connection problem.  If you are hitting max RAM and still want people to use your web site, get a bigger web server with more RAM.

You can try and make Drupal use less memory by disabling modules, but in almost all cases, this is not an option as it would reduce functionality of your site.  Just pay a little more for web hosting.  Another pretty easy thing to do is to free RAM up on your web server by moving your database off onto another server, so you free up any RAM you had allocated to MySQL.  Its always a good idea to do this anyways, and is typically one of the first things to look at when scaling from the bare bones web server + database on a single server.

Too Many PHP Files Being Interpreted

PHP is an interpreted language, which means the files have to be reduced down into opcodes in order to actually be executed by the server.  One of the things you gain by using Drupal is flexibility, but you sacrifice that in terms of overhead of all those PHP files that need to be called to find all the hooks to run.  Luckily, there is an option out there that will store the intermediate code of PHP files in memory so that every request does not result in fetching all those PHP files off disk and interpreting the files every time.

The component is APC, the Alternative PHP Cache, sometimes called a PHP accelerator.

All you do is edit a section in your php.ini file, allocate some memory for it, and turn it on.  By turning it on, I mean use the option apc.stat = 1 that checks the timestamp of the file first to see if the file has been modified before it grabs it from the cache.  That allows a development process to work smoother so as soon as you release code, it will take the new copy.

Just set:

  • apc.enabled = 1
  • apc.shm_size = 16M
  • apc.stat = 1

I have run into some problems if your release process involves doing a svn export over the top of your code.  I still do it, but I've added in a command that clears the APC cache after I export the code so that I make sure the new files are being run.  That is just a given part of my development scripts that is an acceptable for the benefit you get for it.

While I recommend this for Drupal, I cannot recommend for Wordpress with any certainty.  I've had Wordpress sites that just white screen due to APC problems that weren't having code changed, has remained a mystery to this day, I'm not saying don't do it, just be careful and you can't say I didn't warn you.  And these were running on the same server as Drupal sites using APC that were running fine.

Not Covering All Possibilities, Just the Big Ones

So there are a lot of additional little things to look at, but in my experience, these are the most likely culprits when your initial page load speed is suffering.

23 Questions your Web Project Requirements Should Answer

When you are writing your requirements for your next web project, here are some handy questions to ask yourself to make sure that you are doing an effecitve job.  Being a web developer myself, I've seen a lot of variety of requirements from different clients.  From no requirements, to well thought-out and concise requirements.  Realize, that projects with no requirements always end up with requirements by the time we are done discussing your project, so just do them ahead of time.  Specifically, by a list of requirements, I mean written down, clear requirements about the web site and the environment the web site will operate in.

When a potential client comes to me with well written requirements, I know that this web project isn't some random idea that a person isn't serious about.  Also tells me that you have a clue about web sites and running a project, which in my experience always leads to a better relationship and better outcome.  I am a lot more likely to take your project if we can communicate, and the requirements and our initial interview is often all I have to go by.

Why are writing requirements important?

Writing good requirements are most people's least favorite thing to do, but are important in so many ways.

  • Clearly set expectations
  • Allows your web developer to accurately estimate time (so you stay on budget)
  • Makes any requirements meetings before the project starts more efficient (nobody likes to waste time)
  • Your potential web developer will know you are serious and is more likely to take your project
  • Less communication needed during the completion of the project (no wasted time)

All of the tips I'm offering to you are part of my initial interview process that I go through with potential clients and their projects.  This is BEFORE the project is started.  So if these questions are answered in the requirements before we initially talk, the initial interview goes smooth and we can talk about more exciting stuff, like brainstorming the ways your web project can help you in ways you never thought of.

You'll also find that these tips center around figuring out how your web project fits into your business.  This is something your web developer needs to understand.  If I work on your project, I must understand your business so I can make appropriate decisions as I develop the site.  I also can then recommend small, little things that might make a big difference in how your users interact with the site.

Does your pizza delivery guy need to know about pizza?

You wouldn't send a pizza delivery guy out without understanding pizza.  He wouldn't realize that it needs to be delivered hot as a priority, and he wouldn't understand that people might want sprinkle cheese and pepper with their pizza.  He might even flip the pizza box over and ruin all that cheesy deliciousness if he doesn't know what a pizza is.  So in the same way, please make sure your web developer can understand your business before letting them loose on your project.

So here are the questions to ask yourself when writing requirements for your web project.

1. What is the business purpose of your site, how does it fit into your overall business model?

Without knowing the purpose and how it fits in, what will drive the little details of where things are placed, and how things are done.  Don't have your pizza delivery guy know nothing about the pizza.

2. What goals do you have for visitors of your site?

Do you want people visiting your site buy something?  Share something?  Signup for something?  This is really important for layout of your web pages, and how the site is designed to funnel people towards those goals.  Just reading an article is rarely the goal, you want people to form a connection with you, so make it easy for them to do so.

In addition, knowing the goals allows us to put the proper analytics in place to make sure we are tracking those goals.

3. How will your customers reach your site?

Search, Ads, Social media?  Sure, you say all of them right, but what is the reality?  Do you know where they are coming from now if you have an existing site?

4. What geographic location of people does your site serve?

If you are only targeting local people, then the web site should reflect that and it should be obvious to visitors of your site that you are a local company.  Also helps to figure out what your methods to reach out to people should be.  Its easier for local businesses to focus efforts on a small area, than a bigger area like the whole US.

5. What are some examples of similar sites that you like?

This doesn't mean: What site should I copy?  As unique of an idea as you have, there are almost always people who have done similar things, reinventing the wheel is not good.  Even if its bits and pieces from several different sites, this is a tremendous help for a designer to get what color schemes you like, and what navigation you like.

It's also important to know what you don't like.  If you don't like big image sliders at the top of the home page of sites, please tell somebody.  Save everybody some time.

6. How will the web site be hosted?  What are the capabilities?

If you already have a hosting solution, or know what you are going with, this is very important to know.  Why?  What are the hosts capabilities?  The site that is designed must fit into these parameters.  Also, right along with this is what are your backup requirements for your files and database.  Designing these into a system can take some time, so include it in your requriements.  You have thought about backups, right?

7. How many people will have logins to the site?

Its just an estimate.  Also, how many of those people will be members of your organization, and how many will be customers?  Its important for the web developer to have an idea of the type of internal security that is needed, and what roles will be necessary to protect content from internal vs. external vs. anonymous users.

8. What user authentication methods will you use?

When users register, do you want to just user regular old user/password that is saved in your own site's database?  Or do you want to allow Login with Google, Facebook, OpenID?  Or is this an internal site that you want to use an existing login system like LDAP?

9. How many users will be allowed to contribute content and what kind of technical ability do they have?

Designing a rich text editor for a single person with lots of technical experience is a lot easier than designing one that will be stripped down so that a non-technical person can't make posts that look bad.  A technical user can be trusted with a lot more control over the way the content looks and is formatted.

10. What kind of content will people add?

That helps determine the work required to theme the individual page templates.  Typically, individual content types will have their own template, which just means the layout of the individual fields within each content type might be different enough to have its own special HTML/CSS page layout.

Knowing the different type of content also helps to imply what other kind of views might be needed.  Such as, if you have an Event type of content, then it implies you will have a calendar view, and possibly an upcoming events view.  This is another really important part to have for estimating time properly.

11. What roles will users be able to have?

This mainly has to do with security, but also usability.  This is what user groups the users fall into, those user groups can have permissions extended to them.

The security part is to make sure that only certain types of users can add an Event, but that other users cannot.  Maybe you have certain users that are superusers and can edit or delete any existing Event content.

For usability, if a person cannot edit an Event, then they should not see an "edit" link.  Its important in the usability of a site to only have the valid actions presented to a user.

12. How will users interact with the content?

Do you want users to be able to comment on particular types of content?  Do you want them to share on social media using share buttons?  Which social media outlets?  Just putting them all on there is pointless, too many options leads to indecision, narrow down the most important sources and include those.  Can always change later.

13. Any restrictions on technology or licensing used?

An example of this would be no Flash used for videos.  This is probably more important for internally developed software, but it needs to be discussed.  Its a given that the license will be need to be compatible with your intended delivery method, as there are different restrictions if you are selling the software, or just using it internally.

14. What devices do you want to target?

This is where designing for the small screens on mobile devices comes in.  Its also good to prioritize the screens you design for, whether it is PC, tablet, mobile.  All comes down to effort required and you can save money on your project if you can keep the design flexible (buzzword: responsive) enough to work on all devices with a single theme layout.

15. Do you want a single site for mobile?

When it comes to mobile design, there are a couple broad options: single site that does all sizes of devices, or a mobile-only device.  The trend, and the better way in my opinion, is to go with the single site that is flexible to shift with screen size.

16. Do you have any graphics or color scheme that will be provided?

Already have a logo?  Already have a vast library of stock images that will be used?  It's important to mention this so your graphic designer knows what they have to work with.  If your brand has a specific color scheme,  mention that it needs to be adhered to.

17. What are the metrics for success?

This is related to what goals you have for embarking on this web project, but list out what a successful web project completion would mean to your core metrics.  Increased visits?  Increased shares?  More sales or signups?  This helps communicate what is truly most important to you so expectations are clear.

A big part of this is also knowing what those metrics are currently.  Or if you don't have an existing site, what you estimate them to be.  Use of analytics in a web project is a MUST, you need to be able to track your progress, at least to ensure that your web project doesn't cause you to take a step backwards when completed.

18. What are your frustrations with your current site?

You have built up experience using your current site, or even other similar sites, so communicate those frustrations.  They will be #1 on your developer's list of things to take care of on the new project.

19. What do you want to keep the same as the old site?

This is important too, if you like something, its important to understand why you like it so that if possible, it can be incorporated into the new site.  At the very least, a similar feature can be implemented to give you the same benefits.

20. What methods will you use to drive traffic?

If you drive traffic through advertisements, it may mean that you will want to be able to create landing pages easily and track the metrics.  If you are collecting signups on your landing pages, will possibly want to tightly integrate if you are having to make new signup forms often.

21. How will you send email from the site?

Will you have the capability of sending email from your web hosting company, or will you be using a 3rd party like MailChimp to collect signups and send followup emails.

22. Are you taking payments on your site?

Who is your payment gateway and processor?  What kind of products or services are you selling?  This also helps to understand costs incurred with PCI compliance and other taxation factors.

23. Do you need an SSL certificate?

Are you wanting to protect a shopping cart, or the admin area, or both?

Using Firebug to Figure Out Why Your Website Is Running Slow

When trying to figure out why a web site is running slow, the first place I start is pulling up Firebug, a plugin for Firefox, the web developer's utility belt.

Why do I use Firebug?

Its the quickest way to narrow down your search without having to log into your web server, and we all are interested in saving time.  Its also free and it has matured so much over the years that it really is amazing how it can improve your debug skills.

How Do You Know Your Site is Slow?

So you have a slow website, but before you get started trying to speed things up, its good to have in mind what your goal is for performance is.  Most of the time, having a slow website is just a feeling, you know its slow because of your expectations on how fast a website in general is supposed to load.  We all view a lot of web pages, so our expectations are realistic and based on a ton of first-hand experience.  Trust your gut, but back it up with data.

Set a Page Load Time Goal

The first thing I do is set a goal for the page load time.  This is the time from when I type the address in and hit enter, or from when I click a link, to the time that the page is usable.  This page load time goal is different for different types of web sites.

For a standard blog or information site where you are viewing content or product pages, my goal is to be less than 4 seconds.  This isn't based in anything real scientific, just my experience.  And it is also a very achievable goal for those types of sites.  If you are not sure what feels good to you, go visit some similar sites and see what their page load time is.

Let's Get Some Numbers

To figure out what your current page load time is, open firebug, go to the Net tab, and scroll all the way to the bottom and see the times listed on the bottom right corner.  If you didn't have Firebug enabled for the site, you may have to refresh the web site after opening up Firebug for the first time.

The screenshot shows 2.01s, with an onload time of 2.19s.  The onload time is when your site is done loading everything local to your site (HTML, CSS, Javascript code, images).  The first listed time can be longer than the onload time if you are loading things in from other sites using javascript AJAX reqeusts, videos, widgets from other sites like Facebook or Twitter.  Read more about this here.

Which one of those you use for page load time kinda depends on the site, but I typically use the onload number, unless the items loaded from other sites is a large part of your page and the change is apparent to the user, like loading the thumbnail frame in a video player.

Firebug Setup - Disable Browser Cache

For doing the page load time testing, I use Firebug to disable the browser cache.  I do this because the user of my site I'm trying to please is viewing the site for the first time.  First impressions are everything.  If they come to the site, and its slow, they may not even let the page continue loading and go back to google for the next result.

So open Firebug, go to the Net panel, click on the down pointing triangle, and select "Disable Browser Cache".

Checking the performance of your site with the Browser Cache enabled like normal is important, too. You can have caching issues, but your site with caching enabled should always load faster than your un-cached page, so it will meet our page load time goal.

Breaking Down the Net Panel View

Let's look at the top of the Net Panel.  This is for my Zerrtech site which happens to be a Drupal 6 site, with PHP, Apache, and MySQL.

Notice that the first line is the web page I'm loading, www.zerrtech.com, and the next lines following are CSS files.  The web page HTML has to load before the web browser knows what other assets to load, like CSS, Javascript, and image files.

Knowing this is important, you can see that it takes 107ms for the HTML itself to be generated by the server and be transferred to the web browser.  This time would include Apache handling the request, interpreting your PHP code, accessing the MySQL database.  So if that time is too long, then you have a server side performance issue.  If it is fast enough, then you know that you do not need to tweak anything on the server, this eliminates changing most PHP, Apache, or MySQL settings, which eliminates a lot.

Getting Deeper into the Timeline

Let's take a bigger look at the timeline, I'll hover over one of the timeline items to see more detail.

I want to point out the Grey color in the timeline, which shows Blocking.  This means the web browser is waiting for other requests to complete before issuing a new one.  Your web browser is configured to have a maximum number of concurrent, or simultaneous, connections to a single domain.  I'm using Firefox, and this maximum is 6 at a time.  So only 6 simultaneous requests to zerrtech.com can be active at a time.  Different browsers have different limits, see this link on Stack Overflow for an idea how much this differs.

So notice what the picture shows.  You see the first request for www.zerrtech.com, and the next 5 CSS files have no grey in their timeline, but then the next CSS file I'm hovering over has 83ms of blocking.

Reducing blocking time is the reason why it is advised to reduce the overall number of requests your site makes by combining all your CSS and Javascript files into one single file.  Doesn't affect the operation of the site, but can greatly reduce the number of requests and improve the load time.  This is a setting that is built into Drupal, clicking two options via the online admin area will combine these for you.

What Are You Waiting For?

In the Firebug timeline, the Purple part is for Waiting, and green is for Receiving.

The item that is loading here is an image.  So after the request reached the server, it waited 555ms before sending back a response, and that response took 277ms to transfer.

If you have a lot of images, you might spend a lot of time Receiving which is going to be limited by the bandwidth of your connection to the server.  This might mean you have too many images, or they are too big, or you have a slow connection to the server.

Also notice that there is a lot of blocking before this request can start.  Because images are typically the biggest assets loaded by the site, this is why a lot of people put them on a separate domain or a CDN (content delivery network) like Amazon S3.  Remember we talked about the web browser having a max of 6 requests at a time to any one domain.  If you split the images off onto a separate domain, they can start being pulled in parallel to your other requests and will avoid causing other assets to block further.  So you can then have 12 requests in parallel, 6 each to the different domains.

Also, the Waiting time would be time the server takes to pull the file off of the disk or a cache and start sending back.  If it is pulling from a slow disk on the server, or the server is under a lot of I/O load, this could further slow down that waiting time.  If the Waiting time becomes really long, like in the seconds, this can also be a symptom of the web server running into the max requests it can handle at once, in Apache, this is the MaxClients setting.  Also, if your web server is running out of memory to server the requests, it could end up using virtual memory, and that is another thing that would make the Waiting time get huge.

Another Tool to use With Firebug - YSlow

If you want to use a tool that does most of this analysis for you, without needing to know the technical details of how the Firebug Net panel works, use YSlow, which is a Firefox plugin that integrates with Firebug.

YSlow profiles your site and gives you a grade on a bunch of different optimization categories to help on the client side.  Also gives great details on the "Why?" of the optimization and ideas how to fix. But ultimately, you need to prioritize what is causing your slow site performance.  Even though YSlow might give you an F in a category, without using the Firebug Net panel, you don't really know what kind of load time hit it accounts for.  You might have more productive areas to look into.

YSlow is a great educational tool, even if your site is performing adequately, its a good idea to check your site out.

Use Firebug First Before you Waste Time

I always use Firebug before diving into tweaking things on the server.  You can waste a lot of time and not make a lot of difference, so for the sake of being efficient, start with Firebug first to narrow down your problems.


Subscribe to Front page feed