Eliminating large JS files to optimize SharePoint 2010 internet sites

Back in the SharePoint 2007 timeframe, I wrote my checklist for optimizing SharePoint sites – this was an aggregation of knowledge from various sources (referenced in the article) and from diagnosing performance issues for my clients, and it’s still one of my more popular posts. Nearly all of the recommendations there are still valid for SP 2010, and the core tips like output caching, BLOB caching, IIS compression etc. can have a huge impact on the speed of your site. Those who developed SharePoint internet sites may remember that suppressing large JavaScript files such as core.js was another key step, since SharePoint 2007 added these to every page, even for anonymous users. This meant that the ‘page weight’ for SharePoint pages was pretty bad, with a lot of data going over the wire for each page load. This made SharePoint internet sites slower than they needed to be, since anonymous users didn’t actually need core.js (since it facilitates editing functionality typically only needed for authenticated users) and indeed Microsoft published a workaround using custom code here.

The SP2010 problem

To alleviate some of this problem, SharePoint 2010 introduces the Script On Demand framework (SOD) – this is designed to only send JavaScript files which are actually needed, and in many cases can load them in the background after the page has finished loading. Additionally, the JavaScript files themselves are minified so they are much smaller. Sounds great. However, in my experience it doesn’t completely solve the issue, and there are many variables such as how the developers reference JavaScript files. I’m guessing this is an area where Your Mileage May Vary, but certainly on my current employer’s site (www.contentandcode.com) we were concerned that SP2010 was still adding some heavy JS files for anonymous users, albeit some apparently after page load thanks to SOD. Some of the bigger files were for ribbon functionality, and this seemed crazy since our site doesn’t even use the ribbon for anonymous users. I’ve been asked about the issue several times now, so clearly other people have the same concern. Waldek also has an awesome solution to this problem involving creation of two sets of master pages/page layouts for authenticated/anonymous users, but  that wasn’t an option in our case.

 N.B. Remember that we are primarily discussing the “first-time” user experience here – on subsequent page loads, files will be cached by the browser. However, on internet sites it’s the first-time experience that we tend to care a lot about!

When I use Firebug, I can see that no less than 480KB of JavaScript is being loaded, with an overall page weight of 888KB (and consider that, although this is an image-heavy site, it is fairly optimized with sprite maps for images etc.):

If we had a way to suppress some of those bigger files for anonymous users entirely, we’d have 123KB of JavaScript with an overall page weight of 478.5KB (70% of it now being the images):

But what about page load times?

Right now, if you’ve been paying attention you should be saying “But Chris, those files should be loading after the UI anyway due to Script On Demand, so who cares? Users won’t notice!”. That’s what I thought too. However, this doesn’t seem to add up when you take measurements. I thought long and hard about which tool to measure this with – I decided to use Hammerhead, a tool developed by highly-regarded web performance specialist Steve Souders of Google. Hammerhead makes it easy to hit a website say 10 times, then average the results. As a sidenote, Hammerhead and Firebug do reasssuringly record the same page load time – if you’ve ever wondered about this in Firebug, it’s the red line in Firebug which which we care about. Mozilla documentation defines the blue and red lines (shown in the screenshots above) as:

• Blue = DOMContentLoaded. Fired when the page's DOM is ready, but the referenced stylesheets, images, and subframes may not be done loading.
• Red = load. Use the “load” event to detect a fully-loaded page.

Additionally, Hammerhead conveniently simulates first-time site visitors (“Empty cache”) and returning visitors (“Primed cache”) - I’m focusing primary on the first category. Here are the page load times I recorded:

Without large JS files suppressed:

With large JS files suppressed:

Reading into the page load times

Brief statistics diversion - I suggest we consider both the median and average (arithmetic mean) when comparing, in case you disagree with my logic on this. Personally I think we can use average, since we might have outliers but that’s fairly representative of any server and it’s workload. Anyway, by my maths the differences (using both measures) for a new visitor are:

• Median – 16% faster with JS suppressed
• Average – 24% faster with JS suppressed

Either way, I’ll definitely take that for one optimization. We’ve also shaved something off the subsequent page loads which is nice.

The next thing to consider here is network latency. The tests were performed locally on my dev VM – this means that in terms of geographic distance between user and server, it’s approximately 0.0 metres, or 0.000 if you prefer that to 3 decimal places. Unless your global website audience happens to be camped out in your server room, real-life conditions would clearly be ‘worse’ meaning the benefit could be greater than my stats suggest. This would especially be the case if your site has visitors located in other continents to the servers or if users otherwise have slow connections – in these cases, page weight is accepted to be an even bigger factor in site performance than usual.

How it’s done

The approach I took was to prevent SharePoint from adding the unnecessary JS files to the page in the first place. This is actually tricky because script references can originate from anywhere (user controls, web parts, delegate controls etc.) – however, SharePoint typically adds the large JS files using a ClientScriptManager or ScriptLink control and both work the same way. Controls on the page register which JS files they need during the page init cycle (early), and then the respective links get added to the page during the prerender phase (late). Since I know that some files aren’t actually needed, we can simply remove registrations from the collection (it’s in HttpContext.Current.Items) before the rendering happens – this is done via a control in the master page. The bad news is that some reflection is required in the code (to read, not write), but frankly we’re fine with that if it means a faster website. If you’re interested in the details, it’s because it’s not a collection of strings which are stored in HttpContext.Current.Items, but Microsoft.SharePoint.WebControls.ScriptLinkInfo objects (internal).

Control reference (note that files to suppress is configurable):

<!-- the SuppressScriptsForAnonymous control MUST go before the ScriptLink control in the master page -->

<COB:SuppressScriptsForAnonymous runat="server" FilesToSuppress="cui.js;core.js;SP.Ribbon.js" />

<SharePoint:ScriptLink language="javascript" Defer="true" OnDemand="true" runat="server"/> 

The code:

using System;

usingSystem.Collections;

using System.Collections.Generic;

using System.Reflection;

using System.Web;

using System.Web.UI;

namespace COB.SharePoint.WebControls

{

    /// <summary>

    /// Ensures anonymous users of a SharePoint 2010 site do not receive unnecessary large JavaScript files (slows down first page load). Files to suppress are specified 

    /// in the FilesToSuppress property (a semi-colon separated list). This control *must* be placed before the main OOTB ScriptLink control (Microsoft.SharePoint.WebControls.ScriptLink) in the 

    /// markup for the master page.

    /// </summary>

    /// <remarks>

    /// This control works by manipulating the HttpContext.Current.Items key which contains the script links added by various server-side registrations. Since SharePoint uses sealed/internal 

    /// code to manage this list, some minor reflection is required to read values. However, this is preferable to end-users downloading huge JS files which they do not need.

    /// </remarks>

    [ToolboxData("<{0}:SuppressScriptsForAnonymous runat=\"server\" />")]

    public class SuppressScriptsForAnonymous : Control

    {

        private const string HTTPCONTEXT_SCRIPTLINKS = "sp-scriptlinks";

        private List<string> files = new List<string>();

        private List<int> indiciesOfFilesToBeRemoved = new List<int>();

        public string FilesToSuppress

        {

            get;

            set;

        }

        protected override void OnInit(EventArgs e)

        {

            files.AddRange(FilesToSuppress.Split(';'));

            base.OnInit(e);

        }

        protected override void OnPreRender(EventArgs e)

        {

            // only process if user is anonymous..

            if (!HttpContext.Current.User.Identity.IsAuthenticated)

            {

                // get list of registered script files which will be loaded..

                object oFiles = HttpContext.Current.Items[HTTPCONTEXT_SCRIPTLINKS];

                IList registeredFiles = (IList)oFiles;

                int i = 0;

                foreach (var file in registeredFiles)

                {

                    // use reflection to get the ScriptLinkInfo.Filename property, then check if in FilesToSuppress list and remove from collection if so..

                    Type t = file.GetType();

                    PropertyInfo prop = t.GetProperty("Filename");

                    if (prop != null)

                    {

                        string filename = prop.GetValue(file, null).ToString();

                        if (!string.IsNullOrEmpty(files.Find(delegate(string sFound)

                        {

                            return filename.ToLower().Contains(sFound.ToLower());

                        })))

                        {

                            indiciesOfFilesToBeRemoved.Add(i);

                        }

                    }

                    i++;

                }

                int iRemoved = 0;

                foreach (int j in indiciesOfFilesToBeRemoved)

                {

                    registeredFiles.RemoveAt(j - iRemoved);

                    iRemoved++;

                }

                // overwrite cached value with amended collection.. 

                HttpContext.Current.Items[HTTPCONTEXT_SCRIPTLINKS] = registeredFiles;

            }

            base.OnPreRender(e);

        }

    }

}

Usage considerations

For us, this was an entirely acceptable solution. It’s hard to say whether an approach like this would be officially supported, but it would be simple to add a “disable” switch to potentially assuage those concerns for support calls. Ultimately, it doesn’t feel too different to the approach used in the 2007 timeframe to me, but in any case it would be an implementation decision for each deployment and it may not be suitable for all. Interestingly, I’ve shared this code previously with some folks and last I heard it was probably going to be used on a high-traffic *.microsoft.com site running SP2010, so it was interesting for me to hear those guys were fine with it too.

Additionally, you need to consider if your site uses any of the JavaScript we’re trying to suppress. Examples of this could be SharePoint 2010’s modal dialogs, status/notification bars, or Client OM etc.

Finally, even better results could probably be achieved by tweaking the files to suppress (some sites may not need init.js for example), and extending the control to deal with CSS files also. Even if you weren’t to do this, test, test, test of course.

Summary

Although there are many ways to optimize SharePoint internet sites, dealing with page weight is a key step and in SharePoint much of it is caused by JavaScript files which are usually unnecessary for anonymous users. Compression can certainly help here, but comes with a trade-off of additional server load, and it’s not easy to calculate load/benefit to arrive at the right compression level. It seems to me that it would be better to just not send those unnecessary files down the pipe in the first place if we care about performance, and that’s where I went with my approach. I’d love to hear from you if you think my testing or analysis is flawed in any way, since ultimately a good outcome for me would be to discover it’s a problem which doesn’t really need solving so that the whole issue goes away!