Avoiding bugs from cached JavaScript and CSS files in SharePoint

This article got quite detailed so here’s an executive summary – if you store custom JavaScript/CSS files etc. somewhere under SharePoint’s ‘layouts’ folder (or upload them manually to a library within the site), make sure the URL changes on each update. This is usually done with a querystring parameter in the URL e.g. /_layouts/MyProject/JS/MyScript.js?Rev=2011.11.20.1

Update 5th Dec 2011: Correction - if you exclusively use Microsoft's ScriptLink control to add JavaScript references to a page, the issue will NOT affect you if you store the .js files in “layouts”. It WILL, however, if you store them in a library and do not recycle IIS every time the file changes - something that should happen automatically for a WSP deployment to “layouts” of course. You may find though that it's not always possible to use ScriptLink exclusively - on my current project for example, we had many dependencies between JavaScript files, and ScriptLink does not allow you to specify the order (sequence) in which custom .js files should be added to the page (AFAIK). So, whilst your mileage may vary, I think the main thrust of the article still applies for many. In any case, many thanks to Mahmoud Hamed's comment below for spotting the flaw in my test results - I've updated the table below.

For most projects I work on, I prefer to store JavaScript/CSS etc. on the filesystem somewhere under the SharePoint root folder (e.g. “14”), usually under the “layouts” folder - for me, these files are often “site infrastructure” and are therefore critical for branding/functionality of the whole platform. That said, there are a couple of situations where I’d go the other route and store them in the site collection (and thus content database) – these would be:

• For CSS/JS files only applicable to one site (e.g. a team site)
• Sandboxed solutions

When I outlined these reasons in this answer to a forum question, the voting showed many agreed, and in general I see many folks using this approach. However, I’ve noticed recently that many are unaware of a crucial gotcha – and it’s so important that I’m constantly amazed it’s doesn’t get discussed more. The issue is end users not seeing updated JavaScript and CSS files following a recent code deployment. And OK, OK, I cheated a little bit with the article title – in fact the problem is not unique to SharePoint’s “layouts” folder (or even IIS), and when I say “JavaScript and CSS files”, I really mean JavaScript/CSS/images/audio/video and in fact most static files. But hey, sometimes framing a topic in a SharePointy way is the best way to remind folks that general behaviour of the web also applies to us ‘special’ SharePoint-types. It seems many folks may miss this stuff because they are focused on, say, some super-cool SharePoint ribbon development, and in terms of technical reading I think it’s fair to say that many focus almost exclusively SharePoint content.

Problem

As with many of life’s simplest pitfalls, this one is so easy to fall into. Remember that it applies only to SharePoint projects which do some degree of customization (e.g. branding/custom code) and store files on the filesystem – if those two things don’t apply to you, the problem is purely with Microsoft and they take care of it for their files, so there’s nothing to worry about. Otherwise, imagine the following:

1. Developer makes an update to some custom code – previously deployed files are updated. Let’s say we’re deploying some updated JavaScript and CSS.
2. The associated WSPs are deployed to SharePoint, and the files get updated across all web servers in the farm.
3. Users do not see the Javascript/CSS changes until they do a hard refresh (CTRL + F5). Worse still, they could experience script bugs or broken formatting which could make the application completely unusable.

This scenario typically results in the oft-heard question “Hmm, what happens if you do a CTRL + F5?”. Which is all fine, of course, for the test users who are in touch with the developers. In production, it’s certainly not fine for the rest of the organisation – strangely enough, asking 30,000 information workers to do a CTRL + F5 does NOT go down well in the enterprise! Things get even better in an internet/WCM scenario – would you really hear about JavaScript errors or CSS bugs from site users? Would that message actually get to the implementation team (who, let’s remember, are probably not seeing errors in event logs from this stuff)? Or would the users just go elsewhere?

Cause

The reason this happens is because the first time a file is requested, the web server (IIS here of course, but the same applies to most) serves it with a HTTP header which specifies that it can be cached for a long time. For the “layouts” folder, by default this is 31536000 seconds (1 year):

Browsers and proxies typically obey this, so on subsequent page requests, this file is pulled from ‘Temporary internet files’ on the user’s machine or a proxy between the user and the server. Without this, of course, web pages would load much more slowly. My testing showed that, generally, files served from content databases do not get served with the same headers and therefore updates show immediately. However, it turns out that SharePoint 2010 (and maybe earlier) treats files deployed using a Feature differently to files uploaded manually to the same library. My table below attempts to explain most of my testing and results:

Location	Blob cache enabled?	Result	Notes
Style library (publishing site)	No	OK
Style library (publishing site)	Yes	OK
Style library (publishing site)	Yes (now with max-age)	OK	Added by Feature
Style library (publishing site)	Yes (now with max-age)	Issue	Added manually to library
Layouts	Yes	OK	ScriptLink DID add querystring ?rev=q%2Fb304kubfwrNJVD%2BdYdxg%3D%3D but not clear when this gets updated – NOT when file changes!
Layouts	No	Issue	BLOB cache only relevant for files in content DB anyway
Custom library	Yes	Issue	Added manually to library
Custom library	Yes	OK	Added by Feature

Conclusions and notes:

• The issue applies to files in ‘layouts’ or files in a library which were not deployed using a Module tag in a Feature (or at least, which were uploaded through the browser)
  • N.B. I did actually go digging in Reflector to find the source of this behavior, but didn’t find it.
• For JavaScript files, it doesn’t matter how the file is added to the page (e.g. ScriptLink control or plain <script> tag)
• BLOB caching does not make any change
• Sandboxed or farm solution does not make any change
• Note – tested on SharePoint 2010 RTM (old) and SharePoint 2010 SP1 + August CU (fairly new at time of writing)

Solution

For the affected cases, to avoid browsers/proxies caching a file despite it being updated, wherever the file is linked simply add/change a querystring value on each deployment. So instead of something like:

• /_layouts/MyProject/JS/MyScript.js

..we need something like one of these:

• /_layouts/MyProject/JS/MyScript.js?Rev=1.0.0.0
• /_layouts/MyProject/JS/MyScript.js?Rev=2011.11.20.1

The parameter can have any name (though ‘rev’ or ‘revision’ is common), but the important thing is to change/increment it in some way on every modification. If you’re incrementing assembly versions (using AssemblyFileVersion rather than the main version number remember), then it may be appropriate to use same value – on my current project we do this in some places* (more on this later) so that we don’t have to remember to increment links like those above manually. Effectively we use code to read the current assembly file version, and append that as the ‘Rev’ querystring value (and this value will be the TFS automatically-generated build number for the current build). The trade-off, of course, is that you may be forcing users to re-download files the first time after an update which haven’t actually changed – but it’s safer than having bugs. As an aside, you may have noticed Microsoft do this with their links to CSS/JS files – their values will get updated in service packs/cumulative updates etc:

Attempts to solve the problem in a better way

So we can solve the problem by adding the querystring onto any URL which links to a CSS/JS file, great! In many cases though, this means remembering to do a find/replace in Visual Studio on all the locations which have links to these files before every deployment. This isn’t ideal – in our current project, this can be master pages, user controls, code-behind which uses ClientScriptManager and many other locations. What would be ideal is if all this could take care of itself on every release!

I haven’t spent too much time on this, but in our project we do use a solution for CSS files at least. All of our CSS files are emitted to the page using SharePoint’s CssRegistration and CssLink controls – this means all of the links are generated by one control, giving us a convenient ‘funnel’ which we can amend. So, we simply derive from the CssLink control, and in our custom class we rewrite the links to have the ‘Rev’ parameter. The value used is tied to the AssemblyFileVersion of our ‘core’ assembly (which also hosts this control), the idea being that the CI process updates this for every release. Of course, you could use whatever scheme you like:

   1: public class CacheSafeCssLinkControl : CssLink

   2: {

   3:     protected override void Render(HtmlTextWriter output)

   4:     {  

   5:         using (HtmlTextWriter tempWriter = new HtmlTextWriter(new StringWriter()))

   6:         {

   7:             base.Render(tempWriter);

   8:             string html = tempWriter.InnerWriter.ToString();

   9:             if (!string.IsNullOrEmpty(html))

  10:             {

  11:                 html = html.ToLower().Replace(".css\"", string.Format(".css?Rev={0}\"", Core.Common.Utilities.GetAssemblyFileVersion(Assembly.GetExecutingAssembly())));   

  12:             }

  13:             output.Write(html);

  14:         }

  15:     }

  16: }

So this works great for CSS files. But what about other files, especially JavaScript? Well, here’s where we hit the snag – unfortunately Microsoft.SharePoint.WebControls.ScriptLink (the equivalent control for JavaScript links) is sealed, so we can’t use the same approach. Interestingly, if you were happy to use reflection (perhaps with some caching), you could add the ‘Rev’ parameter using the same technique I use in Eliminating large JS files to optimize SharePoint 2010 internet sites – since the collection of links is stored in HttpContext.Current.Items, there’s an opportunity to modify the links before they are written to the page with no need to swap out Microsoft’s control.

I’d probably be comfortable with that, but the real issue in our project at least, is that JavaScript tends to be added to pages through other routes too such as ClientScriptManager and <script> tags. We’d need to update the (large) codebase to exclusively use ScriptLink so that we have a more effective funnel. Still, it gives me an idea of how I might look to do things in the future - in the long term though, it would be nice if Microsoft’s own controls offered a means of adding a cache-busting querystring.

Summary

Whilst developers happily use CTRL + F5 in their development processes, it’s often overlooked that end users will face the same issue of not seeing updated CSS and JavaScript files by default. Clearly this can cause all sorts of issues, including JavaScript bugs if the page HTML structure has changed but the corresponding JavaScript is not being used. The issue is mainly seen where such files are stored in the ‘layouts’ folder, but interestingly appears where files are manually uploaded to a library too (in SharePoint 2010 at least). The key is ensuring that the URL changes every time the file is updated, and since a querystring will suffice there’s no need to actually rename the file. Ultimately it doesn’t matter how you change the querystring value, whether it’s a simple manual update or a more automated process tied into CI.