Provisioning Managed Metadata fields in Office 365 – Part 2: building WSP packages for a specific environment/tenancy

In the previous post I talked about how provisioning Managed Metadata fields is tricky for Office 365 (since developers cannot use server-side code to “hook-up” the field to the Term Store, like we can on-premises). I talked about the way we approach this, and in this post I’ll show the Visual Studio/MSBuild customisation we use to build WSPs specific to an Office 365 tenancy. Here’s how the articles relate to each other:

1. Provisioning Managed Metadata fields in Office 365 – dealing with multiple environments
2. Provisioning Managed Metadata fields in Office 365 – building WSP packages for a specific environment/tenancy [this article]

Using MSBuild/custom Project Configurations to build packages per Office 365 tenancy

Here’s what I came up with – it took quite a few long nights in the murky hall-of-mirrors world of “batching” in MSBuild, roughly 5 million discarded approaches and several new gray hairs. But finally, it works great for us:

We now have some custom project configurations which can be selected in Visual Studio:

 
So instead of just ‘Debug’ and ‘Release’ we are using:

• “CandCVM” – Content and Code (name of my employer) virtual machine. Used for when you are developing against your local VM rather than the cloud
• “DevTenancy” – the Office 365 tenancy we are using for DEV purposes
• “TestTenancy” – the Office 365 tenancy we are using for TEST purposes

Those are all the environments we need to worry about for now, but of course others will be needed eventually.

We also have an XML file in the VS project, which defines the Managed Metadata IDs for each environment – since we now only have to worry about the SspId, that’s the only value the file contains:

Once the project (or solution) configuration has been switched, we just build/publish the WSP to the filesystem as usual:

..and as you’d expect, we get the WSP(s) which will work against that environment:

What has happened, and how?

As the WSPs were being packaged, some custom MSBuild executed. What this does is:

• Finds the SspId_Replacements.xml file
• Reads the SspId value for the appropriate environment
• Performs a find/replace across all the of the XML files about to go into the package(s), and replaces the SspId value with the one read from the ‘replacements’ file

Notes:

1. Although it seems inefficient to look for an SspId and try to replace across *all* XML files (and it is), we do need to ensure that we catch:
   1. Managed Metadata field declarations (i.e. in elements.xml files)
   2. The section in schema.xml files (for list definitions) where the fields are duplicated
      
      I did look at trying to constrain the search beyond just the .xml extension, but found this difficult in MSBuild. I was also mindful of the fact that a developer could choose to have a Feature elements file NOT named elements.xml :) Fortunately the performance hit for us is negligible – the overall penalty in packaging for using this stuff (compared to a plain Debug or Release build) seems to be around 1-3 seconds – perfectly tolerable right now.
      
2. The replacement is XML-based (using XPath rather than string manipulation), so if you have XML comments nearby, this won’t trip up the replacement.

Benefit: integrating with Continuous Integration/automated build

One reason I wanted to implement in MSBuild is because I knew that it would be painless to use in a CI process. When implementing the build in TFS, we select the solutions/projects to build, and then define any custom Configurations we wish to use there also (i.e. in addition to developer machines). In our case, our CI builds deploy to our test tenancy:

We then make sure our build definition is using this Configuration (instead of Debug or Release):

Want to use this? Here’s the MSBuild..

If you think this could be useful to you, here are the details – essentially you need to edit your .csproj file, add the SspId_Replacements.xml file to your project and finally define the custom Configuration in Visual Studio. The sequence doesn’t matter, so long as all pieces are in place before you try a build. In team development, these steps need to be performed just once per VS project (e.g. by the lead developer).

Step 1 – add the custom MSBuild to your project:

Edit your .csproj file (each of them if you have multiple – the solution is currently “project-scoped”) to include this at the bottom:

** N.B. My newer code samples do not show in RSS Readers - click here for full article **

Step 2 – add the XML file with your config values:

Add an XML file within your project at the path “_Replacements\SspId_Replacements.xml” (N.B. this is configurable, see the MSBuild above). Add in the XML shown above, substituting the relevant SspId(s) for your environment(s). If you don’t know where to find the SspId for your environment, my colleague Luis mentions that in his post.

Step 3 – define the custom Configurations in Visual Studio:

In Visual Studio, define a custom Configuration for each environment you need to build WSPs for. Start by going to Configuration Manager and selecting “New..”:

..then create the Configuration, ensuring the same name is used as specified in the XML file – these two need to match. You’ll generally want to copy the setting from “Release”:

Now close Configuration Manager – the implementation is complete.

Building packages

To create WSPs, use the Configuration dropdown to switch to the “TestTenancy” (or whatever label you used) build type – be careful that the Platform stays as “Any CPU” here, you may need to change it back if not. Then just right-click on the project in Solution Explorer and click “Package” as usual – you should then get a WSP package in which the SspId find/replace has occured. You can now test deploying this to your Office 365 tenancy which corresponds to this build type.

A note on requirements

Note that the MSBuild used here is not compatible with SP2010/Visual Studio 2010 (but that’s OK because the entire technique is not needed there). There is a dependency on .NET 4.0 for the XmlPeek/XmlPoke MSBuild activities which are used to update the XML files.

Hopefully this is useful to some folks.

Provisioning Managed Metadata fields in Office 365 - Part 1: dealing with multiple environments

Managed Metadata fields have always been slightly painful for SharePoint developers, but if you did any kind of site templating or Feature development in SharePoint 2010, chances are that you did some research, read some blog articles and came to understand the solution. Here I’d like to talk about it for Office 365, and show a little customization I built to help with the process we currently use. This article became long, so I split it over two articles:

1. Provisioning Managed Metadata fields in Office 365 – dealing with multiple environments [this article]
2. Provisioning Managed Metadata fields in Office 365 – building WSP packages for a specific environment/tenancy

So, back on the SharePoint 2010 situation - the deal is that some Managed Metadata details change between SharePoint environments – so unlike other fields, the same provisioning XML could not be used across dev/test/UAT/production. To be specific, the IDs of the Term Store, Group and Term Set all change between environments. As a reminder, broadly the solution was to:

• Use XML to define the “static” details of the field
• Use server-side code to “finish the job” – i.e. use the API to ask SharePoint what the IDs are (for the current environment), and then update the field config with that value

Without the 2nd step, the field will be created but will be broken – it will be grayed out and users cannot use it (SP2010 example shown here):

Posts by Ari Bakker (whose image I’m using above, thanks Ari), Wictor Wilen and Andrew Connell were popular in discussing the steps to solve this.

It’s the same deal in SharePoint 2013/Office 365 – but it turns out we need different techniques.

Why the approach doesn’t work for Office 365/SharePoint Online

Well firstly, code in sandboxed solutions is deprecated. Full-stop. [RELATED - did you hear? Microsoft are starting to clarify that sandboxed solutions without code aren’t really deprecated after all (and hopefully MSDN will reflect this soon), but CODE in sandboxed solutions is deprecated and could be phased out in future versions. Clearly this is a very important distinction.]

But even if we were happy to use sandboxed code - in Office 365/SharePoint Online, we cannot use the Microsoft.SharePoint.Taxonomy namespace in server-side code anyway – the net result is that we are unable to “finish the job” in this way to ensure the field is correctly bound to the Term Store. This is a problem! Even worse, whilst it is possible in the CSOM API to bind the field, having this execute in the provisioning process (e.g. as a site is being created from the template) is challenging, maybe impossible. Maybe you could come up with some imaginative hack, but that’s probably what it would be. And what happens if this remote code (e.g. a Remote Event Receiver) fails?

Possible solutions

A colleague of mine, Luis Mañez, did some great research – I’ll give you a quick summary here, but I strongly recommend reading his article - Deploying Managed Metadata Fields declaratively in SharePoint 2013 Online (Office 365). Here’s a summary:

In fact, it IS possible to provision Managed Metadata fields without any code, if you are willing to accept a big trade-off – you can declaratively specify the key details (such as the Term Store ID (also known as the SspId), the Group ID, the Term Set ID etc.) into your XML field definitions. Wictor alluded to this possibility in his post. But remember, these details change between environments!

So in other words, the trade-off is that you would need to rebuild your WSPs for each environment.

This is tricky for us, because on this project we choose to run multiple Office 365 tenancies, for development/test/production (something I’ll talk about in future posts) – just like a traditional mature process. So at first we said “No way! That’s against all of our ALM principles!  The exact same packages MUST move between the environments!”. But then we rationally looked at the alternatives we could see:

• Option 1 - Some elaborate “remote code” solution, perhaps involving code running separately AFTER the site has been provisioned. Until this code executed, it would not be possible to upload documents to libraries with MM fields within the sites (and similarly if this remote call actually fails for any reason,  these libraries would not function correctly until an administrator intervenes).
• Option 2 - The client needs to fix-up any Managed Metadata fields manually – across all 5000 sites we were expecting. In every list and library. Knowing that some lists/libraries have up to 5 such fields. Yeah….

Since neither of these was attractive, we continued looking at this idea of a 100% declarative definition of Managed Metadata fields. And then we realized that..

..if you do things in a certain way, you can get to the point where ONLY THE TERM STORE ID (SspId) CHANGES BETWEEN ENVIRONMENTS. That’s kinda interesting. It means that just one find/replace operation is all that’s needed – assuming you’re happy to accept the overall trade-off. Of course, having to replace the SspId is still sub-optimal, error-prone and less than awesome. But maybe we could work on that too – and that’s what these posts are really about - to show a Visual Studio customization we made to simplify this process, and make it less prone to human-error. If you want to skip ahead to this, see Provisioning Managed Metadata fields in Office 365 – Part 2: building WSP packages for a specific environment/tenancy. 

But first, let’s talk about that “if you do things in a certain way” thing (which means that the only the SspId changes between environments)..

The full recipe for Managed Metadata fields

Taking a step back for a second, if you are in “development mode” (e.g. creating a template for SharePoint sites), then successful provisioning actually involves more than just provisioning the field itself in a certain way. Effectively you should seek to provision both the Term Sets AND the fields. Do not allow administrators to create new Term Sets in the admin interface. This is because:

• This way, you can control the ID of all your Term Sets – rather than let SharePoint generate that GUID
• Because this is a static “known” ID, we can reference it elsewhere

Here’s what needs to happen:

• Term Sets are provisioned into the Term Store with “known” IDs
  • We create Term Sets (often with some default Terms) in Office 365 using PowerShell and CSOM, instead of through the admin interface. In the article I referenced earlier, Luis provides you with the core PowerShell to do “taxonomy provisioning” in this way. In our project, we basically hook this up to an XML file which defines the desired Term Store structure (e.g. Groups, Term Sets etc.) – and this XML file defines the GUIDs of such objects; WE generated them – not SharePoint/Office 365
• The “known IDs” are then used in the XML definition of the fields
  • The code sample below is an example of a Managed Metadata field being provisioned the 100% declarative way. Notice all the properties being specified in the ‘Customization’ section (something a field using the combined declarative + code approach does not have):
    
    ** N.B. My newer code samples do not show in RSS Readers - click here for full article **

If you do this, your Managed Metadata fields will work just fine:

Great. It’s certainly very valuable to know this is possible for Office 365. So now we have things so that only the SspId value needs to change between environments. But that’s still a nasty find/replace operation – how could we make this situation better?

I describe the mechanism we use in the next post - Provisioning Managed Metadata fields in Office 365 – Part 2: building WSP packages for a specific environment/tenancy.