Avoiding dependency issues in SharePoint Framework (SPFx) development

One area I talk about in my “Avoiding pitfalls when developing with the SharePoint Framework” talk is dependencies, and the problems you can run into. Whether you depend on jQuery, Angular, React or other JS libraries is up to you, but there will always be some – and npm is typically the way to bring these libraries into your SPFx solution. Using a TypeScript/JavaScript-based code stack and npm packages is completely different from .NET, and I think most SharePoint developers are fairly new to this world. There’s quite a lot to learn with npm itself, but this is important for successful team development with SPFx. Of course, npm, gulp and node.js might be familiar territory if you’ve already done lots of modern JS-focused web development (by which I don’t mean jQuery!), but there are deep topics here beyond just the coding aspect.

To get the development and ALM process right with SPFx, some of the answers lie in use of commands like npm shrinkwrap, having a solid understanding of the package.json file and how developers specify different types of dependencies. But before we get to that in detail, let’s start by talking about the problems and some general background to understand.

I use the slide below to summarise some key pitfalls in this area, splitting them into “dev” and “shipping” considerations:

What are caret dependencies (never mind why they can be a pitfall)? What does –save do on the npm install command? If those are questions you’re asking right now, then my companion post SPFx - an overview of node_modules, package.json and other node concepts might be useful.

Otherwise, let’s get to the detail.

Not specifying --save on npm install

This is a newbie pitfall for teams/developers not very familiar with npm. When a developer adds a package with npm install, the package is downloaded from the npm repository and installed to your application – at least, in the machine and folder that the command is run on. Adding the --save flag ensures the dependency is also recorded in package.json – thereafter, when any dev runs npm install or npm update for this application, npm will again ensure the appropriate package is downloaded and installed locally (because the package.json file is shared via source control). It’s the equivalent of adding a .NET dependency via NuGet, compared to just adding a local file reference. As ever, checking the full set of packages themselves into source control is usually a bad idea. So, the pitfall is simple - if the dev forgets to use the --save flag when installing a package, no entry in package.json is made and code may fail due to a missing library on other machines. You’ll see a TypeScript compile error like “Cannot find module” when another developer tries to build or run the code (e.g. with gulp serve):

Recommendation – always track your dependencies by specifying the --save parameter (or one of the options such as –save-exact) when running npm install.

You might also consider a custom .npmrc file for your team here, which can automatically specify certain parameters (e.g. --save or --save-exact) without the dev having to remember them. I talk about how a custom .npmrc file works in the other post.

N.B. if you do fall into the trap of using a dependency without it being properly stored in package.json, simply install it again *with* the --save (or similar) flag. Then have each dev run an npm update to ensure everyone is using the same version.

Pitfall - using caret dependencies without thinking about it

When adding an npm package, the default is to use a caret dependency. However, both caret and tilde dependencies bring an element of risk to the dev process (but can have benefits too). If you’re not familiar with version numbers which use carets or tildes, read either my background post or semantic versioning (semver) in the npm docs.

Briefly, the caret character (^) denotes that any version up to (but not including) 4.0.0 is acceptable. Caret dependencies like this are the default with npm, but can lead to danger because there can be all sorts of changes within that version range, potentially including breaking changes, depending on how the package author handles things. Not everyone plays by the semver rules. So, that means that in team development, one person on the team can be using 3.1.1 and another 3.9.9 (i.e. releases of the same major version, but different minor versions), depending on how when each last ran the npm update command. So, there can be lots of variability from caret dependencies like these.

Tilde dependencies (~) are similar, except here the range is more narrow – so ~3.1.1 would allow any version up to (but not including) 3.2.0 (i.e. releases of the same major/minor version, but different patch versions).

Note also that I’m simplifying things here slightly, since npm deals with things differently when a zero is used in the version number. See the semver page for more details.

So are caret and tilde dependencies always bad?

Well no, not really. You might decide that you DO want the latest and greatest of libraries you are using during the dev process, without the hassle of managing each individually. Often you do indeed want those minor bug fixes and performance enhancements – and avoiding “exact” dependencies gives you that (so long as you npm update regularly).

Recommendation – make an explicit decision about whether you want “floating” dependencies in dev.

If NO, consider these options:

• Ensure all developers always use the –save-exact flag instead of the –save flag
  • This will have the effect of writing in an exact reference into package.json, rather than a caret or tilde dependency
  • Use a custom .npmrc file across the team to have the same effect (without developers having to remember –save-exact every time they install a package)
• Again, see my SPFx - an overview of node_modules, package.json and other node concepts for details on this technique

If YES, then:

• Stick with caret dependencies, but consider becoming “tighter” as you approach releasing your web part – ensure all devs are running npm update regularly, and perhaps switch to exact dependencies just before shipping.

Whether you use exact dependencies in dev or not, there’s something that I recommend you always do to freeze dependencies at ship time of each release, and that’s to run npm shrinkwrap. We’ll talk about that next..

Pitfall – not locking down dependencies when you make a release

In SPFx and node.js development, it’s crucial to remember that the node_modules folder is an entire tree of dependencies. First level dependencies that you know about have their own dependencies, which are stored in child node_modules folders – that’s why the node_modules folder can be big in size and go quite deep. The steps we’ve talked about so far, such as considering use of --save-exact to avoid caret/tilde dependencies, only get you so far – and that’s because they deal with your first level dependencies only. Consequently, you’ll find it difficult to recreate the *exact* build you shipped as v1 because you can’t restore the exact node_modules folder you had at the time of this build. Sure, you could if you always checked the entire mode_modules folder into source control for every build, but yuck!

What you need is for the entire tree of dependencies to be frozen for each release – the npm shrinkwrap command gives you this. It creates a JSON file containing details of which version was resolved of every package used:

Thereafter, if the npm install or npm update commands are used against a folder containing such a file (named npm-shrinkwrap.json), npm will look to it to restore the packages. This is different to the default behaviour which will use any caret or tilde dependencies (etc.) specified in the package.json file. Thus, with this approach you can restore the entire build as it happened at a certain time, without having the node_modules folder in its state at that time.

Recommendation – run npm shrinkwrap every time you release a version of your code. Store the npm-shrinkwrap.json in source control.

Summary

Creating SharePoint Framework solutions effectively involves more than just learning the new APIs. You need to build a good understanding of the underlying web stack, and arguably some elements are more important than others – modules, packages and npm is a combined area that I recommend spending time on. Without this, it’s easy to fall into some pitfalls when developing or shipping your SPFx solution. Overall I recommend:

• Taking care when developers add libraries – ensure they are tracked in package.json
• Deciding on whether you’re happy with floating dependencies during dev, or whether you prefer to use exact dependencies
• Running npm shrinkwrap each time you release a version of your solution, and checking the npm-shrinkwrap.json file into source control. This will allow you to rebuild the app properly later on, when never versions of your dependencies have been released

SPFx - an overview of node_modules, package.json and other node concepts

N.B. this is a companion post to Avoiding dependency issues in SharePoint Framework (SPFx) development

Any SharePoint Framework (SPFx) solution that does something useful will use external JavaScript libraries – whether bigger frameworks such as React or Angular, or smaller, more focused libraries such as LoDash, moment.js or something else. Since SPFx is based on a node.js-based web stack, the best way to integrate such libraries into your solution is usually to add them from npm, the internet-based package manager for JavaScript. Npm is effectively an equivalent of NuGet in the .NET world. Once you’re set up with npm, a developer simply runs npm install jquery [or whatever] and the source code for the package will be added to your application’s files. As is normal with node development, the library is added under the “node_modules” subfolder, and it can now be referenced in your code.

The role of the node_modules folder

So the node_modules folder stores all your dependencies. But it also stores any dependencies of those, and so on and so on. So, it’s a tree structure which can be quite deep. An app which uses jQuery and React for example, could look like this:

In this example, jQuery has a dependency on “cache-swap”, which also has its own dependencies in its node_modules folder. And so on..

Modules which are actually used by your code are distributed with your app, so everything works at run time. The bundling mechanism built into the SharePoint Framework tooling takes care of this using webpack.

Ensuring dependencies are tracked with package.json

If the --save flag is also specified when adding a package with npm install, an entry is written into the package.json file in the application’s directory. This goes into the “dependencies” section of the file, and will look something like this:

"jquery": "^3.1.1"

Here’s an example of full package.json file:

Notice the caret (^) in that version number above. I’ll talk about tilde and caret dependencies later. But the purpose of that –save flag is is to ensure that the dependency is recorded. The node_modules folder itself is typically *not* checked-in to source control (in the same way all NuGet packages would not be), partly because it is large and unwieldy. Instead, each developer has a local copy on their machine, and the shared package.json file plays a critical role in ensuring all devs have the same files. When another developer subsequently obtains the files from source control, running the npm install or npm update commands will ensure all dependencies are restored on his/her machine, based on what’s stored in the package.json file.

As I detail in Avoiding dependency issues in SharePoint Framework development, if the developer forgets to track the dependency (with the --save flag or similar), then other devs are likely to be missing the module on their machines and the code will not work for them – they’ll get a missing module error. However, even if developers do always remember to specify the --save flag, you can run into problems in dev because npm uses caret dependencies by default, and these are a form of “floating version” dependency. I won’t discuss the pitfall further here, since we’re all about the fundamental concepts in this post – but related to all this is the whole topic of semantic versioning (known as semver).

A brief summary of semantic versioning

Semver refers to the different kinds of version numbers which can be used by npm packages. The semver page in the npm docs is recommended reading, but I’d say the core things you need to understand are:

• The 3 part version number format i.e. MAJOR.MINOR.PATCH, and the rules for when each number should be updated – see http://semver.org
• The different forms of version number used in package.json

Version number forms include:

Type	Example	What it does
Caret dependency	^3.1.1	Allows any version on the same major version, i.e. anything below 4.0.0
Tilde dependency	~3.1.1	Allows any version on the same minor version, i.e. anything below 3.2.0

However, my table above is a simplification because npm deals with things differently if there is a zero in the version number. Additionally, npm supports other forms too such as pre-release versions like “1.0.0-alpha.1” and tags. I recommend spending time in the npm docs, but there are LOTS of simplified posts on the internet too if you prefer another format e.g. https://nodesource.com/blog/semver-tilde-and-caret

Setting npm defaults to get consistency across devs

Elsewhere we’ve talked about parameters to pass to npm commands, such as the --save or--save-exact flags used with npm install. However, a useful approach with a dev team can be to set defaults across the team so that each developer does not need to remember a certain flag each time a command is run. This can be done in a couple of ways:

• Setting environment variables on each machine
• Using a .npmrc file – at the project, user or global level

I like the idea of using the .npmrc file at the user level (e.g. C:\users\chris\.npmrc on my machine), and ensuring all team members have this in place. By creating/editing this file, I can specify for example that all dependencies should be saved into package.json with *exact* version numbers, rather than caret or tilde dependencies. I can do this by adding the following to my file:

save = true
save-exact = true

Now, when any dev installs a package, the following differences to the default behaviour will occur:

• The dependency will automatically be added to package.json, even if the dev accidentally forgets the --save flag
• The dependency will be added with an exact version number, not a caret version number

This is useful, as devs won’t accidentally fall into some of the pitfalls that can come about from the npm defaults.

Also check out the save-prefix config flag as an option to override caret dependencies (e.g. to change to tilde dependencies).

Summary

Hopefully this is some useful background information to concepts underneath the SharePoint Framework. It can seem a whole new world in some respects, but at the same time I don’t think you have to be a complete expert in npm, modules, webpack etc. to be productive. A little extra digging around sub-topics like these is recommended though I think. Have fun!

Changes to Custom Actions and handling of JavaScript in SharePoint Online

Recently our Custom Actions stopped working in modern document libraries in Office 365. We were surprised at this, because as far as we were aware we weren’t doing anything that went against “new” guidance for working with modern team sites and modern document libraries. Since the beginning of modern doc libs, Custom Actions which use ScriptSrc or ScriptBlock have not been supported (same with JSLink customizations). But we weren’t doing any of these things – our specific details were:

• A Custom Action using a URL-based action which opened a modal dialog (pop-up) – in our case, to show a custom options page to the user
• Use of “EditControlBlock” as the location, so our menu item appears on the context menu for each document
• Registered on our “base document” content type which is used in our document libraries

And for a while everything was great - as expected, our menu item showed up in both the classic and modern document library experience. However, in early December (2016) our menu item disappeared from the modern experience, across all our tenancies (eventually). These images show our menu item (named “Actions”) visible in the classic view but not modern:

Classic:

Modern:

New rules for Custom Actions

Essentially, Microsoft have made changes in exactly what is supported with Custom Actions. I guess you could say this doesn’t align 100% with previous guidance on the topic – or at least, some low-level details were ambiguous or open to interpretation. For example, the Update on Modern Document Libraries and Extensibility blog post says this:

“We’ve already made some good progress here.  Theming, global navigation links, and URL-based custom actions that extend the ribbon and context menus are already supported in the modern document library experience.  This ensures that customers and partners taking advantage of these features can use the modern document library experience without compromising their customizations.”

For our part, we figured that we should be fine because we’re using a URL (rather than ScriptSrc or ScriptBlock). However, talking with the Office 365 engineering team, it became clear that a URL which contains JavaScript (e.g. Url=’javascript:OpenDialog()’) is no longer permitted – although this did work until early December. Additionally, targeting by content type no longer works – the only option currently is to target by list template type, for example:

• 100 – custom list
• 101 – document library
• ..and so on

UPDATE – targeting by content type now works again. So you can once again target by content type or list template type..

I appreciated the help from folks in the product group who helped me understand what their code is currently doing, and where things might head in the future – thanks guys.

Anyway, we had to make some changes. In our case, since we can no longer open a dialog from the menu (since that requires some JavaScript), we’re changing our experience to navigate the user to a full page instead. Also, we need to change the targeting from content type to list template type – that’s mainly OK, but in truth does give us a slightly different scope to what we really wanted.

Could any of these changes affect you? If so, you’ve probably noticed already – but either way, it’s definitely worth SharePoint developers becoming clear on the latest developments in this space. Overall, there is a longer list of these kind of things which are no longer supported on modern lists and libraries, including:

• Custom master pages
• JSLink (field and view types)
• AlternateCssUrl
• Custom Actions which use ScriptLink to add JavaScript to the page

On the bright side, there is now some official documentation in the form of MSDN articles which provide more detail than I’m including here. I highly recommend going through these:

• Customizing "modern" lists and libraries (most relevant to this article)
• Customizing "modern" team sites
• Customizing "modern" site pages

Using the classic experience as a workaround

Remember of course that these changes don’t affect the classic experience. You have the choice of continuing to use that (e.g. by setting it at the tenant level) if you’re happy with that. We weren’t unfortunately, because we want our uses to have the benefits of the new document library UX (better mobile experience, improved metadata editing, new summary panel, sorting/filtering enhancements and so on).

Looking to the future (especially around JavaScript in Office 365)

Microsoft have noted in several places that new mechanisms will come to SharePoint Online to match many of the previous capabilities. They may not take exactly the same form as the previous mechanisms, but it should be possible to achieve the same overall result. These include:

• Custom master pages --> some other “deep branding” controls (which may or may not provide the ability to control the full HTML)
• JSLink --> some other mechanisms to control the rendering of list views and fields
• CustomAction + ScriptLink --> some other mechanism to add JavaScript to pages

Thinking specifically about JavaScript, I think it’s clear that there will be increased controls around administrator approval of scripts – the idea is to provide more governance so that arbitrary scripts cannot be added by a developer without wider approval. After all, that’s why Microsoft have made some of the recent changes to pull back on previously supported approaches – they did leave holes which could result in trouble for some organizations. Despite the occasional pain on the journey, I think we all need to accept that these changes are for the better and the resulting position should be better overall.