Access end-user data (in the host web) from a SharePoint 2013 app

Continuing my series on developing SP2013 apps, in this article we’ll look at the special approaches needed to access data in the host web – meaning the regular lists/libraries the user interacts with outside of any created by your app. This is something that some apps will need to do, but others may not. For some, the whole premise around what the app does will be working with collaboration data in team sites/My Sites etc., so these techniques will be crucial there. Because there are so many flavors of SharePoint data access in apps, I’m going to spend some time explaining which this article applies to – feel free to scroll ahead to the code samples though!

Whilst this article series focuses on SharePoint-hosted apps some of the code/techniques I describe here may also work in other app scenarios, such as a provider-hosted app (e.g. where the app is implemented as a .NET site hosted on another server). At the time of writing (November 2012), the MSDN samples are *extremely* confusing as to which approach can be used in which scenario – I discuss this somewhat below, but essentially I will add more detail to this post as Office 365 and “provider-hosted” scenarios become clearer to me. For now, the scenario that all the samples below have been tested on is an on-premise SharePoint-hosted app which accesses data in the host web.

Here’s the evolving table of contents for this article series:

1. SharePoint 2013 apps – architecture, capability and UX considerations
2. Getting started – creating lists, content types, fields etc. within a SharePoint app (provisioning)
3. Working with data in the app web, and why you should
4. Access end-user data (in the host web) from a SharePoint 2013 app [this article]
5. Rolling out SharePoint 2013 apps to the enterprise - tenant scope and PowerShell installs
6. Azure is the new SharePoint ‘_layouts’ directory
7. “Host web apps” – provisioning files (e.g. master pages) to the host web
8. “Host web apps” – provisioning fields and content types
9. Deploying SP2013 provider-hosted apps/Remote Event Receivers to Azure Websites (for Office 365 apps)
10. Working with web parts within a SharePoint app

Before we launch into things, if you’re unsure what I mean by “host web” and “app web”, then Working with the app web, and why you should goes into some detail, but here’s a quick reminder:–

• Host web - the regular team site where our users access their documents etc., and where the app is installed
• App web - the “isolated” subweb which gets created to house any SharePoint artifacts the app creates (such as pages, lists etc.), and which the user is typically directed to when they enter the app 

Flavors of remote data access in apps

As you wade through MSDN samples and articles like this one, my recommendation is to understand which “flavor” of task you’re trying to accomplish, and what the article is discussing. The permutations are mind-boggling, and lead to much confusion on the journey to understanding apps – here are some examples:

App type	Remote data access
SharePoint-hosted	Page in app web accessing host web data using the cross-domain library and REST
SharePoint-hosted	Page in app web accessing host web data using JavaScript CSOM
Auto-hosted (Azure)/ provider-hosted/O365 app	Remote HTML page which uses cross-domain library to access data in app web
Auto-hosted (Azure)/ provider-hosted/O365 app	Remote HTML page which JavaScript CSOM to access data in app web
Auto-hosted (Azure)/ provider-hosted/O365 app	Remote HTML page which uses cross-domain library to access data in host web (maybe – haven’t seen this done yet)
Auto-hosted (Azure)/ provider-hosted/O365 app	Remote HTML page which crafts a standard AJAX request to host web or other SharePoint site, adding an OAuth token to the ‘Authorization’ header
Auto-hosted (Azure)/ provider-hosted/O365 app	Remote .NET code (e.g. web service, scheduled task etc.) which uses the .NET CSOM to request data in host web or other SharePoint site (using TokenHelper class to pass OAuth token)

Did I miss any? And note these are just accessing SharePoint list data – there are more scenarios around say, accessing SQL data in Azure/other locations.

The italicized ones are what I’m focusing on in this article. Interestingly, I notice the 3rd and 4th flavors are covered in some MSDN samples (see here and here respectively), so the techniques I’m writing about here should have broader usage than just my first two flavors. Confusingly however, those samples only appear to work when the provider-hosted remote site is running on http://localhost, and there are no notes to explain this – if you configure it as a bona-fide, deployable IIS website then the code stops working (and other things start to become required such as the X-UA-Compatible metatag). My guess is that browser/JavaScript security checks aren’t performed with localhost (it knows it isn’t really remote), and when this isn’t the case then “high-trust” app configuration is required for on-premise scenarios (this involves specifying the app client ID, using Register-SPAppPrincipal and so on). I’ll confirm this in the future.

One notable omission is “accessing data in the app web from the host web” (i.e. the reverse direction) – an example of this could be a web part within a team site which “reaches into” an app to read/write some data. The scenario sounds kinda useful to me, but I suspected the app model would not allow it. My tests showed that this is NOT possible, and there are blocking mechanisms in place – you’ll get a "Sorry, this site hasn't been shared with you" error message when you execute your remote call.

When and why – using the JavaScript cross-domain library (SP.RequestExecutor)

If you’ve done development with the SharePoint client object model (CSOM) before, you might be asking why we need a special JavaScript library to access some data from an app – surely we just pass the URL to the SP.ClientContext object right? Well no, this doesn’t always work. The answer is around cross-site scripting – or more correctly, the protection that browsers have to prevent cross-site scripting attacks. This is centered on the ‘same-origin’ policy used by JavaScript/web browsers, which specifies that some client code can only access data in the same URL domain – otherwise websites could effectively hack each other and the internet would be chaos.

However, in many app SharePoint app scenarios, you might want to access data which isn’t on the same URL domain as your code (typically JavaScript within a web page). So for example, if you’re following best practice and your SharePoint apps run on a separate URL domain (e.g. http://[identifier].cob-apps.dev compared to http://team.cob.dev in my test VM) then this will be an issue for you.

The cross-domain library is designed to simplify these JavaScript scenarios – particularly by removing the need to work with OAuth tokens.

How it works

So if JavaScript requests are usually prohibited from talking across URL domains, how is the cross-domain library able to do it? In short, the technique used is to dynamically create an IFrame on to a page in the host web – by default this is an out-of-the-box application page called AppWebProxy.aspx, though it’s also possible to use your own. This effectively turns the “remote” call into local call by accessing it on a host web URL such as http://team.cob.dev/_layouts/15/AppWebProxy.aspx  – the JavaScript code which talks to the SharePoint lists executes there, and from here it’s possible for the code to read the value displayed in the IFrame. Although the JavaScript is somewhat obfuscated since Microsoft generated it with Script# (like the rest of the CSOM), you can see some details of the implementation in SP.RequestExecutor.js:

 

Context - how I use all this in my “learner” time—tracking app

Before we get to the examples, for folks who are vaguely following this article series here’s how I’m using the cross-domain library in the dummy app I’m building. My app mainly stores it’s data in the app web, but just for fun there is a list in the host web which stores the “target hours” a user must log that week:

I use the library to query this list for the current user, then display it within my app:

 

Examples

1. Get title, URL or other properties of host web (_api/REST):

In this first example, I’ll try to cover some fundamentals for working with the REST service. (N.B. This is async/AJAX programming in JavaScript - if you haven’t done much of this before, you might need another reference to get you started. Check out Using the JavaScript Client OM and jQuery to work with lists). Essentially we build a REST URL (which we can test in the browser address bar), and then specify whether we want our data returned as JSON or XML. Most samples tend to use JSON, but I’ll show how to get XML later on - JSON does often make sense though, because it works so well with JavaScript/jQuery. So, we then  use some boilerplate async code to get a JSON object from the string returned by the service. As we work through these examples, the key thing to note is the data we asked for will always be in the jsonObject.d property – this will make sense as you see more of these samples, but it’s crucial to understand when you’re writing your own code:

var hostweburl;

var appweburl;

function sharePointReady() {

    // retrieve passed app web/host web URLs..

    hostweburl = decodeURIComponent($.getUrlVar("SPHostUrl"));

    appweburl = decodeURIComponent($.getUrlVar("SPAppWebUrl"));

    loadDependentScripts();

}

function loadDependentScripts() {

    var scriptbase = hostweburl + "/_layouts/15/";

    // Load the js files and continue to the successHandler

    $.getScript(scriptbase + "SP.Runtime.js",

        function () {

            $.getScript(scriptbase + "SP.js",

                function () { $.getScript(scriptbase + "SP.RequestExecutor.js", getAllHostWebPropsUsingREST); }

                );

        }

    );

}

function getAllHostWebPropsUsingREST() {

    var executor;

    // although we're fetching data from the host web, SP.RequestExecutor gets initialized with the app web URL..

    executor = new SP.RequestExecutor(appweburl);

    executor.executeAsync(

        {

            url:

                appweburl +

                "/_api/SP.AppContextSite(@target)/web/?@target='" + hostweburl + "'",

            method: "GET",

            headers: { "Accept": "application/json; odata=verbose" },

            success: onGetAllHostWebPropsUsingRESTSuccess,

            error: onGetAllHostWebPropsUsingRESTFail

        }

    );

}

function onGetAllHostWebPropsUsingRESTSuccess(data) {

    var jsonObject = JSON.parse(data.body);

    // note that here our jsonObject.d object (representing the web) has ALL properties because 

    // we did not select specific ones in our REST URL. However, we're just displaying the web title here..

    $('#hostWebTitle').text(jsonObject.d.Title);

}

function onGetAllHostWebPropsUsingRESTFail(data, errorCode, errorMessage) {

    alert('Failed to get host site. Error:' + errorMessage);

}

// jQuery plugin for fetching querystring parameters..

jQuery.extend({

    getUrlVars: function () {

        var vars = [], hash;

        var hashes = window.location.href.slice(window.location.href.indexOf('?') + 1).split('&');

        for (var i = 0; i < hashes.length; i++) {

            hash = hashes[i].split('=');

            vars.push(hash[0]);

            vars[hash[0]] = hash[1];

        }

        return vars;

    },

    getUrlVar: function (name) {

        return jQuery.getUrlVars()[name];

    }

});

So, given that our REST request asked for the host web and all it’s properties, if I use the IE Dev Tools script debugger to pause on the jsonObject.d property, I see some recognizable properties of the web:

In the code sample above, you can see that I ask for the “Title” property and put it in a DIV on the page with the following line:

$('#hostWebTitle').text(jsonObject.d.Title);

And to no surprise, this is what it looks like on the screen:

Just on a related note, if you only need one property (or a couple) from the web, it’s far more performant and efficient to only ask the server for these properties (since less data is downloaded to the client). To do this, the REST URL should change to something like (asking for the Title only here):

appweburl + "/_api/SP.AppContextSite(@target)/web/Title?@target='" + hostweburl + "'"

2. Querying for lists in the host web (_api/REST)

In this sample we ask for all the lists in the host web – for each one, we list out the title and the current number of items. The change to the REST part is simple (we use “/web/lists" as the URL), but once we have the data a little thought is needed – we have a collection of items here, and so some iteration is needed. Therefore, in my success handler I’m using jQuery’s each method with a callback which uses the Title and ItemCount properties.

Note that this is a cut-down sample:

// omitted for brevity: variable declarations, setup code to load dependent scripts, get hostweburl from querystring etc. (see previous example for this)

function getHostWebListsUsingREST() {

    var executor;

    // although we're fetching data from the host web, SP.RequestExecutor gets initialized with the app web URL..

    executor = new SP.RequestExecutor(appweburl);

    executor.executeAsync(

        {

            url:

                appweburl +

                "/_api/SP.AppContextSite(@target)/web/lists/?@target='" + hostweburl + "'",

            method: "GET",

            headers: { "Accept": "application/json; odata=verbose" },

            success: onGetHostWebListsUsingRESTSuccess,

            error: onGetHostWebListsUsingRESTFail

        }

    );

}

function onGetHostWebListsUsingRESTSuccess(data) {

    var jsonObject = JSON.parse(data.body);

    var lists = jsonObject.d.results;

    var listsHtml = $.each(lists, function (index, list) {

        $('#lists').append(list.Title + "(" + list.ItemCount + ")<br />");

    });

}

function onGetHostWebListsUsingRESTFail(data, errorCode, errorMessage) {

    alert('Failed to get host site. Error:' + errorMessage);

}

If I look at the data returned in a JavaScript debugger, I can see my collection:

..and on the page, I see my lists with their respective item counts:

3. Querying for specific items in a specific list in the host web (_api/REST)

This sample is my ‘real’ code in my time-tracking app - it's used in the screenshot displaying "target hours" that I showed earlier. Here, I do the usual setup work to ensure scripts are loaded first. Then, in this case, I’m querying for records belonging to the current user – so I need an initial CSOM request to find out who this is, and then once I have the username I can proceed with the main query. Because of this 2-step process I'll show the full code for this sample. As in the previous examples, I then use SP.RequestExecutor to call over to the host web and get my data - effectively I build up a REST URL with the details of the list I want to query/column I want to filter on, and pass this to RequestExecutor.executeAsync() – my data is then returned in JSON format:

var context;

var user;

var hostweburl;

var appweburl;

var utilTargetsList = "Utilisation%20targets";

var utilTargetsListFriendlyName = "Utilisation targets";

var userName = "Chris OBrien";

var fetchedUserName;

var targetHoursResult;

// This function is executed after the DOM is ready and SharePoint scripts are loaded

// Place any code you want to run when Default.aspx is loaded in this function

// The code creates a context object which is needed to use the SharePoint object model

function sharePointReady() {

    // retrieve passed app web/host web URLs..

    hostweburl = decodeURIComponent($.getUrlVar("SPHostUrl"));

    appweburl = decodeURIComponent($.getUrlVar("SPAppWebUrl"));

    loadDependentScripts();

}

function loadDependentScripts() {

    var scriptbase = hostweburl + "/_layouts/15/";

    // Load the js files and continue to the successHandler

    $.getScript(scriptbase + "SP.Runtime.js",

        function () {

            $.getScript(scriptbase + "SP.js",

                function () { $.getScript(scriptbase + "SP.RequestExecutor.js", runCode); }

                );

        }

    );

}

function runCode() {

    loadUser();

}

function loadUser() {

    var ctx = new SP.ClientContext.get_current();

    user = ctx.get_web().get_currentUser();

    ctx.load(user);

    ctx.executeQueryAsync(onGetUserNameSuccess, onGetUserNameFail);

}

function onGetUserNameSuccess() {

    fetchedUserName = user.get_title();

    // now we have a username, we can execute our main query..

    fetchTargetHoursFromHostWebUsingREST();

}

// This function is executed if the above OM call fails

function onGetUserNameFail(sender, args) {

    alert('Failed to get user name. Error:' + args.get_message());

}

function fetchTargetHoursFromHostWebUsingREST() {

    var executor;

    // although we're fetching data from the host web, SP.RequestExecutor gets initialized with the app web URL..

    executor = new SP.RequestExecutor(appweburl);

    executor.executeAsync(

        {

            url:

                appweburl +

                "/_api/SP.AppContextSite(@target)/web/lists/getbytitle('" + utilTargetsList + "')/items?@target='" +

                hostweburl + "'&$filter=Title eq '" + userName + "'&$select=Target",

            method: "GET",

            headers: { "Accept": "application/json; odata=verbose" },

            success: onGetTargetHoursByRESTSuccess,

            error: onGetTargetHoursByRESTFail

        }

    );

}

function onGetTargetHoursByRESTSuccess(data) {

    var jsonObject = JSON.parse(data.body);

    $('#targetHours').text(jsonObject.d.results[0].Target);

}

function onGetTargetHoursByRESTFail(data, errorCode, errorMessage) {

    alert('Failed to get host site. Error:' + errorMessage);

}

If you prefer to use the JavaScript CSOM instead of REST, you can do that – let’s look at a sample.

4. Querying for specific items in a specific list in the host web (JavaScript CSOM)

[NOTE – unlike the other techniques, I suspect this one won’t work in other app scenarios such as a provider-hosted app calling back to the app web/host web. To repeat, I’m really focusing about a page in a SharePoint-hosted app here.] So the previous samples have all used REST, but let’s say that we want to use the JavaScript client object model instead. In this case, use of the cross-domain library is not necessary – but there is a trick to being able to query the host web rather than the app web. Remember the app web is the default context because that’s where this code is running. To get to the host web using CSOM, I need to create an instance of SP.AppContextSite, passing it details of both app and host webs. As you’ll know if you’ve used the CSOM in ‘classic’ scenarios, the data isn’t returned in JSON here, but in collections that I can enumerate:

// omitted for brevity: variable declarations, setup code to load dependent scripts, get hostweburl from querystring etc. (see previous example for this) 

// start of core methods to fetch host web data using JavaScript CSOM..

function fetchTargetHoursFromHostWebCSOM() {

    appWebContext = new SP.ClientContext.get_current();

    var hostWebContext = new SP.AppContextSite(appWebContext, hostweburl);

    var targetHoursList = hostWebContext.get_web().get_lists().getByTitle(utilTargetsListFriendlyName);

    var query = new SP.CamlQuery();

    query.set_viewXml("<View><Query><Where><Contains><FieldRef Name='Title'/><Value Type='Text'>" + userName + "</Value></Contains></Where></Query></View>");

    targetHoursResult = targetHoursList.getItems(query);

    appWebContext.load(targetHoursResult);

    appWebContext.executeQueryAsync(onGetTargetHoursByCSOMSuccess, onGetTargetHoursByCSOMFail);

}

function onGetTargetHoursByCSOMSuccess(data) {

    var listEnumerator = targetHoursResult.getEnumerator();

    while (listEnumerator.moveNext()) {

        $('#targetHours').text(listEnumerator.get_current().get_item("Target"));

    }

}

function onGetTargetHoursByCSOMFail(data, errorCode, errorMessage) {

    alert('Failed to get host site by CSOM. Error:' + errorMessage);

}

NOTE:- If you need sample JavaScript for other tasks (e.g. adding an item to a list, deleting a list etc.) - check out How to: Complete basic operations using JavaScript library code in SharePoint 2013 – this deals with many such scenarios.

Tips for working with the REST _api

Determining the REST URL to use

If you’re choosing to use REST rather than CSOM, then the best tip I can give you is to work out the REST URL using the browser first, and then integrate this URL with your code (e.g. dealing with the app web/host web thing) as a second step. What I mean by this is simply typing in the browser address bar and checking the data returned - it will come back as XML by default, but you’ll still see which objects/properties come back.

So here’s me checking that I can fetch a single property of the web with a certain URL:

..and here’s me checking that I’ll get back all the lists in the web with a different URL:

You’ll have a far quicker feedback cycle with this approach, rather than waiting for the app to deploy for each change.

Returning XML rather than JSON

If for some reason you’d prefer to work with XML rather than JSON, then you can specify XML to be returned. Use the Accept header to do this – simply swap this line:

headers: { "Accept": "application/json; odata=verbose" }

..for:

headers: { "Accept": "application/xml" }

You’ll then need to adjust code in your success handler accordingly.

Permissions required to access host web data

Security, authentication, authorization and permission levels are all fairly big topics for SharePoint 2013 apps. However, the main point which is relevant to this article is that by default, code which accesses data in the host web (like my samples above) will not work until the app is granted permissions to data in the host web. Consider that if every app could access every user’s data, even just as read-only – well, the whole thing would be chaos.

The following bullets attempt to summarize the landscape here:

• Like a user, an app has it’s own identity which permissions can be granted to – this is known as the “app principal”
• If the app has an app web, the app principal automatically has Full Control to this area – nothing extra is required to read/write data in here
• In contrast, the app only has “basic permissions” to the host web or any other SharePoint site – you can get some core properties such as the web title, but not much more
• The app developer must code the app with a “permission request” specifying which data in the host web the app should be able to work with
• This is accomplished in the AppManifest.xml file – Visual Studio even has a designer to help get the XML right
• The person installing the app will then see which permissions are required, and must agree to this for the app to be installed

On a related note, in case you’re wondering apps are siloed from each other and cannot read/write each other’s data.

I mentioned that Visual Studio has a designer which helps with this – here it is with the app permissions section highlighted:

Although you may mainly work with AppManifest.xml in designer mode, for clarity let’s now consider things in “Code View” mode. In the examples above and in the time-tracking app I’m building, I want to read data throughout the host web. In this case, I need a permission request like this:

<AppPrincipal>

  <Internal />

</AppPrincipal>

<AppPermissionRequests>

  <AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web" Right="Read" />

</AppPermissionRequests>

For completeness, this is my full AppManifest.xml which contains this:

<?xml version="1.0" encoding="utf-8" ?>

<App xmlns="http://schemas.microsoft.com/sharepoint/2012/app/manifest"

     Name="COBSharePointAppsTimeTracking"

     ProductID="{5fec590d-3fd3-410d-82d9-77d13c2d3bdf}"

     Version="1.0.0.0"

     SharePointMinVersion="15.0.0.0"

>

  <Properties>

    <Title>COB Time Tracking app</Title>

    <StartPage>~appWebUrl/Pages/Default.aspx?{StandardTokens}</StartPage>

  </Properties>

  <!-- start of permission-related elements -->  

  <AppPrincipal>

    <Internal />

  </AppPrincipal>

  <AppPermissionRequests>

    <AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web" Right="Read" />

  </AppPermissionRequests>

  <!-- end of permission-related elements -->  

</App>

In many respects, demanding to read all data in the web is “a big ask” for an app. It might be more appropriate to restrict this to a specific list, in which case the declaration would look like:

<AppPrincipal>

  <Internal />

</AppPrincipal>

<AppPermissionRequests>

  <AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web/list" Right="Read" />

</AppPermissionRequests>

Notice that I can’t specify a particular list. I’m guessing Microsoft designed things this way because lists are pretty dynamic (and have unique identifiers assigned on creation) – so, what happens here is that the administrator installing the app selects *which* list the app has permissions to from a dropdown of all lists in the site:

Permission requests go far beyond data in SharePoint sites. As an app developer, you can also specify that the app needs to be allowed to use core SharePoint services such as search, taxonomy, user profiles and so on. Similarly it’s possible to specify “Capability Prerequisite” and “Feature Prerequisites”, to ensure the app is only used in environments which actually have these services running.

So, as you might imagine there are many permutations of permission requests – for more details, see App permissions in SharePoint 2013.

Summary

Some apps will need to access “real” end-user data outside of the app’s storage area (app web), though the must request permissions (and be granted them at install time) to do this. Several techniques can be used, including REST and the JavaScript CSOM (also known as JSOM). This article presents some code samples, as well trying to be clear on which of the many remote data scenarios they apply to.

In the future I’ll revisit the topic of using the JavaScript cross-domain library in remote provider-hosted apps.

Next time: Working with web parts within a SharePoint app