Announcements from the SharePoint Virtual Summit 2017

For those who were unable to attend the SharePoint Virtual Summit today, while you wait for the recording to become available, or you are unable to watch the full recording, here are most of the announcements that I recorded from the event, and some links to other recent announcements also found in the Q&A.

Read more…

SharePoint 2013: The Branding Dilemma – Conflicting Guidance on Themes with Custom CSS vs Master Pages

June 22, 2015 2 comments

The Background

We are working with a global client who has a hybrid SharePoint 2013 environment that is growing in scale that will soon consist of many in-country SharePoint farms, as well as having a ‘global’ data centre and using Office 365-D. Within a couple of years these farms will host hundreds of thousands of site collections created based on our template solutions.

These solutions need to be able to be deployed to any one of these SharePoint farms including Office 365-D and so cannot use full-trust server-side solutions, as these are not permitted in these environments, with Office 365-D initially driving the need to avoid full-trust solutions, particularly when considering the planned move to Office 365-D vNext, so we need to try to avoid potential upgrade problems as much as we can.

For this we are embracing the Cloud App Model, i.e. client-side object models and APIs – CSOM, JSOM, REST, and where appropriate create SharePoint Apps/Add-Ins.

Standardised Solution Deployments Using CSOM & XML

Early last year we developed a remote deployment solution that allows us to create and deploy SharePoint artefacts entirely using CSOM and REST, with the solution defined as simple XML, so being able to create complete solutions built on top of out of the box team sites deployed remotely via CSOM, avoiding creating Site Definitions or Web Templates that have dependencies on WSP solutions (which Microsoft are trying to move away from). We designed this deployment solution such that the same package can be deployed into O365-D, O365-Tenanted, or On-Premises without any change to the code/package. Of course we still have to consider the nuances of each type of environment and make sure our solutions will work with those. These differences are largely handled within the deployment solution, using the APIs appropriately for each type of environment.

This is all working well and we have deployed many solutions using this approach. We are following Microsoft guidance, patterns and practices, and industry good practices. I don’t like the term ‘best practice’ as it seems arrogant to me if I might say one approach is the ‘best’ – this can be considered subjective where there might be more than one good approach and another might be better depending on the circumstances. Indeed there are several Microsoft Patterns & Practices that achieve a similar outcome.

Determining Branding & UI Customisation Guidance

Anyway, on to the main point of this post! Our client often requires a complex customised user interface built on SharePoint. We’re trying to formulate guidance on user interface customisations that we can give to design agencies, which often have no SharePoint experience and therefore produce fancy UI designs that generally do not fit SharePoint very well. The guidance will also be given to the developers who need to implement these designs in SharePoint using our recommended approaches. We want to define approaches that are consistent with Microsoft and industry good practice, and while we have been able to deal with most issues and have managed to get the CSOM APIs extended in some cases, we have encountered a dilemma with what appears to be conflicting guidance vs what the client-side APIs currently support around branding a site, or customising the UI.

Note that the solutions need to be fully automated for deployment as well as upgrades, and so changes/upgrades to UI must also be able to be applied through an automated approach without use of server-side code. Our remote deployment solution handles this, so we just need to determine the client-side methods to achieve the required branding/custom UI outcomes.

I will first explain the guidance that I am considering, then follow with why I believe these are incompatible…

Guidance #1: Use Themes

The first piece of guidance from Microsoft I’d like to consider is to use themes for minor UI customisations, where simply changing the colours of existing elements on the page will be sufficient. Without ‘customising’ any aspect of the site the colours can be configured simply by selecting some new theme colours, or pick/create a Composed Look with a new colour theme. Therefore this appears to be a good approach to use that does not require any development if this is sufficient to meet the requirements.

Guidance #2: Make Custom CSS Themable

More complex UI customisations usually require some custom CSS. No problem! Custom CSS files can be used in SharePoint, however they often include hard-coded colours. Given SharePoint has a great theming engine that provides a configurable colour palette it seems appropriate that those colour palette entries are used directly in the CSS rather than hard-coding colours in CSS, so colours can be updated without reworking the CSS. Microsoft guidance also seems to recommend this.

This approach seems particularly useful for light to medium UI customisations where a Theme does not quite cover all of the UI needs, and a little CSS may be needed – perhaps some additional elements to be shown on the page needs to have some theme colours applied to them.

For details on how to create a Themable CSS file and applying the Theme, see “How to: Make custom CSS files themable in SharePoint 2013“.

In simple terms, you mark-up the CSS with annotations in the form of specific CSS comments that indicate which colour palette entry to be used, with some other abilities such as recolouring images, changing fonts, and so on. SharePoint 2013 Themes also now provide the ability to define new colour entries in the palette and reference those in CSS – this was sorely missing from SharePoint 2010 Themes, and is a welcome addition in SharePoint 2013. This now also makes it possible to consider using Themes for more complex UI customisations where we can now define new colours that are not tied to colours already used for out of the box components which was a significant limitation I encountered in SP 2010.

To make SharePoint take notice of these theme annotations, i.e. to generate and use a themed version of the CSS automatically, involves having to register the CSS on the site. This is achieved by adding a element in the head of the Master Page pointing to the CSS file, ensuring the CSS file is placed in the Themable folder in the Style Library, and having all necessary theme annotations in the CSS. Then whenever another theme is selected, SharePoint will auto-generate a themed copy of the custom CSS with colours substituted, and the CssRegistration element will output the correct CSS link on each page.

This approach therefore requires modifying the Master Page, which brings us to the next piece of Microsoft guidance…

Guidance #3: Try To Avoid Modifying Master Pages

While through a quick search I cannot easily find any direct written guidance on this, I have heard this guidance direct from several Microsoft sources, and read the same advice on several blogs and forum posts – it makes sense. The following Microsoft Patterns & Practices articles provide patterns for adding CSS without modifying a master page, and the 2nd article confirms this guidance saying “Custom master pages should be avoided to ensure that any updates or enhancements added to the out of the box master pages are automatically in use of the sites”.

SharePoint upgrades usually come with updated master pages, and increasingly in SharePoint Online, smaller upgrades are being rolled out more regularly, which form time to time may include updates to the master pages. Any time we modify a master page to add our own components, or add links to scripts or CSS, we effectively lock ourselves into the current modified version of the out of the box master page so it will not pick up any further enhancements as SharePoint evolves. This could mean we do not benefit from some improvements, whole new features, or even in the worst case, for a significant upgrade, the master page having to be redeveloped just to keep it working in the new version of SharePoint (or maybe we remain locked to an ‘old’ UI as many will have found when upgrading from SP 2010 to SP 2013). So where possible it would be better to avoid changing the master page and use other ways to add any necessary customisations, such as injection of JavaScript or CSS as discussed in the following Patterns & Practices:

CSS INJECTION PATTERN – Uses JavaScript injection via a User Custom Action to add a CSS link without directly modifying the master page. A problem found with this approach is that the page loads completely before this JavaScript is executed which then loads the CSS, so the net effect of this is the page flickers as it first renders the page then updates the page afterwards with the new CSS. Using this approach for loading JavaScript may be more acceptable in many cases as the delay may not cause any noticeable visual problems if the JavaScript does not result in restructuring or recolouring the page. Dynamic loading ‘spinners’ can be used to indicate when a component is still loading without significantly impacting the overall user experience.

ALTERNATECSSURL & SITELOGOURL PROPERTIES IN WEB OBJECT – It is now possible to set an Alternate CSS URL via CSOM since April 2014 CU, so this now seems a better alternative to the CSS Injection Pattern where the CSS is loaded and applied during page load and applies to all pages in the site. While the UI only makes this setting available on Publishing Sites (or sites with Publishing Infrastructure features enabled), it seems via the CSOM API it is now possible to set Alternate CSS URL via CSOM on all types of sites.

The Dilemma – We Can’t Apply A Themed CSS File Without Modifying A Master Page

My problem with using these patterns is that they still do not apply the theme to the CSS. It seems only use of the CssRegistration element added to the master page will actually ensure that the CSS has the theme correctly applied to it. Both CSS Injection and AlternateCssUrl approaches will not apply a theme to the CSS.

So we appear to be stuck between conflicting guidance. On the one-hand we should use Themes more, especially for more simple UI customisations, and where necessary implement Themable CSS to keep the site looking consistent when the theme colours are changed. However on the other hand, we cannot actually apply a Themable CSS without modifying the master page to add a CssRegistration element, which we are also advised that we should try to avoid.

At present if we were to try to follow Microsoft guidance around both theming custom CSS and not modifying a master page, we will fail as this seems to be a gap in SharePoint. I will be encouraging Microsoft to fill this gap, but won’t expect this to be covered any time soon.

Given that we would not want to have to rectify master pages on hundreds of thousands of site collections, we would prefer not to have to add CssRegistration to each master page. So it seems the ‘lesser evil’ will have to be adopted, and that is to consider dropping the recommendation to make your Custom CSS Themable as it would not be able to pick up the theme colours. This does not sit nicely with me, but until we receive an update for this in SharePoint, there does not seem to be any other way we can achieve our aims.

Update: 24th July 2015

Microsoft has acknowledged this issue and has raised a Design Change Request but they now need to identify other clients who are also affected by this, and have requested this to be logged in ‘user voice’ so others can vote it up to raise the priority. Here is the link if you would like to help raise the profile of this issue: Issue logged in user voice – please vote.

SharePoint 2010: List View Threshold Explained

June 13, 2013 10 comments

In this post I try to explain the List View Threshold that was introduced in SharePoint 2010, the benefits that it provides, the operations that are prevented when the threshold is exceeded, and possible ways to get them working again.

What is the List View Threshold?

SharePoint 2010 has introduced a List View Threshold which helps to keep the SharePoint servers performing well by restricting the amount of list data that can be queried and displayed. This applies to lists and libraries including document libraries.

This configuration applies to each web application independently, i.e. Portal, Workplaces, Archive, and My Sites. The default limit is set at 5,000 items.

Once a query exceeds the configured limit, the query is stopped and usually a message is displayed indicating something to the effect that it is stopped because the List View Threshold is exceeded.

Typically users will see such queries in use by List Views and List View Web Parts, so when the List View Threshold takes effect, the List View or List View Web Part will display a message instead of the expected items. 

Why is the List View Threshold set at only 5,000 items?

The reason the List View Threshold is set by default to 5,000 items is because of the way that SQL Server behaves when querying large numbers of items without a suitable index to work with. When sorting and/or filtering a list SQL Server has to scan the sorted/filtered fields in all records in the database table, i.e. all items in the SharePoint list.

After approx. 5,000 items, SQL Server usually determines that it is more efficient to apply a table lock to resolve contention issues with other queries also trying to query the same items, enabling it to access all items quickly. The consequence of the table lock is to temporarily block other queries on the table until it is complete. This can impact the performance of SharePoint with lots of concurrent users as some queries must wait for others to complete. The threshold prevents this from happening. 

Wasn’t this a problem on SharePoint 2007?

The List View Threshold did not exist in SharePoint 2007, and so it did not restrict queries to list data. This meant that there was no limit to be exceeded so queries on lists with over 5,000 items could cause the table lock to occur and the SharePoint performance could be impacted.

A SharePoint 2007 environment was therefore likely to have been impacted with lists and document libraries containing more than 5000 items, but this was just the way it behaved and nothing prevented this from happening. Microsoft appeared to have recognised that this is an issue for some and introduced the List View Threshold in SharePoint 2010 so that this issue cannot arise without deliberately increasing or removing the threshold first. Removing the threshold would appear to make SharePoint 2010 perform more like SharePoint 2007 did when handling large lists.

What operations may be blocked because of the List View Threshold?

The blocked operations listed below are taken from the Designing Large Lists and Maximizing List Performance document published by Microsoft for SharePoint 2010.

Blocked operations when list exceeds the List View Threshold

Add/Remove/Update a list column All columns including lookup and calculated columns, in addition to many types of updates, such as a type change or a uniqueness change. Some updates, such as a name change, are not blocked because they do not affect every item in the list.
Add/Remove/Update a List Content Type Affects every item in the list so it is blocked for any list that has more items than the list view threshold.
Create/Remove an Index Affects every item in the list so it is blocked for any list that has more items than the list view threshold.
Manage files which have no checked in version A non-indexed recursive query that fails for any list that has more items than the list view threshold.
Non-indexed recursive queries Includes filters and some sorts. This operation fails when the list size is greater than the list view threshold. Because there is no index, it does a full scan against the entire list. Also it returns all items, and it ignores folders.
Cross list query Includes queries by the content query Web Part and follows the list view threshold setting for auditors and administrators, which by default is 20,000. If the operation involves more than 20,000 items, the query fails.
Lookup columns that enforce relationship behavior You cannot create lookup columns that enforce relationship behavior when the list it references contains more items than the list view threshold.
Deleting a list Affects every item in the list so it is blocked for any list that has more items than the list view threshold.
Deleting a site If the sum of all items in a site is greater than the list view threshold, deleting the site is prevented because it affects too many items.
Save List as Template with Data Affects every item in the list so it is blocked for any list that has more items than the list view threshold.
Showing Totals in List Views Performs a query against every item in the list so it is blocked for any list that has more items than the list view threshold.
Enable/disable attachments in a list Affects every item in the list so it is blocked for any list that has more items than the list view threshold.

Blocked operations when container exceeds the List View Threshold

Delete/Copy/Rename a folder Fails when the folder contains more items than the list view threshold because it affects too many rows.
Queries that filter on non-indexed columns Fails when the container (folder or list) contains more items than the list the view threshold because it does a full scan against the entire folder because there is no index.
Set fine grained security permissions Fails whenever the list or folder on which you are trying to set fine grained permissions contains more items than the list view threshold because it affects too many rows. You can still set fine-grained permissions on child items, such as documents, in a large list, although you cannot set the permissions on the list itself or on folders that contain more items than the list view threshold.
Open with Explorer Does not show any items if a container has more items than the list view threshold (excluding items in sub folders). If a folder has 8,000 items total, but it has a sub folder that contains 4,000 items and only 4,000 items in the root, then Open with Explorer will work. If the root of a list contains more items than the list view threshold then Open with Explorer will not show anything. To use Open with Explorer the list must have items organized into folders in amounts less than the list view threshold in the root of any container.

Available features that might not work as expected

Datasheet view

The datasheet view button that is available in the Library ribbon tab of a document library is not disabled if the list grows above the list view threshold. However, if the list size exceeds the list view threshold, the view loads some items, but it displays a message that says, “You do not have permission to view the entire list because it is larger than the list view threshold enforced by the administrator.” You can disable the datasheet view option from the ribbon in the settings for the list. There is also a hard limit of 50,000 items so this view will be blocked even if the list view threshold is above 50,000.

How can some of these problems be resolved?

The List View Threshold can simply be manually increased to a high enough number that it is effectively disabled for all lists in the web application, or programmatically it can be completely disabled or disabled only on individual lists. However this will result in the potential for queries to perform poorly, undoing all of the performance benefits that the threshold was designed to resolve.

The threshold only applies to queries against lists where the sort order or the filter uses a column that does not have an index. Therefore adding an index to such columns will not only potentially improve performance of those queries as they use an index rather than performing a full table scan in SQL Server, but also this means a large list can be queried without the threshold stopping it from completing.

We can therefore prevent web parts and list views from displaying the threshold error message by adding an index to the fields that are sorted or filtered in the view.

For instance, if you have a web part or list view which displays all items checked out to ‘me’, then adding an index to the “Checked Out To” field enables the web part or view to work properly without the threshold preventing it from completing.

Adding an index to the “Checked Out To” field according to this MSDN article will also resolve the same problem on the “Manage files which have no checked in version” page that is found via the document library settings page.

It is not clear what kind of index could be added to a list, or what other action could be taken to resolve the fine grained security permissions issue as none of the fields in the list appear to be relevant when managing permissions. Therefore to ensure fine grained security permissions will work in large lists, it appears only raising the List View Threshold will achieve this, along with the potential performance degradation that this is designed to prevent.

Note, if a list already exceeds the List View Threshold then this will prevent the ability to add an index to the list as this is one of the many operations that are blocked by the List View Threshold, so this can only be performed when the threshold is lifted.

Alternative to permanently or temporarily lifting the List View Threshold, SharePoint web applications can be configured with a “Daily Time Window for Large Queries”, which is often set to enable these kinds of operations out of hours when the system is under reduced load and is therefore less likely to experience poor performance resulting from these queries. This effectively removes the List View Threshold during the configured time window so that the operations which are usually blocked can be performed out of hours without significantly affecting other users.

The List View Threshold settings are available via SharePoint Central Administration. Click on Manage web applications, select the web application to configure, then in the ribbon under General Settings, click to drop-down the menu of options, and choose Resource Throttling. All settings on this page apply to the selected web application.

Test what effect removing the List View Threshold will have on your environment

Before deciding to effectively remove the List View Threshold during business hours on a live production environment, it would be beneficial to benchmark the performance with some likely scenarios to test whether the environment will still perform sufficiently well under load. Then an informed decision could be taken on the trade-off between keeping the List View Threshold and having to live with the limitations, using an out-of-hours Daily Time Window for Large Queries, or perhaps even raising the threshold sufficiently to not have any effect altogether. Even so, in real live situations, if performance problems do occur, it would be wise to keep this in mind with the option to bring back the threshold if necessary to keep the environment performing well.

Microsoft recommend that the threshold should not be lifted, but they provide the capability to do so if needed, and the “Daily Time Window for Large Queries” seems a sensible approach to meeting the needs half-way so that these often necessary operations can be executed, while protecting the performance of the system when under heavier load.

SharePoint: How to check the SharePoint Installation Type – Standalone or Farm

I have just been given a client’s development workstation virtual machine to deploy a full SharePoint solution to before developing some updates on it. As I needed to make extensive use of the User Profile Service and Managed Properties for search, I wanted to make sure I was working on a full farm install rather than a standalone installation to try to avoid any problems where the environments are not similar enough to the production environment.

I asked the person who setup the machine but it was so long ago when it was setup he had forgotten, although he thought it might be a full farm install.

It wasn’t long before he came back to me and told me how to identify what kind of installation was used. He found the solution in a discussion thread.

The solution is to launch the Registry Editor (regedit.exe) and check for the Server Role at the following location depending on the SharePoint version:

  • SharePoint 2007: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\12.0\WSS\ServerRole
  • SharePoint 2010: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\14.0\WSS\ServerRole
  • SharePoint 2013: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\15.0\WSS\ServerRole

The possible values are as follows:

  • SINGLESERVER – means this is a standalone / single server installation
  • APPLICATION – means this is an application server on a full farm installation (could also include the web front end role as well if multiple roles are used)
  • WFE – means this is a web front-end server on a full farm installation

SharePoint: How to check which Site Template was used to create a site just using a web browser

May 18, 2013 15 comments

I recently needed to check which site template was used to create a site on a production system so I did not have the option to use SharePoint Manager or write some code or PowerShell to open the site and check the site template.

A quick search revealed lots of methods to find this out but they all required that I actually log on to the SharePoint Servers. I needed to find out the site template just using the browser.

I came across a discussion thread where this question was asked and one respondent came up with the answer. You might think there is an application page in the _layouts virtual folder that will provide this information, but no.

The solution is simply to browse to any page on the site, view the source HTML of the page, then search for “SiteTemplateID” where you will be taken straight to a line of JavaScript embedded into the page such as the following where the site template ID including configuration is assigned to a JavaScript variable:

var g_wsaSiteTemplateId = ‘STS#1’;

I am sure I saw this when inspecting the HTML of pages several times in the past but I didn’t think about this when I needed it.

Nice and easy when you know how :-)

SharePoint Online: Adding Term Store Administrators

April 6, 2013 3 comments

When you first start using the Term Store Management Tool in SharePoint Online, it may not be immediately obvious why you cannot initially add/modify/delete the term sets or the terms within them.

When you access the Term Store Management Tool from the Site Settings page of a site collection, you will be able to see the term sets that are provided by default and cannot be changed as shown below:

Just like with SharePoint Server, you first need to configure the Term Store with one or more Term Store Administrators.

In SharePoint Server this is achieved through Central Administration, Manage Service Applications, and picking the Managed Metadata Service to administer. However in SharePoint Online, there is no Central Administration in the form that we may be familiar with in SharePoint Server, and we cannot manage the service applications as this is a shared environment. Instead SharePoint Online has a SharePoint admin center which provides the necessary administrative functions specific to our SharePoint Online / Office 365 subscription, which are normally provided through Central Administration.

The SharePoint admin center is accessed via the Admin menu and picking SharePoint in the menu.

Once in the SharePoint admin center, you can find “term store” in the left navigation pane. Clicking this will show the Term Store Management Tool again but this time, just like within Central Administration in SharePoint Server, you will have the ability to set the Term Store Administrators as shown below, after which you must scroll down and click Save for the changes to take effect:

Returning to the Term Store Management Tool will now allow the Term Store Administrators to add/modify/delete the term sets etc.

%d bloggers like this: