SharePoint 2010: Managed Metadata Converts Ampersand and Double Quotes to Unicode
We have been making a lot of use of Managed Metadata lately. One use is to associate people with a team, so we have created a user profile property associated with a Teams term set. We also created a CSV export for people search so it will export this field among many others for all people returned in the search, with the intention that this will be opened in Excel.
This was working fine until we tested a CSV export in Excel where the Team field contained an ampersand (&). It unexpectedly appeared in Excel as a collection of symbols. I had noticed previously that the ampersand looked a little unusual in SharePoint but thought nothing more of it until now.
It transpires that the ampersand (&) is automatically converted to a Unicode/wide character version (&) when it is added to the Term Store. This is represented using hex code FF06 or integer 65286. The only documentation I have found confirming this so far is in the TaxonomyItem.NormalizeName method which “Normalizes a name of a TaxonomyItem object”, or in other words, converts the given string to the format used in the Term Store.
The remarks section says:
The name will be normalized to trim consecutive spaces into one and replace the & character with the wide character version of the character (\uFF06).
However upon reflecting the method I found the following code:
return CommonValidator.trimSpacesRegex.Replace(name, " ").Replace('&', 0xff06).Replace('"', 0xff02);
This shows that not only does it modify the spaces and ampersand but it also replaces double-quotes (“) with a Unicode version too ("), using hex code FF02 or integer 65282. This was unexpected and to my knowledge undocumented.
Now that we are armed with this information, to fix the CSV export issues now simply involves replacing these Unicode characters back to the original ASCII characters that we used.
You might also use the TaxonomyItem.NormalizeName method if you want to compare a string with a term name taken from the term store, by using it to ‘normalize’ the string prior to comparing with the term.
Leave a Reply Cancel reply
Top Posts
Archive
- June 2019 (1)
- May 2019 (1)
- January 2019 (1)
- October 2018 (1)
- July 2018 (1)
- October 2017 (1)
- May 2017 (1)
- June 2015 (1)
- June 2013 (1)
- May 2013 (2)
- April 2013 (3)
- March 2013 (1)
- February 2013 (1)
- December 2012 (1)
- November 2012 (3)
- October 2012 (3)
- August 2012 (1)
- July 2012 (1)
- June 2012 (1)
- May 2012 (1)
- April 2012 (2)
- March 2012 (5)
- February 2012 (8)
- September 2011 (1)
- July 2011 (2)
- June 2011 (6)
- August 2010 (1)
Tag Cloud
Links

MCPD: SharePoint Developer 2010
Nick, thanks for this. It’s helped up track down an issue for a client our. Certain users who have certain browsers are seeing the unicode ampersand as a small box when the term is displayed. So thanks for the cogent analysis for this.
Are you aware if this is considered by Microsoft to be a bug? I’m certain there was a good reason for doing it in the first place, but it seems to have unintended consequences.
Hi Ken
Glad it helped. I didn’t check with Microsoft on this one since the ampersand was somewhat documented, albeit hard to find!
Nick
Thank you for your help Nick, this is the exact problem we’ve been having recently. Forgive my ignorance (new to SharePoint) but where do I go/how do I edit the code above or change the TaxonomyItem.NormalizeName so that the ampersand isn’t replaced?
Thank you!
Hi Brendan
Unfortunately you cannot change this behaviour. You can only work around it by being aware of the problem and updating your code to recognise the converted characters or to convert back when reading them.
Nick
Hi Nick, Thanks a ton for this article. I struggled a lot for finding the fix. Microsoft did a stupid thing converting into a different ampersand.
Hi Nick – How can I get around this ? I have a no – code solution so using your workaround above will not work (or have I missed something)
Thanks
Nigel
Apologies Nigel – probably too late now – I didn’t get the notification when you posted your message.
As a no-code solution, I’m not sure what you are attempting to do. However if you are looking for a CSV extract, then I’d suggest getting the extract, then adding a formula in a new column to transform the data by replacing the 2 characters in Excel, then moving the new values back to the original column to save as the CSV.