10. Unique Document ID’s: Every doc has a unique identifier and unique link.  I can send you a link to a specific document and the link will work no matter where the real document lives.
9. SharePoint Workspaces: Documents …heck, whole SharePoint sites, can be taken offline using the SharePoint Workspace.
8. Document Storage: Storage capacities have been massively increased and documents can optionally be pushed outside SQL Server, such as to the file system.
7. Easily Find Documents: Transform the act of “search” into “finding” using tools such as metadata filter web parts and enterprise search to locate the documents you need.
6. Edit Documents in Real-time: Edit docs right in the browser with Office Web Apps.
5. Enterprise Document Management: Manage all the core aspects of your documents including check in/check, versioning, permissions, history and workflow.
4. Content  Organizer: A user can submit documents to ‘SharePoint’ and the Content Organizer will make sure it goes to the right place, even if that place resides in another site collection!!
3. Faster: User interface is much, much faster with no browser refreshes or long waits.
2. Taxonomy and Metadata: Metadata fields (like Customer Name) can be used across SharePoint, not just within a single Site Collection.  A centralized taxonomy allows terms to be managed across the organization.
1. Enterprise Records Management: Records can live anywhere–a document can be converted into a ‘protected’ record yet still remain in the document library it was created.  Or, move it to a Record Center but leave a link behind.  From storage to retention and disposition, information management policies rigorously address all records management requirements.

I’m Josh Stickler, the Program Manager responsible for Variations. In this post, I will explain:

• Where to find the View Changes button
• When the View Changes button is available
• Our envisioned content editing process using View Changes

The most common application of the Variations feature is in multi-language sites.  Let's look at View Changes from the perspective of Anders, an English-to-Danish translator working with the Danish subsidiary of AdventureWorks, an international camping goods retailer.

AdventureWorks is set up with an English (EN-US) site as its source label and target labels for international markets, each corresponding to a different language.  Pages from the source label automatically propagate to the target labels when they are published so AdventureWorks' global web presence is in sync.  Translators at each of the targets then process the English-language content for localized consumption. AdventureWorks’ Variations hierarchy looks like this:

Initial Localization

Automatic Creation

Let's imagine that content authors at AdventureWorks in the United States have just published a new page with a sneak peek of this winter's new product lineup. Since “Automatic Creation” is enabled (this is the case by default), the page is picked up by the Variations Propagate Pages timer job and copied to all target labels, including Danish (DA-DK).

E-mail Notification

As the designated owner of the new page, Anders gets an e-mail informing him that this page has been copied to the target label by the Variations feature and is ready for processing. 

Target Translation and Publication

Anders navigates to the page on the Danish (DA-DK) variation of the AdventureWorks website and sees the English language content.  Since it's all new, he translates all of this content into Danish and submits the page for approval.  The page is approved and published and now appears on the Danish variation of the website.

Since Anders received an entirely new page to translate, there were no changes to view; hence, the View Changes button is not available.

Subsequent Modification

Source Modification and Publication

Back in the United States, AdventureWorks decides to announce a new product in its sneak peek lineup.  English language content authors add a paragraph describing this new product, an ultra light sleeping bag, and publish the page. The page now propagates to the Danish variation.

E-mail Notification

Anders receives an e-mail notification that new content is ready for processing.  He visits the appropriate page on the Danish variation site and the English content appears and is waiting for translation.

But wait, there is a lot of English content here, and Anders has already translated most of it.  Only one paragraph has been added.  How will Anders know that he doesn’t need to re-translate the whole page?

It’s at this point that the View Changes button comes to the rescue and is available. 

Please note that View Changes requires the Variations Propagate Pages timer job to be enabled. View Changes only compares changes between a source version of a page and a target version that has been copied using the Variations Propagate Pages operation.

Version Comparison

Anders clicks the button and a version differential window pops up, highlighting the new paragraph that has been added.  Now, Anders knows that only this paragraph has been added and doesn't have to scan through the new and old versions of the English content to determine what he has to translate. 

Restore Previous Version

Anders decides he prefers to revert back to the translated Danish version of the page as a basis for adding the new paragraph.  With the View Changes window open, Anders knows exactly which paragraph to translate and where it goes.  He adds the new content in Danish, submits for approval, and it's published live on AdventureWorks’ Danish variation site.  Fantastisk.

In addition to providing target variation site translators with insight into what content has changed when pages are copied from the source, SharePoint 2010 also enables authors on the source to decide when to propagate content to targets. By default in MOSS 2007, when content authors published pages in the source variation site, that page would automatically propagate to all target variation sites, even for small changes that are relevant only to the source variation site.

SharePoint 2010 provides the ability to disable automatic page propagation; source variation site content authors can then use the Update Variations button to propagate content on demand. See my previous post, “Site and Page Propagation” for more information on how to enable this setting.

Thanks for reading! Keep checking back for new blog posts.

Regards
Josh Stickler
Program Manager

This blog post delves deeper into the various metrics available to analyze the site usage data. There are three categories of the SharePoint Web Analytics reports: Traffic, Search, and Inventory. The reports are aggregated for various SharePoint entities like Site, Site Collection, and Web Application for each farm. Further, reports are also aggregated per search service application. By default, the reports show the data for a period of 30 days. One can change the time period to view data for up to 25 months by going to ‘Analyze’ tab.

Visually we show the metrics in one of the two ways: trend reports and rank reports. A trend report shows how a particular metric is doing over a period of time. While a rank report, shows the top 2000 results for a particular metric. Figure 1, 2 show examples of a trend and rank report respectively. That’s not all; you can further analyze the reports by applying filters like string match in the URL, user name, queries, browser and others.

Figure 1:  Example of a Trend Report showing Number of Page Views for each day for a default period of 30 days.

Figure 2:  Example of a Rank Report showing the Top Pages sorted on the Number of Page Views for a default period of 30 days.

What follows is an overview of each type of the report and the associated metrics. Also, summarized are the kind of reports available for each level of aggregation i.e. Site, Site Collection and Web Application and Search Service Application.

Traffic Reports

The traffic reports capture the user behavior information related to total clicks, frequent users, popular pages, and information about navigation to and from the current SharePoint component.

Trend Reports

1. Number of Page Views: Total number of page views each day.
2. Number of Daily Unique Visitors: Total number of unique visitors each day. SharePoint authenticated users and anonymous users (using IP address) are counted as visitors.
3. Number of Referrers: Total number of unique URL’s external to the current entity (parent entity is treated as external as well), from where the users navigated to the current entity.

Rank Reports

1. Top Pages: Most viewed pages in the current entity.
2. Top Visitors: Most frequent visitors of the current entity.
3. Top Referrers: Top URL’s external to the current entity from where users navigated to the current entity.
4. Top Destinations: Similar to Referrers, these are the top external URL’s that the user visited from the current entity.
5. Top Browsers: Top browsers being used to visit the current entity.

Report Scope	Site	Site Collection	Web Application
Number of Page Views
Number of Unique Visitors
Number of Referrers
Top Pages
Top Visitors
Top Referrers
Top Destinations
Top Browsers

Table 1:  Summary of the traffic reports availability at different SharePoint hierarchy levels

Note: Traffic Reports do not apply at Search Service Application level.

Search Reports

The search reports capture the user behavior information related to the queries on the site.

Trend Reports

1. Number of Queries: Total number of queries each day.

Rank Reports

1. Top Queries: Most issued queries per day.
2. Failed Queries: Most issued queries for which either there were no results or the user did not click on any results.
3. No Result Queries: Most issued queries for which no results were returned.

Other Reports

1. Best Bet Suggestion Report: Recommends URLs as most likely results for particular search queries based on analysis of usage patterns. The site administrators can accept or reject these suggestions. If they accept, the corresponding query-URL pair is added to the search keywords list.
2. Best Bet Usage: Shows how Best Bet suggestions are doing over time. For every Best Bet query issued, the report shows the percentage of clicks on the Best Bet URL compared to other URLs.
3. Best Bet Action History Report: Tracks the actions performed by the site administrator on the ‘Best Bet Suggestion’ Report.

Report Scope	Site Collection	Web Application	Search Service Application
Number of Queries
Top Queries
Failed Queries
No Result Queries
Best Bet Usage
Best Bet Suggestions
Best Bet Suggestion Action History

Table 2:  Summary of the search reports availability at different SharePoint component hierarchy levels

Note: The search reports do not apply at Site Level.

Inventory Reports

The inventory reports are targeted to help the site administrators in managing the site by keeping track of the site structure and storage and version issues.

Trend Reports

1. Number of Site Collections: Total number of site collections for each Web Service Application for each day.
2. Storage Usage: Total storage used in Megabyte (MB) for a site collection and the ‘Maximum Storage Allowed’ in MB for each day.
3. Number of Sites: Total number of sites within each Site Collection for each day.

Rank Reports

1. Top Site Product Versions: The ‘Site Product Version’ sorted in the order of ‘Number of Sites’ or ‘Percentage of Overall’ sites using the corresponding version for this site collection.
2. Top Site Languages: The ‘Site Product Language’ sorted in the order of ‘Number of Sites’ or ‘Percentage of Overall’ sites using that language for this site collection.

Report Scope	Site	Site Collection	Web Application
Number of Site Collections
Storage Usage
Number of Sites
Top Site Product Versions
Top Site Languages

Table 3:  Summary of Inventory Reports availability at different SharePoint component hierarchy levels

Note: Traffic Reports do not apply at Search Service Application level.

Look out for more to come

Keep an eye out for more blogs on customizing the reports using Excel, using workflow feature to scheduled reports and alerts and adding the ‘What’s Popular’ Web Part to your pages.

CMIS (Content Management Interoperability Services is a specification that Microsoft developed in along with IBM, EMC, Alfresco, OpenText, SAP and Oracle to enable greater interoperability between content management repositories and to enable a whole new range of Composite Content Applications that can be build agnostic of the underlying repository.

We see CMIS as a great solution to help our customers more effectively leverage content maintained in a heterogeneous environment but more importantly, we see the specification as a way to enable a whole new range of Composite Content Applications that can be build agnostic of the underlying repository.

For further reading on CMIS, visit these sites:

• Original announcement of the CMIS specification and Microsoft’s involvement
• CMIS reaches the public review stage in the OASIS process
• OASIS CMIS Technical Committee

Public voting on the CMIS specification ends on April 30th and we expect that the specification will be ratified as a standard shortly afterwards.  We are excited that our work on the specification alongside the other leading ECM vendors is coming to fruition and are looking forward to providing support for the standard in SharePoint 2010.

Ryan Duguid
Senior Product Manager
Microsoft Corporation

We’re excited about the upcoming release of Office 2010 and SharePoint 2010 and are looking forward to meeting with you in Philadelphia.  Before we head out East, I’d like to introduce you to our speakers:

This is the largest gathering of Microsoft speakers since our SharePoint Conference in Las Vegas last year and we’re looking forward to meeting you in person next week.  We hope you can attend the SharePoint 2010 Summit @ AIIM Expo or join us on the Expo Hall floor.  Be sure to bring your burning SharePoint questions and make the most of this opportunity to talk with the experts.

Ryan Duguid
Senior Product Manager
Microsoft Corporation

If there are additional areas that are of particular interest to you, please post in the comments section and I will try to address as many as I can. I’d really appreciate getting any and all feedback. Thanks!

What is the Variations feature?

Variations is a SharePoint feature that facilitates the management and maintenance of content that can be served to multiple audiences. These audiences can vary in terms of different languages, countries, or regions, but they can also represent different brands or devices.

How does Variations work?

For each channel you wish to serve content, you can specify a Variations label. Labels are instantiated as SharePoint publishing sites and the full set of labels in a site collection is referred to as the Variations Hierarchy. I refer to SharePoint publishing sites created and managed by the Variations feature as “variation sites.”

Using variations, target variation sites reflect one source variation site in terms of pages and site structure. When setting up variations, specify one variation site as the source; all other variation sites are targets. By default, pages published on the source variation site are copied to all target variation sites as draft versions and sites created on the source are created (not copied – this is an important distinction) on all target variation sites. You can only have one source variation site per Variation Hierarchy and you can only have one Variation Hierarchy per site collection.

What’s new in SharePoint 2010?

The concept and core architecture of Variations, in which pages and site structure are replicated across multiple variation sites in a site collection remains the same as in Microsoft Office SharePoint Server 2007; however, we have made significant improvements to better meet the needs of enterprise customers serving content across multiple channels.

These improvements can be divided into four categories:

• Server Citizenship
• Content Distribution
• Editing Experience
• Reliability

Server Citizenship

Variations operations now execute in the background via timer jobs. For the end user, this means that you no longer have to wait at a progress screen for operations to complete.  For the system administrator, this means that the cost of resource-intensive operations like Create Hierarchies can be better managed.

You can adjust the frequency with which Variations operations run in Central Administration. Next, I’ll explain the difference between the “Create” and “Propagate” timer jobs in the context of improvements we’ve made to the Variations content distribution models.

Site and Page Propagation

MOSS 2007 featured two models for distributing pages across your Variations Hierarchy:

1. Automatic Creation: If “Automatic Creation” is enabled on the Variation settings page (it is enabled by default), then publishing a page on the source variation site will cause that page to be copied to all target variation sites.

2. Manual Creation: If “Automatic Creation” is disabled, then the “Create Variations” Ribbon button is the only way to copy a new page to a specific, individual target variation site.

We’ve received feedback that there are often cases in which changes need to be published locally to the source variation site without being propagated to all targets. For instance, if the source variation site has a typo in English, the correction may not be relevant to a target site in German, so if the correction is published in the source page, it can be unnecessarily confusing to copy this changed English version to all target sites.

In SharePoint 2010, we introduce a third, “hybrid” content distribution model:

3. On-Demand Page Propagation

A setting has been added (configurable through the Object Model) to disable Automatic Page Propagation. When the setting is enabled, publishing or approving a page on the source variation site will not cause that page to be copied to any target variation sites. The "Automatic Creation" setting will be ignored for pages. "Update Variation" and "Create Variation” are the means by which a user can distribute content across the Variation hierarchy on-demand.

I’ll go into more detail on content distribution models in a future post. But so as not to keep you in suspense on how to configure on-demand page propagation, here are the PowerShell commands:

Enable On-Demand Page Propagation:

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
$site = new-object Microsoft.SharePoint.SPSite("http://yourserver/sites/abc")
$folder = $site.RootWeb.Lists["Relationships List"].RootFolder
$folder.Properties.Add("DisableAutomaticPropagation", "True")
$folder.Update();

Disable On-Demand Page Propagation:

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
$site = new-object Microsoft.SharePoint.SPSite("http://yourserver/sites/abc")
$folder = $site.RootWeb.Lists["Relationships List"].RootFolder
$folder.Properties.Remove("DisableAutomaticPropagation")
$folder.Update();

We’ve also made improvements for target variation site content owners to better understand what has changed on the source variation site when new draft versions appear on a target variation site.

Editing Experience

To make efficient use of their time and effort, target variation content editors need an easy and informative way to determine what content is new when pages are propagated from the source variation.

A new “View Changes” button compares the most recent source version propagated to the target with the most recent source version published on the target.  Changes are highlighted in a pop-up report to enable content processing directly in the rich-text editor.

Highlighted report

Corresponding location in the Rich Text Editor

This button is available on a target variation page after it has been published once and a new draft version has been copied from the source variation site via one of the Variations timer jobs. I will go into more detail on this new feature in an upcoming blog post dedicated to explaining View Changes with screenshots, a sample workflow, and an example scenario.

Reliability

One of our main goals for Variations in SharePoint 2010 is to make the feature more reliable so enterprise customers can entrust management and distribution of content across multiple channels to Variations.

Now that Create Hierarchies runs in the timer service, we support pausing and resuming this operation during timer service recycles to support long-running operations in large deployments. This also means that the process is not affected by Application Pool recycles. We’ve also made the relationships list, which tracks all target pages linked to a source page, more robust. We now track variations pages using GUIDs for better performance and scale.

Thanks for reading. Check back soon for upcoming blog posts on what’s new in Variations and other exciting developments in Enterprise Content Management.

Regards,

Josh Stickler

Program Manager

Hi, my name is Kevin Reynolds and I’m a Program Manager on the SharePoint team. Today I will walk you through the process for creating the page above, from creating the page to having it go live on the internet. I will show you the enhanced Web Authoring experience in SharePoint 2010, including editing content, applying styles, using the new UI, changing the layout of the page, and even applying themes to your site. I would highly encourage you to create your own Publishing site and follow along to get a personal feel for the SharePoint 2010 Authoring Experience.

Create New Page

Let’s begin with creating a new page. To create a new page click on the Site Actions menu and choose the New Page option, now in the dialog that comes up type in a name for the page – for this example we will use SharePoint 2010 Communities - feel free to insert your own name. Here is what you should currently see:

Now click Create on the dialog. A new page is created and you can see the Ribbon ()at the top of the page that exposes the most common options that you will use while editing the page. The page should look a lot like this:

Add and Edit Rich Text

Now let’s add some content into the Page and then we’ll get back to exploring more options available in the Ribbon. For this example I’ll add in the following text:

SharePoint 2010 Communities: Work Together Effectively
?As part of the 2010 release, SharePoint Communities provides a comprehensive, flexible platform that empowers people to work together in ways that are most effective for them. Allow your people to collaborate in groups, share knowledge and ideas, connect with colleagues, and find information and experts naturally.
Work Together the Way You Want
?The global workforce of the twenty-first century is more diverse than ever. Connect and engage all of your employees with a flexible collaboration platform and a diverse set of tools that range from Wikis to Workflows to Workspaces—allowing people to work together the way they want.
Rely on a Secure Collaboration Platform
?Let your IT staff rely on an enterprise-ready collaboration platform that is secure and easy to manage and will support your organization’s growing needs. SharePoint 2010 makes social safe with granular security and privacy controls, centralized management and policy setting, and robust reporting and analysis.
Extend the Value of Your Community Solutions
The SharePoint platform seamlessly integrates with the rest of the Microsoft Business Productivity infrastructure, including the Office applications, Exchange Server, Office Communications Server, SQL Server, and Dynamics. In addition, SharePoint provides Business Connectivity Services and adheres to open standards and protocols, making it easy to integrate third-party applications.

Feel free to copy and paste that text into your page as you follow along. We will use the text above to demonstrate the functionality of the editor on the page. Select the first line of text SharePoint 2010 Communities: Work Together Effectively, click on the font color drop down (), put your mouse over the red color, notice how the selected text turns red, now select the orange color, and notice how the text turns orange. We will change the font size now, choose the font size drop down (), choose 18 from the list and notice that the selected text now becomes larger. Choose the text work together the Way You Want, click the Markup Styles menu (), select Heading 1, and now do that for Rely on a Secure Collaboration Platform and Extend the value of Your Community Solutions. Now for those users savvy in HTML if you look at the markup of the page you will notice that the text is wrapped in a <H1> header tags, so you’ve applied a style and have well formed markup. If that last sentence doesn’t mean much to you, no worries, you can just use the menu as a set of styles on your text and leave the HTML markup thoughts to the experts. Now take a moment to play around with the text yourself, apply some fonts, apply some colors, highlights, font size, or adjust your paragraph alignment. No rush, I’ll wait…Really, it’s ok you can come back and continue the blog in a couple of minutes…Welcome back, here is roughly what the current page will look like depending on the formatting you’ve tried out:

Change the Page Layout

We will change the layout of the page, this will allow us to use a standard template that helps us to layout content in a consistent way across the site. Now to change the layout go to the Page () tab, select Page Layout (), and now you can choose a new layout for your page. For this demo I’ll be using a custom page layout – In a later blog I will show you how to create your own page layouts. Click on the Image on right layout and noticed all the new fields that show up and how the page is laid out differently now.

Insert a Picture

The new layout that we have chosen has a Page Image control that allows us to insert a picture onto the page in a specific location. To insert a picture click on the Click here to insert a picture from SharePoint text, then on the dialog that comes up click the first Browse… button, this launches the new Asset Picker, that allows you to choose an image that is already stored on SharePoint, if you haven’t uploaded any pictures don’t worry there are a few that come in the box, you can go to Site Collection Images, choose the Home picture, click OK, and click OK on the next dialog. You’ve inserted your first picture into a page in SharePoint 2010. The page should look like the following:

Insert a Video

In SharePoint 2010 we have enhanced the richness of the media that is natively integrated into pages and now everyone can easily add video and audio files to their page. To add a video put your selection below the text on the page, click the Insert Ribbon tab (), click the Video and Audio button (), now click on the Media Web Part () that is inserted into the page, and you will see a new contextual tab come up that contains commands specific to the Media Web Part:

Click the bottom part of the Change Media button to drop down a menu and choose from computer, this will bring up a new dialog where you can upload a video, click the Browse… button, choose a video on your computer, click OK, change the Upload to: box to be Images, and click OK, and Save on the dialog that comes up after the video is uploaded. You have now inserted your first video in SharePoint 2010, take a moment to use the player, watch the video, and play with the features. The video player will be covered in-depth in an upcoming blog post.

Theme the Site

We have also enhanced the theming capabilities in SharePoint 2010 to make it easy to apply a new set of colors to your site. This will give your site an updated look and feel which can easily be created and updated as your needs change. To update the theme of the site go to Site Actions and choose Site Settings, this will bring up a new page with a bunch of links, click on the Site Theme link, and now you will be in the new theming UI for SharePoint 2010.

We will cover theming of the site and this entire UI in a later blog post, for now let’s update the theme that goes with our content, in the large box with a list of theme choose the Ricasso theme, and click OK at the bottom of the page. Now navigate back to your page and you’ll see that the colors of your page have been updated, according to the new theme that you had chosen:

Apply a New Master Page

The page is really coming together, now we will see how easy it is to change our Master Page. The Master Page is the main component with theming that gives a site it’s look and feel. A master page defines where the company logo goes (or if there is one), where the Ribbon shows up, where the search box is, and all the common elements that should apply to every page. To update the master page go to Site Actions, choose Site Settings, and then on the Site Settings page click on the Master page link.

In the section labeled Site Master Page, click on the drop down box that currently says nightandday.master and change it to v4.master. This tells SharePoint that for this site you want all pages that you author to us the v4.master master page. Now navigate back to the page that you’ve been creating:

Now that you have all the right content and the page looks good, it’s time to get it live.

Go Live

To make the page available to others you will submit it for approval using the Ribbon. This will send the page off to the appropriate approvers for these pages and they will review it and then publish it to customers. To submit this page for approval go to the Publish tab and click the Submit button (), this will bring up a new dialog that will check the spelling and allow you to add comments for the reviewer of the page , add in a comment and click Continue, this will start the approval workflow for the page, a new form will come up, click Start, and now you will be taken back to the page. Now the approver will review the page and Publish it to go live.

You have now created your first page in SharePoint 2010 and you already know how to add pictures, insert videos, change the layout of the page, update the site theme, change the master page, and publish the page to go live. We will go deeper into each of these topics in future blog posts.

Thank you for reading and for following along,

Kevin Reynolds

Program Manager

Enterprise Content Management

When I throw out numbers like that, I’m not talking about just big, static libraries with content that just sits there. We want you to do crazy things with SharePoint 2010 like stuff a million document sets in a single document library with workflows running every which way, a hundred different retention policies firing off actions when you least expect them, and users uploading, tagging, and searching day in and day out. All the goodness of the SharePoint platform will be available to you whether you’re building a team site, a collaborative repository, a knowledge base, or a super large archive.

Like a plump, juicy sausage, much of the good stuff in SharePoint 2010 to give it delicious scalability are things that most people don’t need (or want) to know about. For the most part, scale just works. However, the chef (or information architect) is still a super important player. A well-planned repository is one that will have your users coming back for seconds and writing rave reviews; a poorly-planned one is one that will have them chugging Pepto-Bismol the next morning. Just because you can stuff a bunch of documents in a SharePoint 2010 library without your server igniting in flames the next day at doesn’t mean that you should without first thinking through how to best use the tools available to deliver an excellent experience to your end users.

So, even though scale in SharePoint 2010 just works, you’re not going to install the bits on day 1 and have a massive, searchable, beautiful content storefront on day 2. Guidance still matters, and believe me, we know it; this blog entry is just the beginning of the content we’re planning on delivering to help you on this front. I wouldn’t even call this blog entry guidance; it’s just a primer on the features and capabilities of SharePoint 2010 that you will grow to love if you’re passionate about scale at the library level – if you want to shove a whole bunch of documents in one place and have it be a great experience for both IT and your end users.

So what are these features and capabilities? Here are a few of the most important ones that I’m going to blog about now and in the near future:

1. We protect your database backend from dangerous queries. If you run a query against any database that requires it to scan through millions of items to find the ones you’re asking for, you’re going to balk at how long it ties up the server’s CPU. Quite frankly, SharePoint is not an exception. Even in SharePoint 2010, there is a class of user operations in certain scenarios that make unreasonable demands on the backend. For these operations, our strategy is to nip them in the bud before they’re executed, which keeps your high-demand servers healthy and responsive. Knowing when this throttling will kick in and planning for it is an important part of large list planning.
2. We give end-users tools to find content. When you have a sea of documents, the specific one you’re looking for can seem like a needle in a haystack. Structured metadata, easy tagging, metadata navigation, and built-in search refinement make this a less daunting task in SharePoint 2010 out of the box. This is an area we are particularly passionate about; after all, what good is a hugely scalable library if your end users hate it and can’t find what they’re looking for?
3. We help developers write excellent code. In SharePoint 2007, we didn’t give developers the right tools to write code that scaled well as the amount of content in your site grew. Even worse, it was pretty hard to tell why and when code was bad, and if your site was running slowly, which one out of your ten custom web parts was bringing things down. You had to “build around” SharePoint and do things “just so” to avoid this from happening. In SharePoint 2010, we give you a bunch of tools to make this story better.

Dangerous Queries

One challenge we’ve consistently seen customers run into when building large repositories on SharePoint 2007 is trouble with large containers. As the number of documents in any single container grows – either at the root of a library, or in a folder – bad things start to happen. For one, as your document to container ratio increases, it becomes harder and harder to find exactly what you’re looking for. More serious are the performance implications of large containers. Any of the out of the box ways of retrieving content from containers in SharePoint 2007 – like the All Documents view, the Explorer view, or a Content Query web part – would work, but they don’t scale very well. Loading All Documents in a library with a million items at the root would take a long time to finish. The big problem here is that you wouldn’t be the only one affected; all your friends running SharePoint sites on that same database server would experience things slowing to a crawl as well, as the database server dutifully iterated over those million documents to find the right ones.

Why does this happen? Any time you ask for content from SharePoint, you have to specify how it’s sorted – for example, the All Documents view in SharePoint 2007 asks for the top 100 results, sorted by filename. But items aren’t sorted by filename in the SharePoint content database – so, to bring you this view, SharePoint has to gather up all these million items, sort them, and finally display the 100 ones at the top of the sorted list. Imagine this as being like flipping through the residential section of a phonebook to find the first 100 addresses, sorted in alphabetical order. This would be a miserable task, because the telephone book isn’t sorted in this way – so in order to ensure your sorted list was accurate in the end, you’d have to look through the entire residential section, from start to finish, because after all, the last person listed in the phone book might live at 1000 Aardvark Lane.

Large Lists and Fallback Queries

The laws of physics are the same in SharePoint 2010 as they were in SharePoint 2007; if you run a query that needs to touch a very large number of items, you’re going to have to wait a long time, and so will everybody else. One prominent thing we did in SharePoint 2010 is to nip these queries in the bud before they get executed. To make a long story short (you can read the long story here), a farm administrator can set a threshold which defines the maximum number of items a single SharePoint query can touch. By default this threshold is 5,000. Any library with more items than this threshold is a large list.

Let’s go back to our example of the library with one million items at the root. Say you had that library in SharePoint 2007, and you upgraded to SharePoint Server 2010. First thing you’ll see upon navigating to this library will look something like this:

See the yellow bar above the list view? That’s a sign you have the Metadata Navigation and Filtering site feature turned on and it’s causing something magical to happen! When you load this view, SharePoint 2010 knows that you’re being greedy and asking it to scan through those million items. Since this query exceeds the maximum number of items a single query is allowed to scan (5,000) it doesn’t run the query. But who wants to stare at an empty list view? Instead of running this query as-is, SharePoint finagles it a bit and transforms it into a query that’s almost as good as the one you were asking for, but won’t make the database buckle under the pressure. In this case, we assume that it’s fairly likely that the document you’re looking for is one of the most recently created items in the library – so instead of scanning all one million items, we only scan the top 1,000 or so recently created documents, sort those by filename, and show them to you in the list view. This is what we call a simple fallback query: a query that doesn’t specify an index and asks for too many items in return, so instead of considering the entire list as being eligible for the query, SharePoint considers only the thousand or so most recently added items.

“Wait a second. You’re telling me that SharePoint throttles queries without asking me first? How on earth am I supposed to find anything in this crazy world of fallback queries and partial results?”

Let me assure you; this throttling business is a good thing. It’s a core ingredient in what makes SharePoint 2010 a resource for addressing your scale challenges. Gone are the sleepless nights where you toss and turn and worry about page faults on your database cluster resulting from Mack in Accounting stuffing 6,000 beer pong tournament photos in the root of a library in a forgotten team site in the dusty corners of your SharePoint deployment. The SharePoint 2010 feature set replaces this overarching concern with a set of well-scoped challenges; instead of worrying about every library that might get big, you get to plan for and craft experiences for the set of libraries that need to get big for business reasons.

I should mention really quickly that throttling is about more than just list views. There is a whole class of operations that involve iterating through all the documents in a list, or all the documents in a folder, that will get throttled (in other words, they will not execute) when the list or container is large. These operations include things like:

• Adding a column to a library
• Creating an index on a library
• Deleting a large folder in a library

Metadata Navigation – finding and working on content in large lists

Above is another screenshot from my million item library. This time, we’ve put a couple of SharePoint 2010 features to work. See that I have “demonstration scripts” selected in the left hand side in the tree view, and my list view is rendering without the yellow bar that’s telling me I’m only seeing newest results. That hierarchy of tags you see there represents a taxonomy, Item Type. I am browsing the documents in this library according to their Item Type; in the screenshot, I am filtering to show all documents with the value “demonstration scripts”. Here are the steps that I took to make this happen:

1. I created a taxonomy that describes my content. You can look forward to some posts from our very own Dan Kogan on this very topic in this very blog in the near future. There’s a lot to learn here. Not just any taxonomy will do here; it needs to be one that broadly divides my content up into evenly-sized buckets. For example, if I had 990,000 demonstration scripts, the query you see above would not get me anywhere. In that case, it wouldn’t make much sense to use Item Type as a piece of metadata and a navigation hierarchy for this library; I would need to find another, more divisive way to pivot the data.
2. I bound that taxonomy to a field in my library called Item Type. Think of a taxonomy field as a choice field on steroids. Instead of picking values out of a flat list, you pick them out of a tree.
3. I configured that field as a navigation hierarchy. Every library now has a Metadata Navigation and Filtering settings page where you can configure navigation hierarchies (the filters you see arranged in the tree view) and key filters (the additional filters that show up beneath the tree view)

In these three easy steps, I made “Item Type” a first class navigational pivot over the data. Instead of just staring at a partial list of content at the root, I can now browse with impunity by this virtual folder structure.

Here’s a couple of cool aspects of this feature that aren’t apparent from a single nifty screenshot:

1. Metadata navigation lets you slice and dice multiple ways. I might have a bunch of taxonomies on my library that classify content in different ways; for example, I might have a Products field, a Region field and a Competitors field, all bound to domain-specific taxonomies that classify the content along those dimensions. Depending on my current task, it might make more sense to filter by the Region field (for example, if I’m looking for the latest sales figures for the North America region). I get more filters than just my virtual folder; I can combine this filter with any number of key filters or list view column filters to drill down to just the content I want (for example, I want to see all demonstration scripts by the ECM team created after 2007).
2. Metadata navigation thinks about indices and large lists so you don’t have to. Hey, remember just a few minutes ago when we were talking about large lists, indices, and being throttled? Well, metadata navigation thinks a lot about indices and how to run queries the “right way” to make them perform well and prevent throttling from happening. For starters, all the fields you configure as navigation hierarchies and key filters get indexed, and the resulting queries are written in a way that ensures the best index is used to make the query succeed.

You aren’t immune from the laws of physics; if you ask for documents tagged with demonstration scripts and there are 10,000 demonstration scripts, we’re not going to be able to show you all of them. In this case, though, you get something better than a simple fallback; you get an indexed fallback, which means that instead of considering the entire list, the query considers only the items that match the indexed portion of your query.

Wrap-up

This article was just the first in my series of posts about architecting and building large lists filled with discoverable content. Here’s what you can expect over the next few weeks:

• A deep dive on metadata navigation, how it works, and some tips to getting the most out of it
• A discussion on how other features, like Search and the Content Query Web Part, fit into the equation, and how to configure their metadata filtering capabilities
• Some geeky developer tips on writing code that plays nicely with large lists

After that, I’ll be widening my scope a bit to talk about the overall knowledge management story in SharePoint 2010 – which is about more than just browsing for content in a library!

Lincoln DeMaris, Program Manager, ECM

This blog post is the first of several that will give you more insight into the enhanced Web Analytics features that we have built into SharePoint 2010. This first post will provide an overview of the new Web Analytics features and we’ll take a deep dive in to specific scenarios in future posts.

Overview

Web Analytics Reports

In SharePoint 2010, we have improved the set of Web Analytics reports that are available out-of-the-box, which will provide insights into the behavior of users of your SharePoint sites.  There are three categories of reports that you will find:

• Traffic reports: These reports provide metrics such as:
1. How much traffic your site gets (Number of Page Views);
2. Who visits your sites (Top Visitors);
3. How visitors arrive at your site (Top Referrers);
4. Daily Unique Visitors, Top Destinations, Top Browsers, etc;
• Search reports: These reports give you insight into what users are searching for, for example:
1. How many times users searched (Number of Queries);
2. What were the most used search terms (Top Queries);
3. What queries have high failure rates (Failed Queries);
4. Best Bet Usage, Search keywords, etc;
• Inventory reports: These reports display key metrics regarding the inventory of your sites:
1. What is the total disk drive space user (Storage Usage);
2. How many sites exist (Number of Sites);
3. Top Site Product Versions, Top Site Languages, etc;

We aggregate these reports aggregated at the following levels:

1. Per web application in the farm
2. Per site collection
3. Per site
4. Per search service application

Out-of-the-box, these reports are visible to Administrators at each level.  For example, site-level reports are available to Site Administrators of those sites.  We have also added a new permission level, “View Web Analytics Data,” that will allow users to access these reports without having to give them Administrator privileges.

You can access Web Analytics reports by going to Site Actions -> Site Settings.  Under the Site Actions heading you will see two links, Site Web Analytics Reports and Site Collection Web Analytics Reports. 

When you click on either link, you are taken to an overview page that shows you key metrics for your site.  You can then drill down to other reports by clicking on them on the left navigation. You can also change the date range for the reports by clicking on the Analyze tab on the Ribbon.

Custom Web Analytics Reports

The out-of-the-box reports are useful to get a general understanding of what is happening on your sites.  However, we have made it easy for you to get a deeper level of analysis, or to simply create your own reports.  To get started, click on the Customize Report button under the Analyze tab in the Ribbon.  Clicking this button will export the data contained in this report to Excel.  Excel is a power analytics tools and makes it easy for non technical users to add your own charts, set specific filters, and combine data from multiple reports.  In addition, the data within Excel is refreshable, which means that, once you customize the report, it will always be up-to-date with the latest data.  To get more details on the great new features in Excel 2010 for building charts, reports and pivot tables, take a look at the Excel Team blog.

Web Analytics Workflows

Web Analytics Workflows is a powerful new feature set that enables you to get reports sent out either on a schedule or when specific conditions are met.  For example, you can set them up to receive an email every time the total number of pages views drop by 80% week over week.

To setup a Web Analytics Workflow, go to the Web Analytics report that you are interested in and click on Schedule Alerts or Reports on the Analyze tab in the Ribbon.

Clicking this button will guide you through a series of steps to create your Workflow.

Best Bets Suggestions

Best Bets allow Search Administrators to determine what the most relevant search result is for a given keyword. Up until now, Search Administrators had to look at different reports and data to determine which best bets needed to be added. That process is no longer necessary as SharePoint 2010 periodically sends out suggestions for new Best Bets using all the search metrics it has collected. Now, Search Administrators can simply look through each of the Best Bet suggestions and easily accept or reject them.

To access the Best Bet Suggestions, go to Site Actions, click on Site Collection Web Analytics Reports, and the click on Best Bets Suggestions on the left navigation.

Web Analytics Web Part

We have created a new web part, the Web Analytics Web Part, targeted at Site Managers. This new Web Part is an end-user facing Web Part that can be easily inserted into any page on your site.  It can be configured to display the ‘most viewed content’ or the ‘most frequent search queries’ in the site. The data in the Web Part is continuously refreshed as new content or new search queries become more popular.

To use this Web Part, go into the Edit mode of one of your Site Pages and click on any place you can add a Web Part.  Then, from the Insert tab on the Ribbon, click on Web Part.  Finally, click on the Content Rollup category and select the Web Analytics Web Part.

After you have inserted the Web Analytics Web Part, you can then configure it to display the data you are interested in.

Conclusion

Using the new Web Analytics features in SharePoint 2010, you will be able to get a deeper understanding of what users are doing, what they want from your site and how you can tailor the SharePoint experience to bets meet their needs.   Keep an eye out for future posts where we will delve deeper into each of the features mentioned above.

Microsoft Office SharePoint Server 2007 included a hold feature that could be used for eDiscovery, but it was scoped to the Records Center site template. With SharePoint Server 2010 the eDiscovery capabilities have been greatly expanded to provide more functionality and the power to use these features across your entire SharePoint deployment.

In this post, I want to highlight three major improvements in SharePoint that support eDiscovery. You can:

• Manage holds and conduct eDiscovery searches on any site collection
• Use SharePoint Server Search or FAST Search for SharePoint out of box to search and process content
• Automatically copy eDiscovery search results to a separate repository for further analysis

Read on to learn how SharePoint Server 2010 can support your eDiscovery initiatives and provide you with the tools you need to manage holds, identify, and collect SharePoint content.

The eDiscovery Process

The Electronic Discovery Reference Model from EDRM (edrm.net) provides an overview of the different parts of the eDiscovery process:

SharePoint Sever 2010 addresses the Information Management, Identification, Preservation and Collection stages. While this blog post will focus mostly on the identification, preservation and collection components, SharePoint provides a rich Information Management platform for Collaboration, Social Computing, Document Management and Records Management.  This means that you can take a proactive approach to eDiscovery by putting a governance framework in place and using appropriate disposition policies to expire content. Managing content and deleting it when it is no longer needed will reduce the amount of content that must be indexed and searched, and collected for eDiscovery.  The result is that eDiscovery costs can be dramatically reduced, changing the problem from finding a needle in a hay stack to finding a needle in a hay bale. Ultimately, the key to achieving legal compliance for eDiscovery obligations is built upon a foundation of robust Information Management.

When an eDiscovery event occurs, such as a receipt of complaint, discovery, or notice of potential legal claim, the identification stage begins. Content that may be subject to eDiscovery must be identified and searches are conducted to find that content. That content needs to be preserved and at some point, the content will be collected.

The eDiscovery Features

Hold and eDiscovery

Hold and eDiscovery is a site level feature that can be activated on any site.

Activating this feature creates a new category in Site Settings that provides links to Holds and Hold Reports lists. There is also a page to discover and hold content that allows you to search for content and add it to a hold. Once the Hold and eDiscovery feature is activated you can create holds and add to hold any content in the site collection. By default only Site Collection administrators have access to the Hold and eDiscovery pages. To give other users permission, add them to the permissions list for the Hold Reports and Holds lists. This will also give access to the Discover and hold content page.

You can manually locate content in SharePoint and add it to a hold, or you can search for content and add the search results to a hold. With the Hold and eDiscovery feature you can create holds in the hold list and then manually add content to the relevant hold by clicking on Compliance Details from the drop down menu for individual items.

Then click on the link to Add/Remove from hold.

And you can select the relevant hold to add to or remove from.

By manually adding an item to hold you will block editing and deletion of that item until it is released from hold. You will notice that the document now has a lock icon showing that it cannot be edited or deleted.

Each night a report for each hold is generated by a timer job. If you need a hold report faster you can manually run the Hold Processing and Reporting timer job in Central Administration.

Search and Process

You can manually add items to hold on any site collection, which is great. But that doesn’t help you find the content you don’t already know about. What if you have a large amount of items you want to find and add to a hold? For that you can use the features on the Discover and hold content page, which is a settings page in Site Settings. From this page you can specify a search query and then preview the results. The configured search service (SharePoint Search Server or FAST Search for SharePoint) will automatically be used. You can then select the option to keep items on hold in place so they cannot be edited or deleted, or if you have configured a Content Organizer Send to location in Central Administration you can have content copied to another site and placed on hold. You may want to create a separate records center site for a particular hold to store all content related to that hold. The Content Organizer is a new SharePoint Server 2010 feature based on the Microsoft Office SharePoint Server 2007 Document Router with richer functionality to automatically classify content based on Content Type or metadata properties. Look for a future blog post covering the Content Organizer.

Holding content in place is recommended if you want to leave content in the location is was created with all the rich context that SharePoint provides, while blocking deletion and editing of content. Be aware that this will prevent users from modifying items. If you prefer users to continue editing documents, then use the copy to another location approach.

When searching and processing, the search will by default be scoped to the entire Site Collection and run with elevated permissions so all content can be discovered. The search can be scoped to specific sites and you can also preview search results before adding the results to a hold. Items can be placed on multiple holds and compliance details will show all of the holds that are applied to an item.

In summary, SharePoint Server 2010 contains key features that make it an essential aspect of your eDiscovery strategy. With the new SharePoint Server 2010 capabilities you can easily apply proper retention policies for all content and make it easier to discover content if an eDiscovery event occurs. eDiscovery often prescribes tight deadlines for production. SharePoint 2010 helps you find the right content and deliver it faster.

Quentin Christensen
Program Manager - Document and Records Management
Microsoft