SharePoint 2010 is more than just SharePoint 2007 plus a bunch of new bullet points on the box. We didn’t just haphazardly build a bunch of new features, look back at the fertile seeds we planted, and muse about how “everything should work pretty well as libraries get large.” We built, and more importantly, tested all the features you’re reading about with scale in mind. We are setting new scale targets for 2010 that go above and beyond what we set in 2007. These numbers are not final yet, but we’re shooting for tens of millions of documents in a single library, depending on some specific parameters of your scenario.

When I throw out numbers like that, I’m not talking about just big, static libraries with content that just sits there. We want you to do crazy things with SharePoint 2010 like stuff a million document sets in a single document library with workflows running every which way, a hundred different retention policies firing off actions when you least expect them, and users uploading, tagging, and searching day in and day out. All the goodness of the SharePoint platform will be available to you whether you’re building a team site, a collaborative repository, a knowledge base, or a super large archive.

Like a plump, juicy sausage, much of the good stuff in SharePoint 2010 to give it delicious scalability are things that most people don’t need (or want) to know about. For the most part, scale just works. However, the chef (or information architect) is still a super important player. A well-planned repository is one that will have your users coming back for seconds and writing rave reviews; a poorly-planned one is one that will have them chugging Pepto-Bismol the next morning. Just because you can stuff a bunch of documents in a SharePoint 2010 library without your server igniting in flames the next day at doesn’t mean that you should without first thinking through how to best use the tools available to deliver an excellent experience to your end users.

So, even though scale in SharePoint 2010 just works, you’re not going to install the bits on day 1 and have a massive, searchable, beautiful content storefront on day 2. Guidance still matters, and believe me, we know it; this blog entry is just the beginning of the content we’re planning on delivering to help you on this front. I wouldn’t even call this blog entry guidance; it’s just a primer on the features and capabilities of SharePoint 2010 that you will grow to love if you’re passionate about scale at the library level – if you want to shove a whole bunch of documents in one place and have it be a great experience for both IT and your end users.

So what are these features and capabilities? Here are a few of the most important ones that I’m going to blog about now and in the near future:

1. We protect your database backend from dangerous queries. If you run a query against any database that requires it to scan through millions of items to find the ones you’re asking for, you’re going to balk at how long it ties up the server’s CPU. Quite frankly, SharePoint is not an exception. Even in SharePoint 2010, there is a class of user operations in certain scenarios that make unreasonable demands on the backend. For these operations, our strategy is to nip them in the bud before they’re executed, which keeps your high-demand servers healthy and responsive. Knowing when this throttling will kick in and planning for it is an important part of large list planning.
2. We give end-users tools to find content. When you have a sea of documents, the specific one you’re looking for can seem like a needle in a haystack. Structured metadata, easy tagging, metadata navigation, and built-in search refinement make this a less daunting task in SharePoint 2010 out of the box. This is an area we are particularly passionate about; after all, what good is a hugely scalable library if your end users hate it and can’t find what they’re looking for?
3. We help developers write excellent code. In SharePoint 2007, we didn’t give developers the right tools to write code that scaled well as the amount of content in your site grew. Even worse, it was pretty hard to tell why and when code was bad, and if your site was running slowly, which one out of your ten custom web parts was bringing things down. You had to “build around” SharePoint and do things “just so” to avoid this from happening. In SharePoint 2010, we give you a bunch of tools to make this story better.

Dangerous Queries

One challenge we’ve consistently seen customers run into when building large repositories on SharePoint 2007 is trouble with large containers. As the number of documents in any single container grows – either at the root of a library, or in a folder – bad things start to happen. For one, as your document to container ratio increases, it becomes harder and harder to find exactly what you’re looking for. More serious are the performance implications of large containers. Any of the out of the box ways of retrieving content from containers in SharePoint 2007 – like the All Documents view, the Explorer view, or a Content Query web part – would work, but they don’t scale very well. Loading All Documents in a library with a million items at the root would take a long time to finish. The big problem here is that you wouldn’t be the only one affected; all your friends running SharePoint sites on that same database server would experience things slowing to a crawl as well, as the database server dutifully iterated over those million documents to find the right ones.

Why does this happen? Any time you ask for content from SharePoint, you have to specify how it’s sorted – for example, the All Documents view in SharePoint 2007 asks for the top 100 results, sorted by filename. But items aren’t sorted by filename in the SharePoint content database – so, to bring you this view, SharePoint has to gather up all these million items, sort them, and finally display the 100 ones at the top of the sorted list. Imagine this as being like flipping through the residential section of a phonebook to find the first 100 addresses, sorted in alphabetical order. This would be a miserable task, because the telephone book isn’t sorted in this way – so in order to ensure your sorted list was accurate in the end, you’d have to look through the entire residential section, from start to finish, because after all, the last person listed in the phone book might live at 1000 Aardvark Lane.

Large Lists and Fallback Queries

The laws of physics are the same in SharePoint 2010 as they were in SharePoint 2007; if you run a query that needs to touch a very large number of items, you’re going to have to wait a long time, and so will everybody else. One prominent thing we did in SharePoint 2010 is to nip these queries in the bud before they get executed. To make a long story short (you can read the long story here), a farm administrator can set a threshold which defines the maximum number of items a single SharePoint query can touch. By default this threshold is 5,000. Any library with more items than this threshold is a large list.

Let’s go back to our example of the library with one million items at the root. Say you had that library in SharePoint 2007, and you upgraded to SharePoint Server 2010. First thing you’ll see upon navigating to this library will look something like this:

See the yellow bar above the list view? That’s a sign you have the Metadata Navigation and Filtering site feature turned on and it’s causing something magical to happen! When you load this view, SharePoint 2010 knows that you’re being greedy and asking it to scan through those million items. Since this query exceeds the maximum number of items a single query is allowed to scan (5,000) it doesn’t run the query. But who wants to stare at an empty list view? Instead of running this query as-is, SharePoint finagles it a bit and transforms it into a query that’s almost as good as the one you were asking for, but won’t make the database buckle under the pressure. In this case, we assume that it’s fairly likely that the document you’re looking for is one of the most recently created items in the library – so instead of scanning all one million items, we only scan the top 1,000 or so recently created documents, sort those by filename, and show them to you in the list view. This is what we call a simple fallback query: a query that doesn’t specify an index and asks for too many items in return, so instead of considering the entire list as being eligible for the query, SharePoint considers only the thousand or so most recently added items.

“Wait a second. You’re telling me that SharePoint throttles queries without asking me first? How on earth am I supposed to find anything in this crazy world of fallback queries and partial results?”

Let me assure you; this throttling business is a good thing. It’s a core ingredient in what makes SharePoint 2010 a resource for addressing your scale challenges. Gone are the sleepless nights where you toss and turn and worry about page faults on your database cluster resulting from Mack in Accounting stuffing 6,000 beer pong tournament photos in the root of a library in a forgotten team site in the dusty corners of your SharePoint deployment. The SharePoint 2010 feature set replaces this overarching concern with a set of well-scoped challenges; instead of worrying about every library that might get big, you get to plan for and craft experiences for the set of libraries that need to get big for business reasons.

I should mention really quickly that throttling is about more than just list views. There is a whole class of operations that involve iterating through all the documents in a list, or all the documents in a folder, that will get throttled (in other words, they will not execute) when the list or container is large. These operations include things like:

• Adding a column to a library
• Creating an index on a library
• Deleting a large folder in a library

Metadata Navigation – finding and working on content in large lists

Above is another screenshot from my million item library. This time, we’ve put a couple of SharePoint 2010 features to work. See that I have “demonstration scripts” selected in the left hand side in the tree view, and my list view is rendering without the yellow bar that’s telling me I’m only seeing newest results. That hierarchy of tags you see there represents a taxonomy, Item Type. I am browsing the documents in this library according to their Item Type; in the screenshot, I am filtering to show all documents with the value “demonstration scripts”. Here are the steps that I took to make this happen:

1. I created a taxonomy that describes my content. You can look forward to some posts from our very own Dan Kogan on this very topic in this very blog in the near future. There’s a lot to learn here. Not just any taxonomy will do here; it needs to be one that broadly divides my content up into evenly-sized buckets. For example, if I had 990,000 demonstration scripts, the query you see above would not get me anywhere. In that case, it wouldn’t make much sense to use Item Type as a piece of metadata and a navigation hierarchy for this library; I would need to find another, more divisive way to pivot the data.
2. I bound that taxonomy to a field in my library called Item Type. Think of a taxonomy field as a choice field on steroids. Instead of picking values out of a flat list, you pick them out of a tree.
3. I configured that field as a navigation hierarchy. Every library now has a Metadata Navigation and Filtering settings page where you can configure navigation hierarchies (the filters you see arranged in the tree view) and key filters (the additional filters that show up beneath the tree view)

In these three easy steps, I made “Item Type” a first class navigational pivot over the data. Instead of just staring at a partial list of content at the root, I can now browse with impunity by this virtual folder structure.

Here’s a couple of cool aspects of this feature that aren’t apparent from a single nifty screenshot:

1. Metadata navigation lets you slice and dice multiple ways. I might have a bunch of taxonomies on my library that classify content in different ways; for example, I might have a Products field, a Region field and a Competitors field, all bound to domain-specific taxonomies that classify the content along those dimensions. Depending on my current task, it might make more sense to filter by the Region field (for example, if I’m looking for the latest sales figures for the North America region). I get more filters than just my virtual folder; I can combine this filter with any number of key filters or list view column filters to drill down to just the content I want (for example, I want to see all demonstration scripts by the ECM team created after 2007).
2. Metadata navigation thinks about indices and large lists so you don’t have to. Hey, remember just a few minutes ago when we were talking about large lists, indices, and being throttled? Well, metadata navigation thinks a lot about indices and how to run queries the “right way” to make them perform well and prevent throttling from happening. For starters, all the fields you configure as navigation hierarchies and key filters get indexed, and the resulting queries are written in a way that ensures the best index is used to make the query succeed.

You aren’t immune from the laws of physics; if you ask for documents tagged with demonstration scripts and there are 10,000 demonstration scripts, we’re not going to be able to show you all of them. In this case, though, you get something better than a simple fallback; you get an indexed fallback, which means that instead of considering the entire list, the query considers only the items that match the indexed portion of your query.

Wrap-up

This article was just the first in my series of posts about architecting and building large lists filled with discoverable content. Here’s what you can expect over the next few weeks:

• A deep dive on metadata navigation, how it works, and some tips to getting the most out of it
• A discussion on how other features, like Search and the Content Query Web Part, fit into the equation, and how to configure their metadata filtering capabilities
• Some geeky developer tips on writing code that plays nicely with large lists

After that, I’ll be widening my scope a bit to talk about the overall knowledge management story in SharePoint 2010 – which is about more than just browsing for content in a library!

Lincoln DeMaris, Program Manager, ECM

Hi everyone, I am Quentin Christensen and I work on document and records management functionality for SharePoint. Electronic discovery (commonly referred to as eDiscovery) is an area we are supporting with new set of capabilities in SharePoint Server 2010. In case you are not familiar with eDiscovery, it is the process of finding, preserving, analyzing and producing content in electronic formats as required by litigation or investigations. eDiscovery is an important concern for all of our customers and given that SharePoint has grown to be an integral part of collaboration, document, and records management for many organizations, we recognize the need to support the eDiscovery process for SharePoint content.

Microsoft Office SharePoint Server 2007 included a hold feature that could be used for eDiscovery, but it was scoped to the Records Center site template. With SharePoint Server 2010 the eDiscovery capabilities have been greatly expanded to provide more functionality and the power to use these features across your entire SharePoint deployment.

In this post, I want to highlight three major improvements in SharePoint that support eDiscovery. You can:

• Manage holds and conduct eDiscovery searches on any site collection
• Use SharePoint Server Search or FAST Search for SharePoint out of box to search and process content
• Automatically copy eDiscovery search results to a separate repository for further analysis

Read on to learn how SharePoint Server 2010 can support your eDiscovery initiatives and provide you with the tools you need to manage holds, identify, and collect SharePoint content.

The eDiscovery Process

The Electronic Discovery Reference Model from EDRM (edrm.net) provides an overview of the different parts of the eDiscovery process:

SharePoint Sever 2010 addresses the Information Management, Identification, Preservation and Collection stages. While this blog post will focus mostly on the identification, preservation and collection components, SharePoint provides a rich Information Management platform for Collaboration, Social Computing, Document Management and Records Management.  This means that you can take a proactive approach to eDiscovery by putting a governance framework in place and using appropriate disposition policies to expire content. Managing content and deleting it when it is no longer needed will reduce the amount of content that must be indexed and searched, and collected for eDiscovery.  The result is that eDiscovery costs can be dramatically reduced, changing the problem from finding a needle in a hay stack to finding a needle in a hay bale. Ultimately, the key to achieving legal compliance for eDiscovery obligations is built upon a foundation of robust Information Management.

When an eDiscovery event occurs, such as a receipt of complaint, discovery, or notice of potential legal claim, the identification stage begins. Content that may be subject to eDiscovery must be identified and searches are conducted to find that content. That content needs to be preserved and at some point, the content will be collected.

The eDiscovery Features

Hold and eDiscovery

Hold and eDiscovery is a site level feature that can be activated on any site.

Activating this feature creates a new category in Site Settings that provides links to Holds and Hold Reports lists. There is also a page to discover and hold content that allows you to search for content and add it to a hold. Once the Hold and eDiscovery feature is activated you can create holds and add to hold any content in the site collection. By default only Site Collection administrators have access to the Hold and eDiscovery pages. To give other users permission, add them to the permissions list for the Hold Reports and Holds lists. This will also give access to the Discover and hold content page.

You can manually locate content in SharePoint and add it to a hold, or you can search for content and add the search results to a hold. With the Hold and eDiscovery feature you can create holds in the hold list and then manually add content to the relevant hold by clicking on Compliance Details from the drop down menu for individual items.

Then click on the link to Add/Remove from hold.

And you can select the relevant hold to add to or remove from.

By manually adding an item to hold you will block editing and deletion of that item until it is released from hold. You will notice that the document now has a lock icon showing that it cannot be edited or deleted.

Each night a report for each hold is generated by a timer job. If you need a hold report faster you can manually run the Hold Processing and Reporting timer job in Central Administration.

Search and Process

You can manually add items to hold on any site collection, which is great. But that doesn’t help you find the content you don’t already know about. What if you have a large amount of items you want to find and add to a hold? For that you can use the features on the Discover and hold content page, which is a settings page in Site Settings. From this page you can specify a search query and then preview the results. The configured search service (SharePoint Search Server or FAST Search for SharePoint) will automatically be used. You can then select the option to keep items on hold in place so they cannot be edited or deleted, or if you have configured a Content Organizer Send to location in Central Administration you can have content copied to another site and placed on hold. You may want to create a separate records center site for a particular hold to store all content related to that hold. The Content Organizer is a new SharePoint Server 2010 feature based on the Microsoft Office SharePoint Server 2007 Document Router with richer functionality to automatically classify content based on Content Type or metadata properties. Look for a future blog post covering the Content Organizer.

Holding content in place is recommended if you want to leave content in the location is was created with all the rich context that SharePoint provides, while blocking deletion and editing of content. Be aware that this will prevent users from modifying items. If you prefer users to continue editing documents, then use the copy to another location approach.

When searching and processing, the search will by default be scoped to the entire Site Collection and run with elevated permissions so all content can be discovered. The search can be scoped to specific sites and you can also preview search results before adding the results to a hold. Items can be placed on multiple holds and compliance details will show all of the holds that are applied to an item.

In summary, SharePoint Server 2010 contains key features that make it an essential aspect of your eDiscovery strategy. With the new SharePoint Server 2010 capabilities you can easily apply proper retention policies for all content and make it easier to discover content if an eDiscovery event occurs. eDiscovery often prescribes tight deadlines for production. SharePoint 2010 helps you find the right content and deliver it faster.

Quentin Christensen
Program Manager – Document and Records Management
Microsoft

Hi everyone.  My name is Adam Harmetz and I work on the engineering team responsible for the SharePoint document and records management vision and features.  Many of you might remember me from the SharePoint 2007 recman blog.  The recman blog was a great way for the team to connect with records managers, IT professionals, and information architects and we’ll be continuing that discussion for the SharePoint 2010 compliance features via the Enterprise Content Management (ECM) Team Blog.

I think it makes sense to combine records management with other facets of ECM into one central blog.  After all, as Jim discussed, records management is a key component of our ECM strategy.  The notion that everyone should participate in ECM processes really served as a guiding principle to help expand the scope of records management in SharePoint 2010.  And for all you records managers out there, I think you’ll benefit greatly from learning about the other facets of ECM along the way.

To kick off the discussion, here are three key things you need to know about records management in SharePoint 2010.

The Records Center – A Place for Hierarchy, Driven By Metadata 

The Records Center was introduced in 2007 as a SharePoint site that served as a conventional records archive.   Content from all over the enterprise can be submitted to a Records Center and then routed to the appropriate place where it picks up the right permissions and policies, such as expiration and auditing.
For SharePoint 2010, we know it’s important to continue to invest here and add even more “traditional” archive features.   When looking at the broad swath of features we had to choose from, our goals here really focused on providing features that allow you to extract the most value out of an archive and find the data you need.  For instance, here are a few of the new features in a SharePoint 2010 Records Center:

• Document ID: Every document can be assigned a unique identifier, which stays with the document even when it’s archived.  This allows records to be easily referenced by an ID no matter where the document moves.


• Multi-Stage Retention: Retention policies can have multiple stages, allowing you to specify the entire document lifecycle as one policy (e.g. review Contracts every year, and delete after 7 years)


• Per-Item Audit Reports: You can generate a customized audit report about an individual record.


• Hierarchal File Plans: You can create deep, hierarchal folder structures and manage retention at each folder in the hierarchy (or inherit from parent folders).


• File Plan Report: You can generate status reports showing the number of items in each stage of the file plan, along with a rollup of the retention policies on each node in the plan.

Here’s the home page of the Records Center in SharePoint 2010 for a fictional government agency, the Joint Task Force.  Notice that the home page is a place for records managers to educate the organization on compliance policy, as well as a place to look up a record by its document identifier.

In addition to adding these traditional records management features to our archive, as product designers we made a big bet on the power of metadata to dive 21st century electronic records management.  This manifests itself in several ways in the SharePoint archive:

• Taxonomy and Centralized Content Types:  The archive will be a consumer of enterprise-wide taxonomies and content types, ensuring consistency and context transfer between the collaborative spaces and the archive.  We’ll be talking a lot more about our 2010 taxonomy investments in future posts.


• Content Organizer: The records router can use metadata to route incoming documents to the right place in the hierarchical file plan.  For instance, it enables you to automatically enforce rules on content that is submitted, like “If a Purchase Agreement is tagged with Project Alpha, send to the Alpha Contracts subfolder and apply that’s folder retention policy to the item.”


• Virtual Folders: The file plan is a great way to manage a repository but often time isn’t what you want to use to navigate and find the content you are looking for.  The SharePoint 2010 Records Center makes use of a new feature called metadata based navigation, which allows you to expose key metadata as virtual folders:

Notice that end users discover content in this Records Center by navigating virtual folders based upon metadata properties on the records.

This bet on metadata is all about empowering the end user, thus increasing the chance of successful adoption of the RM system.  Instead of choosing a complicated node in a file plan, submitters just fill out a few pieces of useful metadata and they’ll use that metadata when they need to find the content again.

In Place Records Management – Injecting Records Management in the Content Creation Experience

With just about every customer engagement my team is involved in, we hear the same message again and again: records management doesn’t start (or stop!) in the archive.  Content isn’t created there and it sure doesn’t live there for the most interesting parts of its life.

We’ve made a huge effort in 2010 to enable you to do effective records management in collaborative spaces.  Auditing, Retention, Expiration, Reporting, Records Workflows, eDiscovery, Legal Hold and Recordization are all features you can use in collaborative space as you are striking a balance between SharePoint’s value to end users and the need for information governance.

Holding all of this together is a new feature in SharePoint 2010 called In Place Records Management.  This allows certain SharePoint documents (or blogs, wikis, web pages, and list items) to be declared records.  The system can prevent such records from being deleted or edited, if necessary by your organization’s definition of what a record is:

Note that some of the documents have locks, implying to the user that they are dealing with records.  When selecting a record, the UI for editing and deleting the item is disabled.

This recordization process can be done either manually, as part of a larger process in a workflow, or as a scheduled part of a document’s retention (e.g. after 2 years).  The key here is that, when declared a record, the content doesn’t move to an archive – it stays where it is so the end users can still find and interact with the content.

Once declared, the system knows about an item’s record status, so you can do things such as create different retention policies for records or use record state when defining workflows in SharePoint Designer.  We also enable a programmability model so you can perform custom processes and policies upon recordization to meet specialized compliance needs.

Is In Place Records a replacement for a traditional archive?  The answer is, of course, sometimes – we’ll find some customers who want to use an in place approach exclusively, some who will want the traditional hierarchy and centralization that an archive brings, and many who will want both.  It’ll be something we’ll talk about a lot on this blog, and our documentation has already started discussing the pros and cons of both approaches.

Scale: We’re Talking Big

With electronic information growing at a crazy pace and businesses spending billions on eDiscovery every year, records managers have enough to keep them up at night.  The scale of their records/content management system shouldn’t be another worry.

As the records management engineering team, we take this burden very seriously and a large part of our effort this release has been spent adding features to make it easier to scale to massive archives.  Features such as Remote Blob Storage, database query optimizations, internal timer job processing improvements, new database indexing strategies and other engineering initiatives enable us to make a great leap forward this release and allow our customers to have:

• Tens of millions of records in a single Records Center


• Hundreds of millions of records in a distributed archive: We’ll talk in more detail in future posts, but many of the features mentioned above light up to allow many Record Centers to bind together to act as one logical repository.

With our partners on the SharePoint blog, we are looking forward to showing more details on the new scale targets and performance profiles for deployments at this scale over the coming months.

Wrapping Up

It’s been a lot of hard work for the team around here to deliver on this vision for 21st century records management.  When combined with the integrated e-mail archiving, retention, and discovery capabilities of Exchange 2010, I think you’ll see the 2010 wave as a breakout release for Microsoft’s records management strategy.

The team here is proud of the work here and eager to talk about it and hear from everyone – feel free to leave suggestions on future blog post ideas in the comments!

Thanks for reading,
Adam Harmetz
Lead Program Manager

P.S. If you are hungry for even for information on SharePoint 2010 records management, check out an interview I did on Don Lueder’s blog.