Hi everyone.  My name is Adam Harmetz and I work on the engineering team responsible for the SharePoint document and records management vision and features.  Many of you might remember me from the SharePoint 2007 recman blog.  The recman blog was a great way for the team to connect with records managers, IT professionals, and information architects and we’ll be continuing that discussion for the SharePoint 2010 compliance features via the Enterprise Content Management (ECM) Team Blog.

I think it makes sense to combine records management with other facets of ECM into one central blog.  After all, as Jim discussed, records management is a key component of our ECM strategy.  The notion that everyone should participate in ECM processes really served as a guiding principle to help expand the scope of records management in SharePoint 2010.  And for all you records managers out there, I think you’ll benefit greatly from learning about the other facets of ECM along the way.

To kick off the discussion, here are three key things you need to know about records management in SharePoint 2010.

The Records Center – A Place for Hierarchy, Driven By Metadata 

The Records Center was introduced in 2007 as a SharePoint site that served as a conventional records archive.   Content from all over the enterprise can be submitted to a Records Center and then routed to the appropriate place where it picks up the right permissions and policies, such as expiration and auditing.
For SharePoint 2010, we know it’s important to continue to invest here and add even more “traditional” archive features.   When looking at the broad swath of features we had to choose from, our goals here really focused on providing features that allow you to extract the most value out of an archive and find the data you need.  For instance, here are a few of the new features in a SharePoint 2010 Records Center:

• Document ID: Every document can be assigned a unique identifier, which stays with the document even when it’s archived.  This allows records to be easily referenced by an ID no matter where the document moves.


• Multi-Stage Retention: Retention policies can have multiple stages, allowing you to specify the entire document lifecycle as one policy (e.g. review Contracts every year, and delete after 7 years)


• Per-Item Audit Reports: You can generate a customized audit report about an individual record.


• Hierarchal File Plans: You can create deep, hierarchal folder structures and manage retention at each folder in the hierarchy (or inherit from parent folders).


• File Plan Report: You can generate status reports showing the number of items in each stage of the file plan, along with a rollup of the retention policies on each node in the plan.

Here’s the home page of the Records Center in SharePoint 2010 for a fictional government agency, the Joint Task Force.  Notice that the home page is a place for records managers to educate the organization on compliance policy, as well as a place to look up a record by its document identifier.

In addition to adding these traditional records management features to our archive, as product designers we made a big bet on the power of metadata to dive 21st century electronic records management.  This manifests itself in several ways in the SharePoint archive:

• Taxonomy and Centralized Content Types:  The archive will be a consumer of enterprise-wide taxonomies and content types, ensuring consistency and context transfer between the collaborative spaces and the archive.  We’ll be talking a lot more about our 2010 taxonomy investments in future posts.


• Content Organizer: The records router can use metadata to route incoming documents to the right place in the hierarchical file plan.  For instance, it enables you to automatically enforce rules on content that is submitted, like “If a Purchase Agreement is tagged with Project Alpha, send to the Alpha Contracts subfolder and apply that’s folder retention policy to the item.”


• Virtual Folders: The file plan is a great way to manage a repository but often time isn’t what you want to use to navigate and find the content you are looking for.  The SharePoint 2010 Records Center makes use of a new feature called metadata based navigation, which allows you to expose key metadata as virtual folders:

Notice that end users discover content in this Records Center by navigating virtual folders based upon metadata properties on the records.

This bet on metadata is all about empowering the end user, thus increasing the chance of successful adoption of the RM system.  Instead of choosing a complicated node in a file plan, submitters just fill out a few pieces of useful metadata and they’ll use that metadata when they need to find the content again.

In Place Records Management – Injecting Records Management in the Content Creation Experience

With just about every customer engagement my team is involved in, we hear the same message again and again: records management doesn’t start (or stop!) in the archive.  Content isn’t created there and it sure doesn’t live there for the most interesting parts of its life.

We’ve made a huge effort in 2010 to enable you to do effective records management in collaborative spaces.  Auditing, Retention, Expiration, Reporting, Records Workflows, eDiscovery, Legal Hold and Recordization are all features you can use in collaborative space as you are striking a balance between SharePoint’s value to end users and the need for information governance.

Holding all of this together is a new feature in SharePoint 2010 called In Place Records Management.  This allows certain SharePoint documents (or blogs, wikis, web pages, and list items) to be declared records.  The system can prevent such records from being deleted or edited, if necessary by your organization’s definition of what a record is:

Note that some of the documents have locks, implying to the user that they are dealing with records.  When selecting a record, the UI for editing and deleting the item is disabled.

This recordization process can be done either manually, as part of a larger process in a workflow, or as a scheduled part of a document’s retention (e.g. after 2 years).  The key here is that, when declared a record, the content doesn’t move to an archive – it stays where it is so the end users can still find and interact with the content.

Once declared, the system knows about an item’s record status, so you can do things such as create different retention policies for records or use record state when defining workflows in SharePoint Designer.  We also enable a programmability model so you can perform custom processes and policies upon recordization to meet specialized compliance needs.

Is In Place Records a replacement for a traditional archive?  The answer is, of course, sometimes – we’ll find some customers who want to use an in place approach exclusively, some who will want the traditional hierarchy and centralization that an archive brings, and many who will want both.  It’ll be something we’ll talk about a lot on this blog, and our documentation has already started discussing the pros and cons of both approaches.

Scale: We’re Talking Big

With electronic information growing at a crazy pace and businesses spending billions on eDiscovery every year, records managers have enough to keep them up at night.  The scale of their records/content management system shouldn’t be another worry.

As the records management engineering team, we take this burden very seriously and a large part of our effort this release has been spent adding features to make it easier to scale to massive archives.  Features such as Remote Blob Storage, database query optimizations, internal timer job processing improvements, new database indexing strategies and other engineering initiatives enable us to make a great leap forward this release and allow our customers to have:

• Tens of millions of records in a single Records Center


• Hundreds of millions of records in a distributed archive: We’ll talk in more detail in future posts, but many of the features mentioned above light up to allow many Record Centers to bind together to act as one logical repository.

With our partners on the SharePoint blog, we are looking forward to showing more details on the new scale targets and performance profiles for deployments at this scale over the coming months.

Wrapping Up

It’s been a lot of hard work for the team around here to deliver on this vision for 21st century records management.  When combined with the integrated e-mail archiving, retention, and discovery capabilities of Exchange 2010, I think you’ll see the 2010 wave as a breakout release for Microsoft’s records management strategy.

The team here is proud of the work here and eager to talk about it and hear from everyone – feel free to leave suggestions on future blog post ideas in the comments!

Thanks for reading,
Adam Harmetz
Lead Program Manager

P.S. If you are hungry for even for information on SharePoint 2010 records management, check out an interview I did on Don Lueder’s blog.

My name is Jim Masson, and I’m the Group Program Manager for the Enterprise Content Management team within SharePoint. My team is part of the engineering team, and is responsible for designing the features around content management, including managing documents, web content, rich media assets, records, and a new service for managing shared content types and taxonomy.

With the coming launch of SharePoint 2010, this seemed like a good time to ramp up the ECM team blog, and start a conversation about the SharePoint 2010 release. In the lead up to the offical launch of the product and beyond, various members of the team will be posting details about the major feature areas and features within ECM in SharePoint 2010, including design overviews, walkthroughs, best practices, and eventually interesting case studies. I hope you will subscribe and participate with us in the conversation

ECM For the Masses

When speaking with customers about the content management features in SharePoint 2010, we often refer to the release as being about ECM for the Masses.  I wanted to take this first post to outline a little bit about our approach to designing and building SharePoint 2010, and how that has helped us deliver on that vision.

• Enterprise Ready – This is all about ensuring that SharePoint more easily scales to the amount of content the largest Enterprises deal with and delivers consistently high performance and reliability at any scale. In addition we provide the feature depth, customizability and extensibility that Enterprises need to support the full breadth of business scenarios around content.


• Easy to Use – Our focus here is on 2 audiences. First, the features must be Easy for the Information Worker, with best-in-class usability, providing supreme user acceptance and speeding deployment and adoption. Second, the product must also be Easy for IT, providing great functionality OOB that is fast to deploy and easy to manage at the Enterprise, Divisional, Team and Workgroup levels.


• Everyone Participates – This is all about ensuring that Everyone in the organization has access to and benefits from the functionality offered by the ECM features – not just a few specialists who have been specially trained, or for whom the organization can justify a high per seat price. This also means that the capabilities can be adjusted to suit the needs of everyone in the organization; from minimal interaction to highly structured and complex workflows – everyone sees exactly as much as they need.

It is my hope that, as we go through the features over the next several months that you will see the impact of those pillars on the product, and how they have helped us to deliver ECM for the masses.

Around two-and-a-half years ago (at the AIIM conference in Philadelphia in May of 2006), my counterparts at IBM and EMC and I started discussing the need to form a group to create an open services standard for interacting with Enterprise Content Management systems (like SharePoint, IBM FileNet P8, EMC Documentum, etc.) in a uniform way. An earlier blog post explained the full rationale – but in short many customers and partners made it apparent that having to create one-off “connectors” between each application (like eDiscovery applications, Portals or Business Process Management systems) and ECM system was making it hard for customers to use more than one ECM system and for partners to build great applications that could “just work” with whatever systems a customer is using.

And now, after working with many other vendors like Alfresco, Nuxeo, OpenText, Oracle, SAP, and others on the CMIS specification, forming a Technical Committee at OASIS to deliver that specification as a truly open standard, and having four “plug-fest” events where we’ve tested actual (prototype) implementations of the spec together to make sure it would work in the real-world – I’m thrilled to announce that on October 23, 2009 Version 1.0 of the CMIS specification entered OASIS’ public review process.

“Public Review” — What does that mean?

Those of you who aren’t familiar with the mechanics of the OASIS standards process (i.e. nearly everyone) are probably wondering what “public review” means, and how it relates to everyone’s ultimate goal of having a final 1.0 specification available so that everyone can start supporting CMIS in their applications. (If you want the full details of how the Public Review Process works, you can read OASIS official “Technical Committee Process” rules – but the summary below is a bit more self-contained and user-friendly).

Public Review is one of the final stages of the OASIS standardization process –it means that the members of the Technical Committee think the spec is (almost completely) done, and that we’re soliciting feedback from the general public about what changes (if any) they’d like to see in the
specification before it becomes the final CMIS 1.0 standard. The public review period lasts for 2 months – so it will continue until December 22, 2009.

Anyone can look at the spec and send us comments–I’ve also copied the full public review announcement below for reference.

The Technical Committee is then required to review and respond to all comments – which could include updating the specification, or responding to those comments without making a change (which we would likely do if the comment is asking for new features or big enough that it would be better deferred to a future version of the specification). If those comments result in substantive changes, then the updated spec would undergo a shorter (15-day) additional round of public review. If not, then the Technical Committee will submit the CMIS 1.0 specification (possibly with some minor clarification updates) for a final approval vote by the OASIS membership – a process which takes about a month.

Once the final OASIS approval vote is closed (assuming of course that CMIS gets sufficient votes to pass), CMIS 1.0 is (finally) an OASIS standard!

OK… so when will CMIS 1.0 be final?

Those of you keeping count from the above paragraph already figured this out – but CMIS is on track to become a final 1.0 standard sometime in the first 3 months of 2010 (exactly how soon will depend on the volume of comments we get in Public Review and the changes required to address them).

Given all of the work that the Technical Committee has done in writing & testing the spec so far, we aren’t expecting to make many (if any) substantive changes – but of course if there are any issues in the spec that will hamper it’s real-world adoptability we want to hear about and address those now in Public Review, rather than waiting until the 1.0 standard is final (when making changes will require a whole new version of the specification.) So please do review the spec and give us your feedback!

When can I expect vendors (including Microsoft) to start supporting CMIS in their products?

At this point, pretty much every vendor in the ECM space is really motivated to start supporting CMIS in their respective products. We’ve all seen the excitement from customers about CMIS — for example, a recent AIIM survey showed that 15% of organizations are already interested in using CMIS. (This is an unbelievable number for a standard that isn’t even final yet!)

Of course, the prerequisite for all this is a final, OASIS-ratified 1.0 standard. While several companies have released prototypes based on interim drafts (which are wonderful proof-points that CMIS is ready for real-world implementation), look for vendors to start disclosing specific plans once the specification is final.

For Microsoft’s part, we announced at the ARMA 2009 Conference and at our own SharePoint Conference in the last two weeks that we are planning to deliver support for CMIS within SharePoint 2010.

Since that announcement we’ve gotten lots of requests for additional details (“give me an exact date!”, “tell me exactly what functionality will be included”, etc.) I wish that this blog post could be the place to provide more detail – but it’s simply not possible at this time. Here’s why — those of you who attended the SharePoint Conference last week have seen that SharePoint 2010 is looking pretty shiny and polished. But until the CMIS 1.0 specification is final, we can’t realistically commit to exact dates when our CMIS support would be ready. This means that our plans need to be flexible to balance the following needs:

1. Not rushing the finalization of the CMIS 1.0 specification in a way that would compromise its quality


2. Release CMIS support as soon as possible for SharePoint 2010 that meets the interoperability needs of our customers and partners

We’re definitely looking forward to having the CMIS standardization process complete so we can lock-down our plans to the point where we can share additional details. Please stay tuned for more information.

Ethan Gur-esh,
Program Manager
CMIS Specification Editor
CMIS Technical Committee Secretary

Copy of the OASIS Public Review Announcement:

To OASIS members, Public Announce Lists:

The OASIS Content Management Interoperability Services (CMIS) TC has 
recently approved the following specification as a Committee Draft and 
approved the package for public review:

Content Management Interoperability Services (CMIS) Version 1.0

The public review starts today, 23 October 2009, and ends 22 December 
2009. This is an open invitation to comment. We strongly encourage 
feedback from potential users, developers and others, whether OASIS 
members or not, for the sake of improving the interoperability and 
quality of OASIS work. Please feel free to distribute this 
announcement within your organization and to other appropriate mail 
lists.

More non-normative information about the specification and the 
technical committee may be found at the public home page of the TC at:
http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=cmis. 
Comments may be submitted to the TC by any person through the use of 
the OASIS TC Comment Facility which can be located via the button 
marked “Send A Comment” at the top of that page, or directly at:
http://www.oasis-open.org/committees/comments/index.php?wg_abbrev=cmis.

Submitted comments (for this work as well as other works of that TC) 
are publicly archived and can be viewed at:
http://lists.oasis-open.org/archives/cmis-comment/. All comments 
submitted to OASIS are subject to the OASIS Feedback License, which 
ensures that the feedback you provide carries the same obligations at 
least as the obligations of the TC members.

The specification document and related files are available here:

Editable Source:
http://docs.oasis-open.org/cmis/CMIS/v1.0/cd04/cmis-spec-v1.0.doc
PDF:
http://docs.oasis-open.org/cmis/CMIS/v1.0/cd04/cmis-spec-v1.0.pdf
HTML:
http://docs.oasis-open.org/cmis/CMIS/v1.0/cd04/cmis-spec-v1.0.html

Schema:
http://docs.oasis-open.org/cmis/CMIS/v1.0/cd04/CMIS-Core.xsd
http://docs.oasis-open.org/cmis/CMIS/v1.0/cd04/CMIS-Messaging.xsd
http://docs.oasis-open.org/cmis/CMIS/v1.0/cd04/CMIS-RestAtom.xsd


OASIS and the CMIS TC welcome your comments.


Mary P McRae
Director, Technical Committee Administration
OASIS: Advancing open standards for the information society
email: mary.mcrae@oasis-open.org
web: www.oasis-open.org
twitter: fiberartisan #oasisopen
phone: 1.603.232.9090