SharePoint world of ECM and Information Management

Cascaded Master-Details lookup that just WORKS!

Last several days I tried to find the solution that allows me to use “master-detail” field relationship based on the lookups. There are ton’s of controls for this, but I hardly found one that works for the lookup fields in the master list.

Scenario:

You have the master list of company’s projects and master list [Projects] of all possible projects’ stages [Stages]. For each new project you create the [Team Project] list where you JOIN the project with the stages from master lists, using lookup column.

Now, you need to create the [Team Staff] list, where you want to assign users to the different sections of your team projects in [Team Projects] list.

For example, we have the following structure of [Team Project] list

You want to assign users to the specific sections, so, for example “John” is responsible for the “Deploy Farm” only in the “Build SharePoint site” project and “Mary” is responsible for “test feature” stuff in another project.

It’s easy to achieve if you use master lists, but not the parent list with lookups that aggregates all values for you and you need assign users for the subset only.

After inspecting number of commercial and free tools the only working component I found is “SharePoint Cascading Drop Down List”. It’s very amazing how it works. Check the following blog, describing how to configure the list correctly http://sharepointsircar.blogspot.com/2009/02/cascading-dropdownlist-custom-field.html. Take into account that their sample is based on several master lists, but you can achieve the same for the aggregated list, as I described before ([Team Projects]), where you use “ProjectName” as parent field, and “Sections” for the child.

Works as a charm! Really amazing work.

First five SharePoint 2010 Tips and Tricks

Published first 5 SharePoint 2010 Tips and Tricks

Tip #65: How to install SharePoint 2010 using local account
Tip #64: How to restore Site Directory template in SharePoint 2010
Tip #63:How to print SharePoint 2010 page without header and navigation
Tip #62: SharePoint 2010 supports folders for KPI values
Tip #61: SharePoint 2010 keeps copy of modified theme

Have something interesting regarding SharePoint 2010 - submit your Trick here

Block diagram - how to choose the SharePoint 2007 topology

I found a really good description of how to choose the SharePoint 2007 topology that is vizualized in block diagram. The full description is in this post "Inside SharePoint Securing External SharePoint Communications"

SharePoint 2007 Farm ports – configuring firewall

Introduction

Depending on the environment you are configuring you might need not only open ports between client and your Web Front End(WFE) servers, but you might have internal topology where your SharePoint servers are separated by layers and are isolated by firewalls. In such situation you need to know the inner-process communication ports and direction, to open the ports on firewall.

The following table describes all ports SharePoint 2007 uses for the communications. Take into account that only 2 posts are used between client PC and WFE (the indenting line). All other ports are for internal and external communications between SharePoint servers.

The advantage of this document is that I summarized all ports together that are described in different documents, and included directions. Microsoft doesn’t provide you the summary info for ports and directions.

SharePoint 2007 Ports

Inter-server communications of SharePoint 2007

Extra -server communications of SharePoint 2007

Firewalls

Depending on you farm design you might require firewall between your farm’s servers. In case of separate networks you should know that one-way trust relationship is required between WFE and  Applications Servers, Application Servers and Database, if they are separated by network.

You need to configure firewall properly for domains and trusts http://support.microsoft.com/kb/179442/

Sources:

• http://go.microsoft.com/fwlink/?LinkId=85533&clcid=0x409 (Visio diagram with ports for the inner-server communications)
• http://blogs.msdn.com/uksharepoint/archive/2009/01/05/sharepoint-ports-proxies-and-protocols-an-overview-of-farm-communications.aspx
• http://blogs.msdn.com/joelo/archive/2007/02/13/protocols-ports-and-firewall-rules.aspx
• http://technet.microsoft.com/en-us/library/cc287966.aspx
• http://technet.microsoft.com/en-us/office/sharepointserver/cc979168.aspx

SharePoint 2007 to SharePoint 2010 Farm Migration. Phase 2 – Security Analysis

Introduction

In the current series I’d like to describe how to analyse the SharePoint Farm and prepare it for the SharePoint 2010 migration. We will review the following 3 sections:

1. Farm Architecture and Configuration Analysis
2. User and Group Analysis (current)
3. Farm Migration

This section describes the user analysis and permission analysis. But first of all – why do we need to analyze users and permissions when we only migrating data?! Can’t our users be migrated automatically?!

The answer is yes and no - users will be migrated automatically, but migration is hardly planning for the sake of migration and usually you are building a new application and trying to fix existing issues. Users, Groups and Permissions are needed to be reorganized and to be fixed before moving content to new environment.

The areas we need to look at are the following:

• number of users and group
• how users are organized in groups
• permissions – users, groups, broken inheritance
• dead users

SharePoint OOTB functionality doesn’t cover all our needs, so we are going to use several 3rd partly tools to gather the necessary information.

Tools

• [free] SharePoint Administration Toolkit (you need to install and to activate “Permission Report” feature manually)
• [free] SharePoint Access Checker
• [free] Bamboo SharePoint Analyzer
• [free] Xavor SharePoint Admin Tool
• [free] SharePoint Work Accelerator Toolkit 2007
• (optional) [commercial] ARK for SharePoint 2007

Additionally, you need to use the following STSADM commands

• enumuser/enumgroups/enumroles
• unlockfgpreport (http://technet.microsoft.com/en-us/library/ee449564.aspx) to unlock how much list items can be enumerated for the Permission Report

There are two approaches to collect required information – using commercial “ARK for SharePoint 2007” reporting tool that covers almost all our needs  or using several free tools to get the same information. We can achieve almost the same via “enumuser/enumgroups/enumroles” command of STSADM, but we need to count the items manually.

The limitation of the majority of free tools is that they don’t provide web-application level information across all site collections. The advantage of “ARK for SharePoint” is that it generates reports for all web applications in our farm.

In this post I’d like to describe the steps of how to get all information without using commercial tools.

Number of Users & Groups

1. Users & Groups number – “Bamboo SharePoint Analyser" –> Farm->Servers->Web Applications->Site Collections –> Web sites and the values are in parentheses for “Users”, “Groups” and “Administrators”
2. Site Administrators – use “Bamboo SharePoint Analyzer” of Central Administration
3. Groups across site collections - use “Xavor SharePoint Admin tool” –> Show Group Security

Users & Groups Association

1. Farm Administrators – Use Central Administration –>Operations-> Update Farm Administrators Group or  “Bamboo SharePoint Analyzer” 
2. Users by Group – ARK for SharePoint provides full info across all web applications. Alternative free solution is to use “Permission Report” tool functionality (Site Settings -> “Broken Inheritance Reports Jobs”) that generates Excel spreadsheet for the Site with the user’s and its groups. 

Permissions

1. Broken inheritance can be found via “Access Checked” tool that shows SharePoint items where permission is broken, but tool doesn’t show what exactly is broken and list of changes. Reports are supported.
2. Broken inheritance Diff can be viewed with “SharePoint Administration Toolkit” and its “Compare Permissions Sets” report that shows the permissions difference between the current and root items, and also the details about permission changes . Reports are supported.
3. User rights – “Check User Access” report of “Access Checker” show the rights for the users across SharePoint elements, including the items where user don’t have access
4. Group rights - “Check Effective Permissions” of “SharePoint Administration Toolkit” shows the items accessible by this group

Unfortunately, all previous tools don’t provide web-applications scope reports and item-level reports. It means that you can’t iterate through all site collections and find the List items or specific pages where user has no access. To get such information use “Xavor SharePoint Admit Tool” that provides reports across web application (but no functionality to save them)

(red – user has no permissions)

Dead Users

When you install and configure the new farm you probably create several test users and groups that should be deleted in the end. Sometimes administrators create such users and then forget to delete them. So, “dead” accounts is a quite common scenario. When you start a new migration you don’t want such users/groups in your new farm and you need to find all of them and delete.

I don’t know any free tools that provide such functionality. And there are only a couple of the commercial tools that allow to do this: DeliverPoint and ControlPoint

Creating the report

The logical outcome of the Security Analysis is the Word document that highlights the security issues, but unfortunately this is not always feasible. Consider the medium farm with 5000 users 300 groups and 400 sites with 30% of broken inheritance. You can physically create the word document but how are you going to analyse the 200 pages document?!

The real Analysis is usually a “multithreaded” task, when you check users’ rights, discuss the grouping with DC admins, fix the broken permissions and etc.

Depending on the content size documenting the following quantative information is recommended:

• Farm Administrators
• Number of users
• Number of groups across web application and per site collection
• Broken inheritance report per site collections and items (depends how much broken items you have)
• Users/AD per Groups (definitely for AD, but depends on number of users)

Unfortunately, it’s hard to define the template for this step, because security analysis is very individual for the farm, and usually you end up with several files – documents describing quantative info, excel spreadsheets with users, groups and permissions, html files describing the broken permission inheritance.

Resume

Security analysis might be a daunting task depending on the level of your permissions customization and user’s assignment to groups. The recommendation is to perform the draft analysis on backup instance where you can experiment with different tools and find all security breaches, and after that fix issues on production.

SharePoint 2010 stuff on SharePoint-Sandbox.com

I finally published my SharePoint 2010 content I discovered for the last 5 months.

Welcome to www.sharepoint-sandbox.com to share and to contribute! New articles and tips are coming

SharePoint 2007 “Meeting Workspace” template limitations

On one of the recent projects we built the monthly based reporting system. We provided users the freedom of choice of the reporting indicators, from the predefined list, and allowed them changing values of selected indicators.

From the very beginning we couldn’t decide which template to use as a baseline for our project and considered using “Meeting Workspaces” (MW). The advantages of them were that you can create number of calendar-driven Workspace sites with custom recurrence. It fitted our conceptual model ideally – you create the content ones and it’s available across all months + you can add custom content for the specific month.

But the reality of Meeting Workspaces is too far from grace, limiting its usage.

First of all, you can’t save Meeting Workspace as  template including the content. As I understand it’s “behaviour-by-design”, because the MW is based on the single huge list, where content is splitted by “InstanceID”. You can’t just save the template for the specific month - SharePoint can’t save only part of your list. Secondly, you can’t set different permissions for the lists per month, due to the same behaviour – it’s the same list not different once. Third, calendar URLs are not “InstanceID” driven and Meeting workspaces don’t provide collaboration behaviour, because when you open the new month the list content is based on active month selection (see this post)

Such limitation changed our decision towards using the Blank site and customized list, saving the site as template to be used for the new period.

I really wonder to know the usage of Meeting Workspace when you can ignore such limitations.

SharePoint 2007 to SharePoint 2010 Farm Migration. Phase 1 – Configuration Analysis

Introduction

In the current series I’d like to describe how to analyse the SharePoint Farm and prepare it for the SharePoint 2010 migration. We will review the following 3 sections:

1. Farm Architecture and Configuration Analysis (current)
2. User and Group Analysis
3. Farm Migration

The first and the most important step in SharePoint 2007 –>SharePoint 2010 migration is the understanding of existing SharePoint environment to get enough information to design the new Farm. 

Let’s review in details the following template that I’m using to document the farm settings and tools that allow to gather all necessary information (we don't describe search, excel and other services settings here)

1. Farm Information
   1. Farm Servers and Services
   2. Web Applications
   3. Content Database and Site Collections
   4. Alternative Access Mapping
   5. Farm Solutions
   6. Enabled Farm Features
   7. Search Settings
2. Site Information
   1. Web Parts
   2. Web.config changes
   3. Site Definitions
   4. Customized & Checked-out Items
3. Sites Topology
   1. Sites
   2. Sites Structure Diagram
   3. Site Collections with Diagrams
4. Issues

1. Farm Information

Content: This section enlists farm servers, components from the farm and general information about farm configuration (email settings)
Tools: STSADM –o “preupgradecheck” (Upgrade Planning Information section for servers and components) & Central Administration to get the mail settings

a) Farm Servers and Services

Content: table with the farm servers and assigned roles
Tool: Use the Central Administration & SharePoint Manager 2007 to get this information

b) Web Applications

Content: List of web applications and its URS
Tool: Central Administration

c) Content Database and Site Collections

Content: Table with the following info – content database name, number of sites, size, list of site collections
Tool: SharePoint Diagnostic tool for the Content DB size; Central Administration and SharePoint Administration Toolkit (Batch Site Manager Solution)

d) Alternative Access Mapping

Content: Table with AAM Internal/External URLs and Zones
Tool: “preupgradecheck” log & SharePoint Diagnostic

e) Farm Solutions

Content: Table with the installed solutions and sites where they are active
Tool: SharePoint Manager 2007 / Bamboo SharePoint Analyser

1.f) Enabled Farm Features

Content: List of enabled features on the farm level
Tool: Central Administration 

1.f) Search settings

Content: SSP settings (servers, databases name, crawling settings)
Tool: Central Administration & SharePoint Diagnostic

2. Site Information

a) Web Parts

Content: list of installed web parts
Tool: Bamboo SharePoint Analyzer

b) Web.config changes

Content: list what was changed in web.config for SharePoint sites
Tool: SharePoint Diagnostic shows web.config for each web application, but developer/admins own the knowledge about the changes

c) Customized & Checked-out Items

Content: list of customized & checked-out items
Tool: SharePoint Designer, choose Sites menu –> Reports –> Shared Content –> Customized Pages / Checked-out items

3. Sites Topology

a) Sites

Content: list of sites
Tool: Site Settings of the root site –> “Site hierarchy” item

a) Site Structure Diagram

Content: diagram of the root site
Tool: SWAT tool –> right mouse click on the site name and Show Site Diagram

4. Sites Topology

• Use “preupgradecheck” log to document all found issues
• use SharePoint Designer Diagnostic tab, to discover the potential issues

Tools

To get the necessary information I recommend to use the following tools:

• Bamboo SharePoint Analyser
• SharePoint Manager 2007
• STSADM –o “preupgradecheck”
• SWAT
• SharePoint Administration Toolkit - SharePoint Diagnostic tool & “Batch Site Manager Solution” feature 
• SharePoint Designer

Resources: http://www.sharepointjoel.com/Lists/Posts/Post.aspx?ID=245

SharePoint 2010 Timer Jobs changes

SharePoint 2010 Timer Jobs undergo changes are in the management and in configuration.

Firstly, the most significant change you will see is that the number of Timer Jobs – new 21 jobs been added. If for an ordinary SharePoint 2007 application we had 39 Timer’s Job, then we have 60 Jobs for SharePoint 2010

Secondly, you are able to edit job properties - setting when the jobs starts via interface (not STSADM) and start job immediately.

Thirdly, UI interaction of Timer Jobs management are improved – jobs are sorted by status, able to modify the definitions from any screen.

You can read detailed overview of Timer Job changes on our www.SharePoint-SandBox.com portal.

SharePoint Tips #50-#60

Finally reached the 60th SharePoint Tip. I wish I published 120th :) but quite busy in these days and publishin new SharePoint 2010 matherials takes a lot of time.

Tip #60: How to change the Spell Checker Language        
Tip #59: How to set the default duration for the new Calendar event      
Tip #58: How to open a list in Edit Mode       
Tip #57: Don't use Farm Admin users for testing       
Tip #56: Don't upload the same file multiple times for testing purposes
Tip #55: Disabled "Require Checkout" impacts on Content DB growth       
Tip #54: Use Web Garden to add resilience and enhance performance       
Tip #53: Don't rely on default settings to crawl a particular WFE        
Tip #52: STSADM doesn't support import subsite as root site
Tip #51: Do you know that "read-only" Database feature is trimming UI edit functionality?!
Tip #50: FBA crawling doesn't support Wizard-based and Dynamic Pages with AJAX 

SharePoint 2010 official resources

In these days SharePoint 2010 is announcing at SharePoint Conference 2009. There are a lot of materials being prepared by MVPs and others,who had access to the Tech Preview of the SharePoint 2010 last 5 months and we will start releasing new info after conference finishes. But for now, I’d like to summarize the info officially published by MS

1. SP2010 TechNet Center http://technet.microsoft.com/en-au/sharepoint/ee263917.aspx
2. SP2010 MSDN Center http://msdn.microsoft.com/en-au/sharepoint/ee514561.aspx
3. Posters http://bink.nu/news/sharepoint-2010-documentation.aspx
4. SharePoint 2010 Forum http://social.technet.microsoft.com/Forums/en-US/category/sharepoint2010

And follow us in tweeter via #spc09 hashtag (I’m using http://twitterfall.com/ to monitor twitter events)

SharePoint 2007 on Windows Server 2008 R2

When you consider using Windows Server 2008 R2 for your SharePoint 2007 platform you need to put additional effort to configure and install SharePoint properly.

There are a few actions you need to undertake for the successful deployment of SharePoint on R2.

1) You can only install SharePoint 2007 SP2 on R2 version. Unfortunately, MS doesn’t provide SharePoint 2007 SP2 slipstream package, so you need to create one by yourself (Take into account, that you need to delete everything from the “\Updates” folder in case of using SP1 slipstream)

2) After you complete installation you will find that whenever you are trying to open the Central Administration the login and password are requested, regardless of your efforts to add Central Administration to the Trust Zone and set the option to logon automatically with the user’s name/password. This is a new change in security of Vista and Windows 7. To remove the login prompt you need to add a key “AuthForwardServerList” to the windows registry. Read the following guidelines.

3) When you create a new site and try to open it you can find the situation when login dialog box is prompted, but you end up with the white screen after you entering credentials three times. Usually, you navigate to c:\Windows\System32\drivers\etc\ folder, open the “hosts” file and add an alias for your site to solve this problem. But, under some circumstances this wont help you in R2, and moreover, your can’t open the site via “localhost” name. The solution is to add the alias not only to host file, but in “lmhosts.sam” as well

SharePoint 2007 32bit->64bit migration Strategy

As you know, Microsoft will release SharePoint 2010 public version soon, and there is no space for 32bit versions anymore. All server stuff come in 64bits only – Windows Server 2008 R2, SharePoint 2010 and etc, so, we need to be ready for this.

I recommend to watch New Zealand TechEd session “Upgrade Planning and Guidance OFC306” that describes how to plan you 32bit –> 63 bit migration for SharePoint Farm

Top Content Management sites

Found a nice description of top Content Management Sites http://aiim.typepad.com/aiim_blog/2009/03/just-the-facts-please-top-content-management-sites.html

All are really good, except … where I can’t find the way to subscribe on RSS updates. It’s a bit awkward – the site is about Information Management and e-Discovery, but no way to find the connections to the information.

Some sites publish information in the aspects of SharePoint.

Interesting facts about ECM systems

Some interesting facts about ECM systems:

1. only 18% percent of ECM projects ever make it into full production
2. for every “current” document in an Exchange Server or on a file share, there are 14 other older, legacy documents in PST files, laptop hard drives, local archives, etc
3. The average user creates or receives between 100 to 300 electronic documents per day
4. if classification process requires five seconds or longer (including referring to the schedule) end-users will ignore request to classify

Source

Use the right data for SharePoint testing

Thanks to Dario Mratovich, Microsoft Consultant, for his awesome recommendations of preparing the testing data.  Citing the section from his “Capacity Planning Testing for SharePoint 2007”

Make sure you have adequate sample data. This tends to be a very common stumbling block – sites are built with only a tiny percentage of content that production will have. Not enough sites, not enough content, not exercising a broad enough sampling of your dataset, not enough users – these can all fatally influence your test results.

Another common problem is having enough content for a reasonable search corpus. What many people try doing is uploading the same document many times – sometimes hundreds or thousands of times – and think that if it has a different file name then it will be okay. Unfortunately in that scenario the search duplicate process can start taking significantly longer than it otherwise normally would, so this too can unfairly reduce your query throughput.

A document that is uploaded multiple times will affect the way that SharePoint performs duplicate detection: where SharePoint’s search calculates a hash based on the contents of a document – it doesn’t look at the filename! So uploading a document 30,000, even with a different filename, will cause the search retrieval to become slower and slower as SharePoint tries to resolve duplicate documents.

You will need tools in all likelihood to populate sample data. Some tools you can start with are on CodePlex at http://www.codeplex.com/sptdatapop. You will probably end up writing additional tools for other data population tasks, or possibly to work in combination with these tools.

Using PowerShell for scripting the creating of objects and data for testing is also very useful.

Pillars of a Collaboration Project

 Michael Porter and Alan Weintraub published really good article regarding four pillars of collaboration project.

When looking to implement a collaboration project, it is critical to understand that there are four pillars and that a single pillar alone will not address the requirements of your project. These four pillars are -  Messaging, Real Time Collaboration, Team Collaboration, and Social Networking. In you projectss you often take capabilities from multiple pillars to address all of your needs.

1. Messaging: enables teams and individuals to communicate in a formal, auditable manner.
2. Real Time Collaboration: focused on ad-hoc communications providing capability to initiat a real-time conversation between one or more individuals
3. Team Collaboration: provides the content management services that collect, store, and manage the documents that are pertinent to the team’s activity. 
4. Social Networking: provides a way for teams and individuals to share knowledge not only within the team, but to be able to reach out to an extended group for their input and feedback

Presenting on Moscow SharePoint UG, Russia

Will be presenting on Moscow SharePoint UG in Russia this September 14th, about "SharePoint Farm Deployment and Configuration - Recommendations"

This is the third version of my whitepaper I was working on this year. In this session I will put attention on farm optimization and provide guidelines how to indentify the right values for the SQL Server settings and Farm Size settings in terms of RPS (requiest per seconds).

Registration is necessary. More details are there http://sharepoint.su/Lists/Posts/Post.aspx?ID=22 (on russian). The place is the same - Microsoft office in Krylatskoe. 

SharePoint 2007 limitations for the ECM solutions

Traditional Enterprise Content Management (ECM) solutions are characterized by number of factors, such as cross-team collaborations and document management. Most of them are aligned with the SharePoint functionality, but  unfortunately, in most cases, SharePoint does not prove to be a solution that can meet all of the organization’s ECM requirements.

In this post I'd like to start collecting the scenarios that hits the SharePoint architecture design limits and lack of functionality on the way to meet ECM requirements. If you know more scenarios, please share via comments.

Requirement 1: Large ECM scenarios can require to store millions documents in the single folder or document repository, including deep, nested folding structure to accommodate all of the organization’s documents.
Issue: SharePoint 2007 has limitation of sub-folders support, number of documents stored in list libraries, crawling limitations and etc. 
Solution:
- SharePoint 2010 solves such issues, and you can store millions items in folders and subfolders without impact on quering data, rendering and search
- "ECM systems" product, because there is no OOTB support in SharePoint. We can emulate sub-folders using meta-data and customizing views, but nothing helps us to have millions items per library, because performance will degrade drastically on showing thousands documents, not millions.

Requirement 2: Certified state of record management system. Government and military organizations requires the standard and classified record management compliance with U.S. government's DOD 5015.2 certification.
Issue: SharePoint 2007 doesn't support classified records compliance (for DOD 5015.2 Chapter 4).
Solution: "ECM systems" product, because there is only general records support. Chapter 2 via The DOD 5015.2 Resource Kit in SharePoint.

Requirement 3: Sites and document repository need to be classified, encrypted and archived
Issue: List item limitations hinders the capabilities of Record Center feature of MOSS, and no support for reviewing archived content and restoring sites from archive.
Solution: n/a, albeit "OpenText ECM Suite" tool provides some features enhancing search and archiving functionality of Record Center.

Requirement 4: Ability to render, search and manipulate print streams such as AFP, Xerox Metacode formats, PCL
Issue: SharePoint supports rendering the mainstream formants only, such as as PDF, Office documents and etc.
Solution: n/a. Custom solution, based on the print stream converters (Crawford and others)

Requirement 5: Ability to collaborate without bounds 
Issue: SharePoint 2007 and SharePoint 2010 collaboration is limited to site collections. Albeit SharePoint 2010 provides additional features to Collaboration scenarios, it still can't solve cross-collections collaboration issue.
Solution: n/a due to architectural design, albeit you can use 3rd part components to achive some collaborations [1], [2]

Update: According to Forrester Research, only 18% percent of ECM projects ever make it into full production (a moment to point out the obvious: 82% do NOT reach full production). Mark Diamond published 5 reasons when ECM may never be deployed

3 suggestions to start with document management practices on SharePoint?

They pointed 3 critical aspects when developing document management for SharePoint:

1. Create taxonomy, first describing the high-level content type and associated attributes that the other content types can inherit from
2. Define the Information Architecture; this is the layouts for the sites, document libraries, and document folders. This is critical as SharePoint has some performance limitations that need to be considered during the design phase
3. Develop a governance plan that addresses content ownership, process, and security rights

These steps should be a part of your SharePoint project documentation

http://www.aiim.org/infonomics/sharepoint-ask-expert.aspx