The SharePoint Farmer's Almanac : Error Messages, How Do I

Fixing MOSS Search

Shane — Mon, 13 Apr 2009 14:56:13 GMT

Had a client today (last week now) who broke search all the way. And in their attempts to straighten it out they changed some pieces that weren't broken. Then while they were in the process of trying to put it all back together I called and said let me at it so they just stopped. Needless to say I picked up the farm in an odd state or more exactly Search was dead.

So the first step always when Search is broke is to go to the SSP Admin and check out things.

Open the SSP Administration page
Click on Search Administration and see what it has to say. (if you don't see Search Administration this means you have not installed the infrastructure update. I would highly recommend at a minimum you have that installed. Get the latest update install guide here)

When I opened the page I saw a Crawl status of Error. That is about worthless.

That is pretty much as generic as they come. You get the same Error when the server is on fire as you do when there is small hiccup. So a much better thing to do is:

Go back to the SSP administration page
Click on Search Settings (which is what we used pre infrastructure update)

This page does a much better job of giving you tangible errors. Here is what I got:

Error: An indexer is not assigned to the Shared Services Provider 'SharedServices1'.

Link to: Configure an indexer and a search database for this Shared Services Provider

Well that is fixable but how did they end up like this? They stopped the Indexing service in the farm by:

Go to Central Admin
Click on Operations
Click Services on Server
They choose their Index server
Then clicked Stop to the right of Office SharePoint Search Service

This doesn't just stop the service. This actually removes the service completely. This also removes the Index server from any SSP configured to use it. Now if you did want to just start and stop the service there is a way to do this:

Open a command prompt
Type net stop osearch and press enter
Type net start osearch and press enter

This will cycle the search service. Usually the only time you need to do something like this is after installing a new ifilter but sometimes it makes you feel better to give it a shot and see if that helps your problem. I do it more often than I should just for that reason.

Back to the task at hand clearing up that error! I double checked and they had already reconfigured the Office SharePoint Search Service on the Index server so all I need to do is go back to the Index server and re-associate the indexer.

From Central Administration click on Shared Services Administration from the left hand side of the page.
Hover over the SSP name, click the drop down arrow and click Edit properties

Scroll to the bottom of the page and select your Index server from the Index Server dropdown. If you see No Indexers in red you need to go back to your Services on Server and make sure you have the Office SharePoint Search service started and configured for the Index role.
Confirm that you have the correct index location. Usually the C: drive is less than ideal.
Click Ok

The SSP is now configured with an Indexer. Let's go make sure Search is happy.

Now click on the Shared Services Provider name to open the SSP admin site.
Click Search Settings

Don't be surprised if you get this error:

Error: The search service is currently offline. Visit the Services on Server page in SharePoint Central Administration to verify whether the service is enabled. This might also be because an indexer move is in progress.

Typically this is because the wheels of Search can move slowly. I have seen this error come up for 10 minutes or so in some farms. What Search is really telling you is it is busy getting the index and the database ready to go so you can start indexing. Be patient grass hopper. At the client this was gone after about 2 minutes.

Once I was able to get to a happy Search Settings page I went ahead and reset the Index back to zero. Not always necessary but they had 33,000 items in the index and 140,000 or errors. I thought better to start everything back to 0.

In order to reset the Index.

From the SSP admin screen click Search Administration
From the left hand column (quick launch for those who know terminology) click Reset all crawled content
Select Deactivate search alerts during reset
Click Reset now

Now you have a completely blank index. Why did we choose to deactivate search alerts? This is to keep from annoying the users. We don't want them all to get new alerts when new content is discovered when we recrawl in a minute. Once the index is back to normal we will re enable the alerts for them.

Ok so now the next step should be doing a full crawl. So let's try that.

From your SSP Administration home page click Search Administration
From the Quick Launch bar (on the left) click Content Sources
Hover over your Content Source, click the drop down arrow, and select full crawl
Now go back to the home page of Search Administration and watch to see if the crawl is running

Unfortunately in our case after about a minute I was left with 0 items in the index and 3 errors. After checking the errors I got Access Denied. L If you haven't done any monkeying around with changing your default content access account then it should have been automatically granted full access to your content source. You can confirm this by checking your Policy for web application in Central administration. If you forget how to do that check this blog post for a reminder. http://msmvps.com/blogs/shane/archive/2007/01/21/become-administrator-of-the-entire-web-application.aspx

If that checked out ok then the next thing I would check is to make sure your web application is set to integrated authentication and not basic authentication. MOSS will not pass basic authentication by default. So if you changed your web application from integrated to basic, so people users don't have to enter their domain for example, then you need to setup a custom crawl rule to pass basic authentication.

From your SSP Administration home page click Search Administration
In the Quick Launch bar click Crawl rules
Click New Crawl Rule
For path enter your web app URL ex: http://portal.company.com/*
For Crawl Configuration select Include all items in this path
For Specify Authentication select Specify a different content access account
Now fill in username and password remembering your domain\username form. I would recommended using your normal search account as you know it already has read access to the content.
Key step de-select the box to Do not allow Basic Authentication
Now do a full crawl. Also, remember if you have multiple web apps you may need more than one of this rules.

For the client this was not this issue but it is an important and often over looked troubleshooting step so I thought throwing it in here would be helpful.

The next thing I take a look at is the dreaded loopback fix. I showed this one to Todd Klindt one time and he wrote a nice post on the issue and a like to the KB for fixing it. http://www.toddklindt.com/blog/Lists/Posts/Post.aspx?ID=107 It is all but a guarantee these days if you have the WFE and Index role on the same server you are going to need to do this. A lot of farms have ran fine for a long time and just recently they have started requiring it. Must have been a Windows update that is causing this to be needed more but I haven't identified it. Another note even though this fix is only listed as applying to Windows 2003 it also applies to Windows 2008, had a different client need it last week. J

Loopback fix in and the server rebooted I tried another Full crawl. Success! Seems this was the root of their issues but as is often the case that happens to all of us, trying to fix it only made the problem worse. LOL

Don't forget to re-enable those search alerts.

From your SSP administration home page click Search Administration
In the System Status section in the center of the page click Search alerts status Enable

Another troubleshooting step I skipped, because the client had already done it was resetting search permissions. Read the blog post John Ross did summing up the steps to get permissions back on the up and up for the Search Service. http://www.sharepoint911.com/blogs/john/archive/2009/04/03/change-to-group-policy-broke-sharepoint-search-–-thanks-conficker-scare.aspx

Something I learned that was new

I am guessing since I didn't realize this is an option (or more probably I knew and forgot) you probably didn't either. So run stsadm –o help like below and take a look at the output.

Use Stsadm.exe from the 12 hive (c:\program files\common files\Microsoft shared\web server extensions\12\). Actually 12\bin to be exact.

C:\ >stsadm -help osearch

stsadm -o osearch

[-action <list|start|stop|showdefaultsspadmin>] required parameters for 'start' (if not already set): role, farmcontactemail, service credentials

[-f (suppress prompts)]

[-role <Index|Query|IndexQuery>]

[-farmcontactemail <email>]

[-farmperformancelevel <Reduced|PartlyReduced|Maximum>]

[-farmserviceaccount <DOMAIN\name> (service credentials)]

[-farmservicepassword <password>]

[-defaultindexlocation <directory>]

[-propagationlocation <directory>]

[-cleansearchdatabase <true|false>]

[-ssp <ssp name>] required parameter for 'cleansearchdatabase'

So really very similar to the options you have available to you from the GUI. The reason I used it was one of the Query servers was stuck in the starting state. In the GUI there is no stop until the service gets too started, not even a reboot will help. With stsadm you can do a stop and get out of the perpetual starting. J A very helpful trick.

If you are still fighting with Search here are couple of other Search troubleshooting things I wrote a while back

Hope you enjoy what feels like a small book

Shane – SharePoint Consulting

InfoPath Form displayed on anonymous site causes login prompt

Shane — Tue, 13 Jan 2009 01:54:24 GMT

Had this one pop up for a second time today so thought I would blog it this time.

Client created a public website for baseball signups. It was a simply publishing portal with anonymous enabled. He then created an InfoPath form and set it up to generate an email with the info from the form. He published this form to a form library using Forms Services and then embedded the form using a page viewer web part on a new ASPX page so people could just navigate and fill out the page.

The problem – the site would pop up an authentication box every time an anonymous person would visit the site. Even stranger if someone would get the authentication box and then enter proper credentials for the remainder of the day anonymous access worked fine. Then each morning back to the same problem. ODD.

Turns out the issue was the lockdown feature used with the publishing portal. Not everyone realizes it but when you create a site collection using the publishing portal there is a hidden feature that fires called ViewFormPagesLockdown. This feature, among other things, sets it so /_layouts directory is not available to the anonymous user. For more details see the ECM Team blog post. So we simply deactivated the feature and reset the anonymous permissions and life was good again.

One thing to pay particular attention to with this feature every time you toggle this feature on or off you need to disable and enable anonymous access for the new settings to take effect. That part can drive you bonkers.

Shane – SharePoint help

Windows Server 2008 WFE will not allow large file uploads

Shane — Wed, 17 Dec 2008 18:51:35 GMT

I had a client a couple weeks ago who upgraded from SPS 2003 to MOSS 2007 using the database attach method. This means the MOSS farm was built on new hardware which we chose Windows Server 2008 for. This was a very challenging upgrade for several reasons but primarly because the 2003 site database was in bad shape. Once we worked through the issues and got things up we noticed strange behavior. When uploading a large file (anything larger than 28 MB) the browser would instantly come back with a 404 error. So our first thought was check the three normal settings for large uploads.

Central Admin > Application Management > General web application settings. By default this is 50 MB. You can increase to 2 GB.
Then you can go into IIS. Find your web application and go to properties. Then change the IIS timeout from 120 seconds to a much larger setting.
Upload.aspx is an application page. Application pages have their own web.config which controls their timeout. The default for these pages is 360 seconds. You need to increase this.

All of these settings apply whether you are using w2k3 or w2k8 and are covered in this kb http://support.microsoft.com/kb/925083

We made all of those changes and no change in behavior. And besides we didn't have a timeout issue because on a 27 MB file is processed for a few seconds and then uploaded no problem. On a 30 MB or greater file it failed instantly.

Todd Klindt to the rescue he pointed me at this KB944981 - You cannot upload files that are larger than 28 MB on a Windows Server 2008-based computer that is running Windows SharePoint Services 3.0

So I gave it a try and it actually made things worse. What the heck? Well then I reread it. They tell you to make a change to the web.config and say just put the change in the <configuration> section. WRONG! Well kind of. You need to make the change in the <configuration> section but it has to be after the </configSections> tag. So I recommend you paste their change between </configSections> and the <SharePoint> tag.

Now everything works great. Do note their change only allows you to upload files with a size of 50 MB. If you want larger you will need to increase the maxAllowedContentLength=.

If you are going to be playing with Windows Server 2008 and SharePoint I recommend you go poke around the SharePoint and w2k8 resource center at http://technet.microsoft.com/en-us/office/sharepointserver/bb735844.aspx. Thanks to Emily Schroeder for the tweet on that page which she runs. J

Even better yet Todd and I have some fun new stuff coming on Windows Server 2008 and 64 bit before the end of the year.

Shane – SharePoint Consulting

Whole farm is down because timer jobs are not running

Shane — Thu, 09 Oct 2008 19:50:04 GMT

One of my clients this week managed to take his entire farm offline this week by upsetting the timer service. First a little background – currently they are scrambling to get SharePoint back to a happy state. Why? Well, as happens with lots of customers, SharePoint is too successful. When we originally setup their farm and upgraded from SPS2003 to MOSS 2007 they had about 20 GB of content that was growing at a very controlled pace. Fast forward a little more than a year and their content database is about 320 GB. YIKES! Even scarier most of their data is in one site collection. This is bad, very bad! Typical guidance is your content databases should be less than 100 GB.

Part of this growth has forced some moving of the databases to different drives and a database restore to deal with another issue. Well, anytime you want to move SharePoint databases around you should run the command stsadm –o preparetomove as documented by Cory Burns in the post Detaching databases in MOSS. If you didn't you will start getting sync errors once an hour such as:

Failure trying to synch web application 09a21da5-4485-4b00-8268-772aea7fea12, ContentDB 65301403-c277-4b4c-ad5a-e822572d10ea: A duplicate site ID 3b3a4372-aa91-4e0c-ba57-2567958d81bb(http://portal/sites/test1) was found. This might be caused by restoring a content database from one server farm into a different server farm without first removing the original database and then running stsadm -o preparetomove. If this is the cause, the stsadm -o preparetomove command can be used with the -OldContentDB command line option to resolve this issue.

Cory then goes on how to fix it using stsadm –o sync. This is where my client was. He ran this command but for some reason (possible him specifying the wrong switches and accidently deleting a content db) the command hung up for a long period of time, and the portal users were unable to access the environment. So he killed the stsadm process. From that point all hell broke loose.

For several hours they attempted a lot of fixes found on the web. One of the fixes had them rename the folder located at C:\Documents and Settings\All Users\Application Data\Microsoft\SharePoint\Config\<guid>\. This was a bad option. The folder contains XML files for all of the timer job definitions that need to be ran and the idea was renaming the folder would cause SharePoint to create a new empty copy of the folder and then it could start creating the xml files again and get back to work. Nope, that isn't how it works. What they needed to do was delete all of the XML files and leave the folder alone. Then when they restarted the timer service the proper XML files would have magically reappeared.

Hope this helps you

Shane

SharePoint Consulting

Another Error Message – Access Denied on Profile Import

Shane — Mon, 30 Jun 2008 15:56:31 GMT

I noticed the error message below repeatedly in event viewer and realized it was happening every time a profile import was happening.

Event Type: Error

Event Source: Office SharePoint Server

Event Category: Office Server General

Event ID: 7888

Date: 6/27/2008

Time: 11:37:17 AM

User: N/A

Computer: ServerName

Description:

A runtime exception was detected. Details follow.

Message: Access Denied! Only site admin can access Data Source object from user profile DB.

Techinal Details:

System.UnauthorizedAccessException: Access Denied! Only site admin can access Data Source object from user profile DB.

at Microsoft.Office.Server.UserProfiles.SRPSite.AdminCheck(String message)

at Microsoft.Office.Server.UserProfiles.DataSource._LoadDataSourceDef(IDataRecord rec)

at Microsoft.Office.Server.UserProfiles.DataSource._LoadDataSourceDef(String strDSName)

at Microsoft.Office.Server.UserProfiles.DataSource..ctor(SRPSite site, Boolean fAllowEveryoneRead)

at Microsoft.Office.Server.UserProfiles.DataSource..ctor(SRPSite site)

at Microsoft.Office.Server.UserProfiles.UserProfileConfigManager.GetDataSource()

at Microsoft.Office.Server.UserProfiles.BDCConnector.RefreshConfiguration(String sspName)

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Turns out that on this server they had a least privileged install done. In this case you need to make sure that within the SSP you have granted your default content access account and your SSP application pool account the manage profile permission. If you need help with that check out this article I wrote on SSPs and their rights. http://msmvps.com/blogs/shane/archive/2007/08/06/give-a-user-access-to-the-ssp.aspx

Shane – SharePoint Consulting

A very common prescan problem that stops you from doing your upgrade

Shane — Sun, 15 Jul 2007 19:30:00 GMT

I have gotten the same email from several people and the fix has been the same just about every time so I thought I would publish the conversation here. Hopefully some searching of the web will help save a couple of email stamps. ;)

Mr. Pugglesworth: What should I do if I want to find out if my SharePoint (WSS v2 or SPS 2003) is ready to be upgraded?

Shane: Use my post here to Download Prescan.exe and then use Joel Oleson's post for the exact syntax.

Mr. Pugglesworth: I ran prescan.exe and then I get these error messages in my prescan log. Can you help?

07/05/2007 15:16:32 Skipping virtual server: http://portal.company.com/. Server

state = NeedUpgrade. Most likely this virtual server is not extended

with WSS v2.

07/05/2007 15:16:32 Scan finished without failure.

07/05/2007 15:16:32

===============================Logs===============================

07/05/2007 15:16:32 Log file:

C:\DOCUME~1\puggle\LOCALS~1\Temp\3\PreupgradeReport_633192453889379947_L

og.txt

07/05/2007 15:16:32 Summary file:

C:\DOCUME~1\puggle\LOCALS~1\Temp\3\PreupgradeReport_633192453889379947_S

ummary.xml

07/05/2007 15:16:32

==============================Totals==============================

07/05/2007 15:16:32 Number of sites skipped (already scanned): 0

07/05/2007 15:16:32 Number of sites scanned: 0

07/05/2007 15:16:32 Number of broken sites: 0

07/05/2007 15:16:32 Number of webs scanned: 0

07/05/2007 15:16:32 Number of broken webs: 0

07/05/2007 15:16:32 Number of webs using custom template: 0

07/05/2007 15:16:32 Number of pages scanned: 0

07/05/2007 15:16:32 Number of unghosted pages: 0

Shane: Usually that message only shows up when WSS truly isn't installed or your virtual server didn't get upgraded after you installed WSS SP2. Try this to confirm my suspicions:

Go to v2 central admin. Then click on Windows SharePoint Service on the left:

Now click on Configure virtual server settings.

Now please take a screenshot of the screen that looks like this and send it back to me.

This will help me tell you what to do next.

Mr. Pugglesworth: Here is what I get. Please help. :)

Shane: You have some form of this problem http://msmvps.com/blogs/shane/archive/2005/11/02/74038.aspx

Essentially, you need to run stsadm.exe –o upgrade –forceupgrade from the 60 hive before you will be able to do an upgrade to MOSS. While this command is running your SharePoint will be unavailable so plan accordingly. Also, please make sure you have a backup before you start running stsadm commands. They are safe but very powerful.

The End

Hopefully this will help you out also. If nothing else I don't have to have this email conversation too many more times. ;)

Shane Young – SharePoint Help

Problems with licensing MOSS

Shane — Tue, 29 May 2007 15:11:56 GMT

So today had someone who had used the 180 day trial key of MOSS to its end. Apparently their key expired while we were all grilling out this weekend whoops.

Well, no problem. Just use my blog article How do I find out when my MOSS trial expires? And it has a section on "How to update from MOSS trial to MOSS Standard or Enterprise". Crisis averted.

Not so fast. She was now getting the error message:

Your license conversion operation failed. Check the error logs for details.

Upon looking in Event Viewer (eventvwr) she was getting

Event Type: Error

Event Source: Windows SharePoint Services 3

Event Category: Timer

Event ID: 6398

Date: 5/29/2007

Time: 8:36:17 AM

User: N/A

Computer: SERVER1

Description:

The Execute method of job definition Microsoft.Office.Server.Administration.LicensingConversionJob (ID b535873a-5ff6-434c-b694-6e19eb40da4e) threw an exception. More information is included below.

Arithmetic operation resulted in an overflow.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Interesting…. Upon further examination it seemed that her collaboration portal was working fine but her publishing portal still thought it was using an expired trial edition. If you checked Central Administration (check above article if you need help doing this) it said everything was fine.

So how do we fix this horrible problem? IISRESET.exe

I am going to do some more digging to see if I can find out why or what happened but best I can tell all is well.

Shane Young – SharePoint Help