Now when I think about it, I think that the hardest part about this article is actually starting writing it. To blog or not – that is always a question. Sharing is a great thing to do but our daily business (and evening trainings – greetings to all of you crazy SPR12 dudes out there) are often not allowing us to do anything besides that. So, let’s start!

Search is an important thing. A biggie actually. Can you imagine your SharePoint deployment without search? I can’t! And I am not talking here only about search n’ find using our wonderful search centers, a lot of great solutions are based on search API as well. If you can make few compromises, it is probably the fastest thing in SharePoint world that will give you a consolidated result set from all over the farm(s). That brings us to the point: search must be fast, reliable, available and returning as fresh results as possible. How are we going to achieve this? Let’s make a scenario and some assumptions.
A company (no, not that well known cycle company – let’s just call it a Firm or The Firm) just upgraded their farm from SharePoint Foundation 2010 to SharePoint 2010 and wants to extend a functionality using enterprise search. Here are the requirements:

• The environment consists of a farm containing 5 million items but as SharePoint is now a strategic collaboration product, predicted content growth rate is 75% per year. The solution should support all requirements for next 3 years
• All search components should be high available
• Indexing components should be placed on dedicated hardware in order not to degrade performance on any other SharePoint component

These three statements are giving us already lot of information to start with planning. Not all required info but enough for an example. In order to do the proper planning, we need to get familiar with components of search itself. So, here are the ingredients:

• A service application
• Administration (Admin component, Admin DB)
• Crawl (Crawl component(s), Crawl DB(s))
• Query (Query component(s), Index partition, Property DB(s), Query Processor)
• Query and Site Settings service
• A service application proxy
• A can of coffee
• A bit of patience 😉
• And more patience for optimizing the damn thing 🙂

Service Application

A service application is a part of a new architecture introduced in SharePoint 2010. Service applications replace old and boring SharePoint 2007 Shared Service Providers (SSPs) and offer us a whole new world of flexibility. Yes, search is actually one of them. Creation of this fellow here will allow us to pack it in appropriate Application Pool and create appropriate databases for a start. We’ll cover this in a minute.

Administration

(Search) Administration is built upon two components: Admin Component and Admin Database. Admin component does various tasks like processing configuration changes and actions of the search admin. Basically, if this fellow here is not available, you won’t be able to do any configuration tasks on your Search Service Application, not through Central Administration UI or through PowerShell. And it is a kind of pity because this is the only non-redundant search component. This means, you can have only one instance per Search Server Application and if it dies… Well, for a start, you won’t be able to do any admin tasks on this particular SSA. Good news is, you can move the Admin component to another server using following few PowerShell lines:

$spSearchApplication = Get-SPEnterpriseSearchServiceApplication
$spSearchAdmin = Get-SPEnterpriseSearchAdministrationComponent -SearchApplication $spSearchApplication
Set-SPEnterpriseSearchAdministrationComponent $spSearchApplication -SearchServerInstance -Force

So, in a way, you really don’t need a redundancy here because if this guy here fails, you will eventually lose your administration functionality but farm, along with the search, will keep its functionality. And in such a case, it is easy to move it to a new server.

We have a completely another thing with Admin Database. If this one is not available, you will lose your complete search capability. This database contains following:

• Search service application configuration
• Component configuration
• Scope definitions
• Search logging
• ACLs
• Crawled property list

So basically, without this one, you’re out of business. Nevertheless, this database can be mirrored so keep it in mind in your HA design.

Crawl

As the name says, crawl topology is there – well, to crawl the content. It is made of one or more crawl components and one or more crawl databases. Number of crawl components and crawl databases does not have to be identical so you can for example have two crawl components for redundancy and one crawl database. But you will need to have at least one crawl component per crawl database.

Crawl component runs in MSSearch.exe process which is actually a Windows Service (“SharePoint Server Search 14”). It does various tasks like monitoring crawl status, indexing documents, link discovery and processing and status update for master crawl component. And of course, partitions index and propagates it to query partitions. Important keyword here: a master crawl component. You will always have one. This is a logical role that will be automatically taken by first crawl component in your crawl topology. It is basically a driver of a crawling process, does a very few tasks like updating and reporting of the current crawl state. It manages assignments of hosts to Crawl DBs, initiates crawl fail over in case one of crawlers dies, terminates crawl and completes post crawl activities. So, if you experienced never-ending crawl process it is most probably because this guy here died and didn’t manage to fail over properly.

Crawl databases are associated with the crawl components and are responsible for storing the crawl queue, anchor and social distance and crawl logs. Crawl databases are pretty demanding in terms of IOPS. Recommendation here is to place them on separate spindles that provide 3,5K – 7K IOPS, if possible in separate SQL instance as well. If you want them redundant, you can mirror them. They can grow pretty big and the main reason is the crawl log. SharePoint will trim the crawl log itself but please bear in mind that default interval is set to 90 days. Depending on your environment you would want to tweak this:

$spSearchApplication = Get-SPEnterpriseSearchServiceApplication
$spSearchApplication.CrawlLogCleanupIntervalInDays=30
$spSearchApplication.Update()

In terms of capacity planning, you should have one crawl database per 25 million items.

Query

Query Topology is responsible for processing search requests and returning results to an end user. The implementation of query topology in SharePoint 2010 is way more flexible and effective as the one in SharePoint 2007. So, what changed? First of all, query topology consists of Index Partition(s), Query Component(s) and Property Database(s).

Index Partitions are something new to SharePoint 2010 and they represent a big improvement in terms of scalability and performance. In SharePoint 2007 we could have only one search index per Shared Service Provider (SSP). SharePoint 2010 offers us a possibility to partition a search index and spread it over different query servers. Not only that we are gaining on high availability here but on performance as well. For example, if we would have three Index Partitions distributed over few different Query Servers, it would mean that each of these Servers would have a subset of a whole search index. Query Processor would send a query to all of Query Components and you would have now three guys searching through smaller index subsets instead having only one poor guy searching through whole search index. If you have mirrored Index Partitions, Query Processor distributes requests using round robin mechanism. All of the above mentioned leads to a conclusion that we can have an enormous performance improvement if search components are properly scaled and placed. Please keep in mind – Microsoft haves a (soft) limit here: one index partition per 10 million items.

An Index Partition can be made of one or more Query Components. Basically, a Query Component processes the query results for its part of index. For redundancy, it can be mirrored to another server. A mirror can be active or failover only. Requests are evenly distributed over active Query Components via round robin, as mentioned earlier. Failover mirrors will be used in two scenarios. If all active Query Components fail, the failover mirror will become active and start accepting requests. Maybe less known fact is that failover mirrors will always be used when a Search Service Application backup is running, in order to remove load caused by backup process from active components.

The Property Database can be associated with one or more index partitions. Property Databases will support around 50 million items so if your search index is about to support more than that, you will have to add another Property Database to your Query Topology. Do not keep them on same spindle as Crawl Databases as this can blow up your performance. You can keep them together with Admin DB if you like.

Query Processor

Query Processor handles search requests. It is basically a running instance of a Query and Site Settings service. You can run this one on any server you like (as opposed to SharePoint 2007 where it was running on all Web Frontends) and it is definitely recommended to run it on more than one server in order to achieve a high availability. When a user executes a search request, a query processor is located through Enterprise Search Service Application Proxy. If you have more than one Query Processor, the Service Application Proxy will round-robin between the instances.

Please bear in mind that Query Processor runs in its own IIS Web Application and own Application Pool as well so you will have to count on additional RAM requirements. Regarding RAM requirements, a recommendation is to have enough available memory to cache security descriptor table from Administration Database.

Service Application Proxy

A Service Application Proxy is a link between Service Applications and their consumers (a Web Application for example). They are not to be confused with WCF proxies as they are not the same thing. Service Application Proxies are associated to Web Applications through defined proxy groups or individually, through a “custom” proxy group. In some cases, they provide additional attributes which define in which way Service Applications and consumers are going to interact.

A can of coffee

Currently, my favorite is Pike Place Roast. Since I am back from Seattle, this one keeps me awake for my training, hands on labs and study hours.

Optimization

I am not going to go into scaling here as I will go more deep into this topic in my next article. For small implementations, defaults will be good in most cases. However, big implementations require some additional tweaking. I’ll try to list most important ones.

Admin DB

Database Server hosting Admin DB should have enough RAM to store MSSSecurityDescriptors table in memory

Storing ACLs in Claims format will have big (positive) impact on performance. The reason for this is that having ACLs in NT format will generate a whole load of requests toward Active Directory Servers during security trimming. In case of having ACLs in Claims format, Query Processor will actually do a string compare between user SID and claims blob instead of using authorization with Active Directory. ACLs will be stored automatically in Claims format if the content is in same format, in case that ACL is bigger than 64k, for content crawled in one-way trust scenario or if it is configured via PowerShell to do this:

$spSearchApplication = Get-SPEnterpriseSearchServiceApplication
$spSearchApplication.SetProperty ("ForceClaimACLs", 1)

There are drawbacks as well – this will break Search Alerts so if you need this functionality, don’t force it. Regardless of settings you choose here, Search Alerts won’t work with following content:

• Content from Claims based Web Applications
• Documents that have ACLs bigger than 64k
• Documents indexed in one-way trust settings

Crawl Database

Generally, I already pointed out what you can do to optimize this one – isolate it! Give it enough resources like separate spindle that provide 3,5K – 7K IOPS, enough RAM and CPU so if possible, store it in separate SQL instance as well. Second thing is already pointed out as well – you should consider trimming the crawl log and tweaking default crawl log cleanup interval.

Crawl component(s)

A possibility to optimize crawl topology would be for example to use host distribution rules to manually assign web applications to desired crawl databases. This gives us a possibility to manually control usage of resources in the farm. If no host distribution rules are defined, a Master Crawler will distribute hosts to particular crawl databases.

Another thing to consider is that default crawler settings assume only 4 GB of RAM which is a kind of stupid because this is something which is taken over from SharePoint 2007. SharePoint 2010 does run only on 64-bit hardware and requires at least 8 GB RAM for production use. So, with default settings on dedicated server for crawling, you would end up with 4 GB used for crawler daemon and rest of RAM would be never used by it. What you really want to do here is to increase defailt values for at least 2 or 3 times – you can get the exact value by monitoring server.

So, how can we set this up? This is achievable through modifying DedicatedFilterProcessMemory and FilterProcessMemory values. These values are stored under following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\14.0\Search\Global\Gathering Manager

For best performance, increase both values in the same manner.

Property Database(s)

It is recommended for database server to have enough RAM to store 33% of MSSDocSDIDs, MSSDocProps and MSSDocResults tables in memory for caching purposes. Store this database on its own spindle if possible.

Query Processor

Query Processor caches entries from security descriptor cache from Admin database. Default value is 1,000 entries in SharePoint 2010 RTM. This is increased in Service Pack 1 to 10,000. This can (and should) be tweaked. To read current setting you can run following:

 $spSearchApplication.GetSetting('config:qp_SecTrimCacheSize') 

To set a new value, you can use following:

$spSearchApplication.UpdateSetting('config:qp_SecTrimCacheSize', $newValue)

To get count of Security Descriptors you can run following T-SQL statement on your Admin DB:

select COUNT(*) from dbo.MSSSecurityDescriptors (nolock)

The main reason for caching this is, obviously to reduce roundtrips to database server and back.

Conclusion

Search is a hell of a topic. At the moment I started writing this article I was not counting with so much material. And hey, I didn’t even mention capacity planning in detail or even scratched our test scenario from a start. Stay tuned as I will cover these in my next article