Sitecore 8 XP ContentSearch: Exclude path from Indexing - indexing

Question

I'm having trouble with Sitecore Indexing of the general indexes "sitecore_master_index", "sitecore_web_index", which take forever because the crawler/indexer checks all items in the database.
I imported thousands of products with a whole lot of specifications and literally have hundreds of thousands of items in the product repository.
If I could exclude the path from indexing it wouldn't have to check a million items for template exclusion.
FOLLOWUP
I implemented a custom-crawler that excludes a list of paths from being indexed:
<index id="sitecore_web_index" type="Sitecore.ContentSearch.SolrProvider.SwitchOnRebuildSolrSearchIndex, Sitecore.ContentSearch.SolrProvider">
<param desc="name">$(id)</param>
<param desc="core">sitecore_web_index</param>
<param desc="rebuildcore">sitecore_web_index_sec</param>
<param desc="propertyStore" ref="contentSearch/indexConfigurations/databasePropertyStore" param1="$(id)" />
<configuration ref="contentSearch/indexConfigurations/defaultSolrIndexConfiguration" />
<strategies hint="list:AddStrategy">
<strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/onPublishEndAsync" />
</strategies>
<locations hint="list:AddCrawler">
<crawler type="Sitecore.ContentSearch.Utilities.Crawler.ExcludePathsItemCrawler, Sitecore.ContentSearch.Utilities">
<Database>web</Database>
<Root>/sitecore</Root>
<ExcludeItemsList hint="list">
<ProductRepository>/sitecore/content/Product Repository</ProductRepository>
</ExcludeItemsList>
</crawler>
</locations>
</index>
In addition I activated SwitchOnSolrRebuildIndex as it's awesome ootb functionality, cheers SC.
using System.Collections.Generic;
using System.Linq;
using Sitecore.ContentSearch;
using Sitecore.Diagnostics;
namespace Sitecore.ContentSearch.Utilities.Crawler
{
public class ExcludePathsItemCrawler : SitecoreItemCrawler
{
private readonly List<string> excludeItemsList = new List<string>();
public List<string> ExcludeItemsList
{
get
{
return excludeItemsList;
}
}
protected override bool IsExcludedFromIndex(SitecoreIndexableItem indexable, bool checkLocation = false)
{
Assert.ArgumentNotNull(indexable, "item");
if (ExcludeItemsList.Any(path => indexable.AbsolutePath.StartsWith(path)))
{
return true;
}
return base.IsExcludedFromIndex(indexable, checkLocation);
}
}
}

Answer 1

You can override SitecoreItemCrawler class which is used by the index you want to change:
<locations hint="list:AddCrawler">
<crawler type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
<Database>master</Database>
<Root>/sitecore</Root>
</crawler>
</locations>
You can then add your own parameters, e.g. ExcludeTree or even a list of ExcludedBranches.
And in the implementation of the class just override method
public override bool IsExcludedFromIndex(IIndexable indexable)
and check whether it is under excluded node.

Answer 2

When importing large amounts of data you should try disabling the indexing of data temporarily otherwise you'll run into issues with a crawler that can't keep up.
There's a great post here on disabling the index while importing data - it's for Lucene but I'm sure you can do the same with Solr,
http://intothecore.cassidy.dk/2010/09/disabling-lucene-indexes.html
Another option could be to store your products in a separate Sitecore database rather than in the master db.
Another post from into the core:
http://intothecore.cassidy.dk/2009/05/working-with-multiple-content-databases.html