.Net中也有很多很多开源的爬虫工具，abot就是其中之一。Abot是一个开源的.net爬虫，速度快，易于使用和扩展。项目的地址是https://code.google.com/p/abot/
对于爬取的Html，使用的分析工具是CsQuery, CsQuery可以算是.net中实现的Jquery, 可以使用类似Jquery中的方法来处理html页面。CsQuery的项目地址是https://github.com/afeiship/CsQuery
一. 对Abot爬虫配置1. 通过属性设置
先创建config对象，然后设置config中的各项属性:
CrawlConfiguration crawlConfig =newCrawlConfiguration();
crawlConfig.CrawlTimeoutSeconds=100;
crawlConfig.MaxConcurrentThreads=10;
crawlConfig.MaxPagesToCrawl=1000;
crawlConfig.UserAgentString="abot v1.0 http://code.google.com/p/abot";
crawlConfig.ConfigurationExtensions.Add("SomeCustomConfigValue1","1111");
crawlConfig.ConfigurationExtensions.Add("SomeCustomConfigValue2","2222");
2. 通过App.config配置
直接从配置文件中读取，但是也任然可以在修改各项属性:
CrawlConfiguration crawlConfig =AbotConfigurationSectionHandler.LoadFromxml().Convert();
crawlConfig.CrawlTimeoutSeconds=100;
crawlConfig.MaxConcurrentThreads=10;
3. 应用配置到爬虫对象
PoliteWebCrawler crawler =newPoliteWebCrawler();
PoliteWebCrawler crawler=newPoliteWebCrawler(crawlConfig,null,null,null,null,null,null,null);
二，使用爬虫，注册各种事件爬虫中主要是4个事件, 页面爬取开始、页面爬取失败、页面不允许爬取事件、页面中的链接不允许爬取事件。
下面是示例代码:
crawlergeCrawlStartingAsync += crawler_PRocessPageCrawlStarting;//单个页面爬取开始crawler.PageCrawlCompletedAsync += crawler_ProcessPageCrawlCompleted;//单个页面爬取结束crawler.PageCrawlDisallowedAsync += crawler_PageCrawlDisallowed;//页面不允许爬取事件crawler.PageLinksCrawlDisallowedAsync += crawler_PageLinksCrawlDisallowed;//页面链接不允许爬取事件voidcrawler_ProcessPageCrawlStarting(objectsender, PageCrawlStartingArgs e)
{
PageToCrawl pageToCrawl=e.PageToCrawl;
Console.WriteLine("About to crawl link {0} which was found on page {1}", pageToCrawl.Uri.AbsoluteUri, pageToCrawl.ParentUri.AbsoluteUri);
}voidcrawler_ProcessPageCrawlCompleted(objectsender, PageCrawlCompletedArgs e)
{
CrawledPage crawledPage=e.CrawledPage;if(crawledPage.WebException !=null|| crawledPage.HttpWebResponse.StatusCode !=HttpStatusCode.OK)
Console.WriteLine("Crawl of page failed {0}", crawledPage.Uri.AbsoluteUri);elseConsole.WriteLine("Crawl of page succeeded {0}", crawledPage.Uri.AbsoluteUri);if(string.IsNullOrEmpty(crawledPage.Content.Text))
Console.WriteLine("Page had no content {0}", crawledPage.Uri.AbsoluteUri);
}voidcrawler_PageLinksCrawlDisallowed(objectsender, PageLinksCrawlDisallowedArgs e)
{
CrawledPage crawledPage=e.CrawledPage;
Console.WriteLine("Did not crawl the links on page {0} due to {1}", crawledPage.Uri.AbsoluteUri, e.DisallowedReason);
}voidcrawler_PageCrawlDisallowed(objectsender, PageCrawlDisallowedArgs e)
{
PageToCrawl pageToCrawl=e.PageToCrawl;
Console.WriteLine("Did not crawl page {0} due to {1}", pageToCrawl.Uri.AbsoluteUri, e.DisallowedReason);
}
三， 为爬虫添加多个附加对象Abot应该是借鉴了asp.netMVC中的ViewBag, 也为爬虫对象设置了对象级别的CrwalBag和Page级别的ViewBag.
PoliteWebCrawler crawler =newPoliteWebCrawler();
crawler.CrawlBag.MyFoo1=newFoo();//对象级别的CrwalBagcrawler.CrawlBag.MyFoo2 =newFoo();
crawler.PageCrawlStartingAsync+=crawler_ProcessPageCrawlStarting;
...voidcrawler_ProcessPageCrawlStarting(objectsender, PageCrawlStartingArgs e)
{//获取CrwalBag中的对象CrawlContext context =e.CrawlContext;
context.CrawlBag.MyFoo1.Bar();//使用CrwalBagcontext.CrawlBag.MyFoo2.Bar();//使用页面级别的PageBage.PageToCrawl.PageBag.Bar =newBar();
}
四，启动爬虫启动爬虫非常简单，调用Crawl方法，指定好开始页面，就可以了。
CrawlResult result = crawler.Crawl(newUri("http://localhost:1111/"));if(result.ErrorOccurred)
Console.WriteLine("Crawl of {0} completed with error: {1}", result.RootUri.AbsoluteUri, result.ErrorException.Message);elseConsole.WriteLine("Crawl of {0} completed without error.", result.RootUri.AbsoluteUri);
五，介绍CsQuery在PageCrawlCompletedAsync事件中, e.CrawledPage.CsQueryDocument就是一个CsQuery对象。
这里介绍一下CsQuery在分析Html上的优势:
cqDocument.Select(".bigtitle > h1")
这里的选择器的用法和Jquery完全相同，这里是取class为.bittitle下的h1标签。如果你能熟练的使用Jquery，那么上手CsQuery会非常快和容易。