[...]Performance problems with vertical/focused web crawling « Ken's Techno Tidbits[...]…

I haven’t been closely tracking Nutch patches, though I see that there have been improvements to how the plugins can communicate information – which was one of the major headaches previously.

As to a comparison, I think the recent post by Stefano Cherchi on the Nutch list is a good example of where Bixo is much easier to use than Nutch. I had to deal with almost exactly the same use case he describes, where you have top-level pages that have links to the real content, but you have to crawl (and not index) these top pages first.

In Bixo I created a workflow that took top level URLs in, fetched them, used a modified parser to extract the links, then fed these into a second fetch pipe, which in turn fed the results into the parser.

So I wound up with one very simple (conceptually) Cascading workflow, which was very reliable…e.g. no error-prone editing of config files in between runs.

But it would be interesting to do a side-by-side comparison of Bixo & Nutch for some vertical crawl project. If you have something in mind, I’d be interested in giving that a try.

– Ken

I don’t understand your question about IScoreGenerator and IGroupingKeyGenerator – maybe post that to the bixo-dev mailing list with more details?

As to the second suggestion, trying to better use the Hadoop map-reduce support via a smarter partitioner is something that we spent time thinking about, but I don’t see how it could possibly work. I’ll write a blog post about that soon, to address the reasons why in greater depth.

– Ken

What about IScoreGenerator and IGroupingKeyGenerator in BIXO, it can be done “up to 1000 URLs from same domain in a single iteration” easily, so that we will have a tail at the very last iteration. However, we may have different crawl-depths on a single fetch step (one of domains have 2000 links from 100 pages, and another one 1000 links from 100 pages).

Another solution would be to avoid sleeping Threads at all. We need partitioner (Grouping Key) such a way that first N domains go to host 1, second N to host 2, etc; single page from a single domain for each iteration (could be 10 pages with Keep-Alive). Different architecture; we don’t have Queue of URLs for same domain in this case, and threads should not be worried about crawl-delay at all, it will be few minutes in practice… and…

…this shows the main problem of Cascading-type of architecture: resource usage is not optimized.

Imagine 100 Hadoop hosts sharing 100Mbps network connectivity. Fetch is I/O bound, 100% network wait, and 0.1% CPU time.

After all, we spend some time on Fetch, and some another time on Parse, and during Parse our network usage is 0 and CPU 100%, but during fetch network 100% and CPU 0%.

As I see from Google and even Yahoo robot logs they are constantly crawling my website without few-hours delay on parsing, indexing, preparing fetch list, and etc.

How to distribute load evenly?

I missed that Roger was setting things up to only have one URL per host…thanks.

Re fetch termination – terminating slow servers definitely helps. I’m dealing with a slightly different problem now, where the partner crawl has a big exponential decay in the URLs/domain graph. A few domains have 10K URLs, while most of the 50K unique domains have but one.

Since it’s a “partner” crawl, I’m adding a PartnerFetchPolicy that adaptively cranks down the crawl delay (to some minimum value), with the calculated value based on trying to crawl all of the ULRs in the target fetch duration. I’ll post on how well that works soon.

As for Fetcher2 slowness, I don’t think it has to do with what you said. I think so because I see Roger Dunk used generate.max.per.host = 1 in https://issues.apache.org/jira/browse/NUTCH-721 .