 |
Normally we wait until the end of the year to figure out our charitable donations, but I’ve been using Wikipedia so much over the past few days that I felt like I needed to donate today. | |
Bixo Mailing List- Support for fetch loop terminationHi all, I just pushed a minor change to Bixo, where the FetcherPolicy now has an isTerminated() method. By default this just checks the crawl end time, and if
- Re: beginner questionHi Brian, ... You should be able to use the Cascading JDBCTap to directly insert values into your DB. See the Bixo SimpleCrawlTool for an example of using the
- Re: Web fetchers. Not so polite but not unpolite... robots.txt revis... [snip] ... [snip] ... I just committed a change to bixo's master in GitHub, where I refactored the robots.txt processing code. You can now customize the
- Re: Web fetchers. Not so polite but not unpolite... robots.txt revis... Hi Ken, You clarified me a lot of things with this. I will try to follow rules as much as possible once you explained me how to do it. But also will be
- Re: Web fetchers. Not so polite but not unpolite... robots.txt revisHi there, ... Depends on what they crawl at these sites. For example Blogger.com's robots.txt file looks like: User-agent: * Disallow: /profile-find.g
- Support for fetch loop termination
Hi Ken, That’s very generous of you to donate to Wikipedia. A lot of people forget that it’s non-profit. At least until Google buys them. lol
I just sent you an e-mail about crawling Wikipedia’s article dump. Check your junk folder if you didn’t get it.
Thanks.
kirk