Comments on: Why fetching web pages doesn’t map well to map-reduce http://ken-blog.krugler.org/2009/12/12/why-fetching-web-pages-doesnt-map-well-to-map-reduce/ Fri, 20 Aug 2010 02:46:48 +0000 hourly 1 http://wordpress.com/ By: kkrugler http://ken-blog.krugler.org/2009/12/12/why-fetching-web-pages-doesnt-map-well-to-map-reduce/#comment-560 kkrugler Mon, 09 Aug 2010 21:11:56 +0000 http://ken-blog.krugler.org/?p=197#comment-560 Hi Xiaomeng, Yes, MR will parallelize this wait time. But you can't run many MR tasks/server, as each one runs in its own child JVM. So to make efficient use of your hardware, you need to achieve greater parallelism, which means layering a multi-threaded model on top of Hadoop. -- Ken Hi Xiaomeng,

Yes, MR will parallelize this wait time. But you can’t run many MR tasks/server, as each one runs in its own child JVM. So to make efficient use of your hardware, you need to achieve greater parallelism, which means layering a multi-threaded model on top of Hadoop.

– Ken

]]> By: xiaomeng wan http://ken-blog.krugler.org/2009/12/12/why-fetching-web-pages-doesnt-map-well-to-map-reduce/#comment-559 xiaomeng wan Mon, 09 Aug 2010 20:54:30 +0000 http://ken-blog.krugler.org/?p=197#comment-559 Regarding the "But since most of the time during a fetch is spent waiting for the server to respond, you’re getting very low utilization of your available hardware & bandwidth." Isn't the waiting time parallelized by MR? so it ends up faster? Regarding the “But since most of the time during a fetch is spent waiting for the server to respond, you’re getting very low utilization of your available hardware & bandwidth.” Isn’t the waiting time parallelized by MR? so it ends up faster?

]]> By: kkrugler http://ken-blog.krugler.org/2009/12/12/why-fetching-web-pages-doesnt-map-well-to-map-reduce/#comment-500 kkrugler Mon, 01 Feb 2010 17:25:54 +0000 http://ken-blog.krugler.org/?p=197#comment-500 Hi Otis, My comment about parallelism being limited by the number of reducers is based on a design goal of trying to use straight MR, without the complexity of additional threading support. Since you don't get sufficient parallelism via just the number of reducers, you have to deal with the added complexity of a multi-threaded reducer. Which is what Nutch and Bixo both do - but it's not pretty :) Hi Otis,

My comment about parallelism being limited by the number of reducers is based on a design goal of trying to use straight MR, without the complexity of additional threading support.

Since you don’t get sufficient parallelism via just the number of reducers, you have to deal with the added complexity of a multi-threaded reducer. Which is what Nutch and Bixo both do – but it’s not pretty :)

]]> By: sematext http://ken-blog.krugler.org/2009/12/12/why-fetching-web-pages-doesnt-map-well-to-map-reduce/#comment-499 sematext Mon, 01 Feb 2010 17:04:08 +0000 http://ken-blog.krugler.org/?p=197#comment-499 Regarding the "the maximum amount of parallelization would be equal to the number of reducers, which typically is something close to the number of ccores (servers * cores/server). So on a 10 server cluster w/dual cores, you’d have 20 threads active." piece: Do number from this example match what you've observed with Nutch? I don't have an example of Nutch log handy, but it does include the number of active threads, so I'm wondering if in your experience it really matches the above theory(?)? Regarding the “the maximum amount of parallelization would be equal to the number of reducers, which typically is something close to the number of ccores (servers * cores/server). So on a 10 server cluster w/dual cores, you’d have 20 threads active.” piece:

Do number from this example match what you’ve observed with Nutch? I don’t have an example of Nutch log handy, but it does include the number of active threads, so I’m wondering if in your experience it really matches the above theory(?)?

]]>