Web Crawling 1.0

In Stuffing Six Million Pages Down Google’s Throat, Tim O’Reilly brings up a point, and some questions:
. . . just how poorly the big search engines index small sites with large collections of data . . .
But it’s worth thinking about absolute (and temporary) limits to the growth of Web 2.0. What constraints do […]