Ever wondered about the subtle nuances of Seach Engine Optimization (SEO's) -- How to get your site placed higher in listings? Here's a couple tips from a good friend who devotes LOTS of his time to SEO figuring out the nuances of those little spiders...
"Server response is a big deal for the spiders. They want to hit a
site, snarf up pages and move on. They reward dedicated servers,
dedicated IP addresses (c-block) for a domain, and fast response times
because they don't have to take as much time to spider the content. So
yes, it's
convenient for the spider.
Again, it's not a deal-breaker in most cases, but if you are on a really slow connection or have slow-loading or large (>100k) pages, it CAN impact the number of pages that will be spidered. SE's are stupid, but they're very logical. If it takes 5ms to grab a page from site X with 10k pages, and Google finds site Y with 10k pages and only takes 1ms to grab a page, the latter site will be crawled more frequently and more deeply because of the efficiency with which G can gather the information.
On a related note, any common file name can be spidered without bias. No preference is shown for htm, html, php, cfm or other common extensions. Domain names are the same way. .edu's are given preference, but that's the only exception. A .biz or .info can outrank a .com or .org any day of the week, so if you can't get the name you want there are other options, unless branding is important, in which case a .com or .org is best."