Please visit our online store at http://store.yahoo.com/samizdat
Update to this article, 7/4/2006 -- Google "Sitemaps"
When I started blogging a few months ago, I expected that would increase traffic to my site. And that in fact happened. I went from an average of 18-19 gigs of traffic per month up to an average of 30-31 gigs. But as I discovered from the AW (Advanced Web) Stats program provided by my Web host (Ochosting), more than a third of that traffic was “not viewed” — most traffic generated by search engine crawlers, mainly from Google.Now I love crawlers, because I love search engines. Most of my traffic, and most of my business comes to me by way of Google. So I didn’t want to do anything to get in the way of being thoroughly indexed by Google. But the extra, unnecessary traffic was wasting money — not a lot of money; but it did nearly double my (previously modest) monthly charges for Web site hosting.
Then I discovered Google Sitemaps, a new free services that lets you make sure that all your pages get included in the Google index and at the same time minimizes web crawler traffic. Basically, you tell Google the URLs for all your pages and the frequency with which those pages are usually updated; so the crawler will come back when it needs to and look for what it needs to, rather than randomly coming back over and over again when little or nothing is new.
At Google http://www.google.com/ click on Business Solutions, then on Website Map, and sign up. Add your site. Go through the quick “verification” process, to confirm that you are the site owner (either creating a file with a particular name in your top directory or adding a particular metatag to your index page). Then when your site appears listed and verified under “my sites”, click on Add a Sitemap.
At first, I was stumped there. The instructions are technical to a degree that I didn’t know what to do. (I am not a UNIX programmer). I considered the option of creating a text file by hand; but I have over 2000 pages at my site, so that would take me many hours, and would be a pain to update in the future. But the text mentioned that third party software is available to help generate a sitemap.xml file. They didn’t mention any names or provide any links (probably a matter of liability). So I used Google Search — looking for “Google sitemap software”.
I found SOFTplus GSiteCrawler at http://gsitecrawler.com which runs on my PC, crawls my Web site, and generates an xml sitemap file that I then can upload to my site (and submit to Google). With my cable-modem connection and 2000+ pages, the crawl took about an hour, in the background. Once I had uploaded the sitemap.xml file to my top directory, I went back to Google Sitemap and told them that it was there; which Google verified within about an hour.
Gsitecrawler also generated a comprehensive and detailed list of link errors found; which I will now use as a guide for fixing them.
Now when I add new pages to my site, I’ll run Gsitecrawler again to generate a new sitemap.xml and go back to Google Sitemap to “resubmit it”. If I just make changes to existing pages, there’s no need for me to do anything — the new text should be crawled and added in the normal course of events.
From that same Google search for “Google sitemap software, I also downloaded and installed Google Sitemap Generator for WordPress from http://www.arnebrachhold.de/2005/06/05/google-sitemaps-generator-v2-final which provides a WordPress plug-in that makes a sitemap for my blog. That installation was a bit tricky. (I’ve found everything having to do with WordPress “tricky”, with far too little explanation.) But I finally got it working too (the toughest part for me being figuring out how to “via chmod” make a file “writeable”).
So now it will be interesting to see the results as I check my Web stats in coming months. I’m hoping for more “real” traffic and far less crawler traffic.
Google Today, this is by far the best and the most popular search engine. If you are indexed here, you really don't need to worry about the rest. This no-frills site is also the search engine for Yahoo. Over a third of the page views at my site come by way of Google. This site archives the Web pages it indexes, so if a page goes away you can still retrieve it from Google.
AltaVista , unfortunately, has slowed down their mechanism for free submissions. Whereas it used to take a couple days to get into the index, now it takes 4-6 weeks; as they try to pressure Web sites into paying for "Express Submission". (Their price is simply outrageous). For free submission, scroll down the page to where you'll find their "submission code." This is letters and numbers in a wide variety of fonts and at odd angles so machines cannot read them. You have to copy those letters into the form (NB -- this is not case sensitive). Then fill in the form below that with the URL for the page that you want to submit for free inclusion in the main index. If you typed the code correctly (which isn't easy given the bizarre type styles), that page is submitted and you can submit four more one at a time without enter a new code. To go beyond 5 you have to start over again with the code. If you type the code wrong, you can try again with a new one. This mechanism is designed to block out automatic submission services that had been spamming AltaVista and all the other free search engines as well; and which had been so clogging the submission process that real individuals and webmasters with important content had been frustrated that it took so long to get their info in or that it never made it. The new system is a clever innovation and a tremendous improvement. Give it a try soon.
By the way, AltaVista is by far the most powerful and precise search engine -- the preferred tool of professional researchers. They also have started selling "sponsored" link placement -- the "featured sites" that appear at the top of the list, before the number of matches. Real matches mean that the words of your query actually appear on the matching pages. With featured sites, the sponsor paid for certain "key words" that happened to appear in your query, but those words may not appear anywhere on the "matching" page. (Bought by Overture, which was then bought by Yahoo; but still available as a separate service).
Alltheweb (AKA Fast Search) No frills and a very large index. Bought by Overture, which was then bought by Yahoo (like AltaVista), they now use the same submission technique as AltaVista
Directories:
The above are all "search engines". For a discussion of the difference
between search engines and directories see my article at http://www.samizdat.com/dir.html
Yahoo, Open Directory, and LookSmart are all directories. They are very important, but the submission process is very different. At Yahoo and Open Directory (http://dmoz.org) you have to navigate through their cascading menus to the subdirectory that you feel is most appropriate for your site, and from that page you click to submit; then fill out a detailed form, including a description of your site. Keep in mind that with a directory you are submitting one description for your entire site, and only the description (not the actual text of your pages) will be entered into the directory. Also, your submission will be reviewed by a human editor, who makes sure that what you say about your site matches what a visitor will see there and also checks to make sure you asked for an appropriate category. LookSmart charges a submission fee.
"Search engines" not worth bothering to submit to:
www.iwon.com and search.msn.com They use Looksmart for their directory and Inktomi for for their index. If you submit to Hotbot, you'll eventually get into their Inktomi index.
aolsearch.aol.com Not really a search engine, they uses the Open Directory
www.snap.com Owned by NCBi. You have to pay to be listed there.
www.overture.com Formerly known as "Goto". You have to pay to be listed here. Advertisers bid for keywords.
www.scourtheweb.com Uses the GoTo which is paid.
www.findwhat.com You have to pay to be listed here. As with GoTo, advertisers bid for keywords.
NB -- "Meta-search engines" like Dogpile, piggy-back on other search engines, sending queries to multiple search indexes at the same time. They do not maintain their own search indexes.
Recent changes:
DirectHit specializes in ranking based on "popularity." In addition
to running their own search site, they also provide services to other search
sites, helping them to add "popularity" to their mix of ranking criteria.
They used to take free submissions. Now, owned by Teoma, they only do paid.
Unfortunately, Excite, which used to be one of my favorites, went Chapter
11 and the the service has since changed radically. They used to include
"fuzzy logic" and a thesaurus to provide you with relevant results even
when you weren't precise about what you are looking for. Now when you click
to go to their submissions page, they connect you with Overture, with its
paid submission service.
Infoseek used to be one of my favorites. Now owned by Go.com, they
try to get you to pay for listings.
Lycos recently switched from free submissions to paid submissions;
so count them out as well.
Hotbot uses Inktomi
for its index and their submission page used to be my best route for inclusion
on all search sites powered by Inktomi. Unfortunately, they have "temporarily"
disabled free submissions. As an alternative, you could choose one of Inktomi's
ridiculously high-priced paid
submission programs.
Northernlight In addition to the Web, this site also indexes its own
private collection of articles which are for sale. Unfortunately, they
no longer welcome free submissions.
MSN This used
to be the best free route to get your content into the Inktomi index (which
is used by many different search engines). (Inktomi, also, bought by Yahoo.)
But now they are charging for submission.
From: "Val Sherbin" <val@ghost-cat.com>
Sent: Wednesday, February 12, 2003 1:17 PM
Subject: Search engines experiments
I was running a few experiments studying three search engines for the last year. Here are some results. I hope you might find them useful.
Altavista:
Basic Submit: It took about 4 month for my website to be indexed. Only
the first page and the pages that it linked to were indexed. After that
I added a few unique test words to the title, keywords, description and
the body. None was found even after two month.
Express Submit ($29 for 6 month): Now the changes are searchable in
48 hours. In spite of the AV's claim:
Dear Valued Customer,the test words added to the keywords were indexed too, and the whole first page was indexed, not just the first paragraph, however the second layer (the pages linked from the main index page) were not indexed at all. That was a step back from the free listing. But my biggest surprise was that after my subscription expired, my site was removed from AV index completely. That means that I have to either keep paying or wait for another 4 month to have my site indexed again.
Alta Vista's express inclusion only indexes the webpage that was
submitted. It does not go into the site and index any further site unless
you decide to add them with the express inclusion service also. The words
that Alta Vista uses as the most important data on your site is your Title
tag, description tag and the first paragraph on your site. The keywords tag
is not as important as having the first couple of words within your title
tag to have relevant content to your site."
Google:
It took less that 2 month for my site to appear in their index. Just
like in the case of free listing with Altavista only the first page and
the pages it linked to were indexed. The test words could not be found
on Google even six month later. I also noticed that the guys who have free
pages on some popular sites or have links to their sites mentioned on popular
sites like ezforum.com have much higher placement than the people who have
their own domains hosted by small providers. My eBay shop has much higher
ranking that my own site.
Alltheweb:
It took about a month for them to index my site. All pages on the site,
including those that were not linked were indexed, just like in good old
times on AV. But no test words added after the indexing could be found
on Alltheweb, same as with Google.
Book
collections on CD and DVD. A library for the price of a book.
Return to B&R Samizdat Express
Sitemap with links to every page at this site
| Internet Business Showcase: | |||
Dog Training and Pet Care Veterinary, Dog/Cat Veterinarians |
Viatical Settlements RFID Tags, RFID Readers Link Popularity & Link Exchanges |
Hair Restoration for Hair Loss Hair Restoration Plastic Surgery & Surgeons |
Used Cars Guide Homecoming & Prom Dresses Dropship & Wholesale Sources |