BUSINESS ON THE WORLD WIDE WEB --

November 7, 1996 -- Impact of Search Engines on Internet Business


Transcript of the live chat session that took place Thursday, November 7, 1996.

These sessions are scheduled for noon-1 PM US Eastern Time (GMT -4) every Thursday.

These sessions are hosted by Richard Seltzer. If you would like to receive email reminders of our chat sessions, simply send a blank email message to businessonthewebchats-subscribe@yahoogroups.com or go to http://groups.yahoo.com/group/businessonthewebchats and sign up there.

For transcripts of other previous sessions and a list of future topics, click here.

For an article on how to make "business chat" work (based on this experience), click here.

Since the chat itself happens at a rapid pace, it's often difficult to note interesting facts in particular URLs as they appear on-line. Here's a place to take a more leisurely look. I've rearranged some of the pieces to try to capture the various threads of discussion (which sometimes get lost in the rush of live chat).

Please send email with your follow-on questions and comments, and suggions for topics we should focus on in future sessions. So long as the volume of email responses is manageable, I'll post the most pertinent ones here for all to see.


Threads (reconstructed after the fact):


INTRODUCTIONS

Richard Seltzer (199.3.129.189) - 11:57am -- The scheduled chat is on Business on the WWW. If you are here for that discussion, please identify yourself.

We're here to share experiences about doing business on the Internet -- particularly the World Wide Web. What works? What doesn't work? Why? What are the trends that matter? How can you/should you adapt to the Internet culture and environment?

I work for the Internet Business Group at Digital Equipment in Littleton, MA. In that capacity, I end up talking to people from large companies about how they can use the Web for business. I also have my own personal Web page -- which is content rich and no frills -- which I do for practically nothing and draws a fair amount of traffic and attention.

In a chat session like this things can get pretty frantic. It's sometimes difficult to follow the threads of conversation. And there's no time to write down interesting URLs and facts. So last week, I took a copy of the raw transcript and edited it to make the threads clearer and posted it at my own little Web site so anyone could take a look. You can see it at http://www.samizdat.com/chat16.html I plan to do the same today. Barring technical difficulties, I hope to have a transcript up later today. I'll post it at the same site, naming this one /chat17.html

If you are here for the chat on Business on the WWW, please identify yourself, and let us know your interests.

barbara (199.93.126.34) - 11:59am -- Hi! I'm here.

Richard Seltzer (199.3.129.189) - 12:01pm -- Hello, Barbara, we have two topics -- instead of one -- on the agenda today: 1) continuing our discussion on "new kinds of money" -- particularly, the question of how to "bootstrap" one of these new forms of virtual currency, quickly getting lots of individuals and companies to use one so it then becomes generally available. 2) the impact of search engines on Web site design. If you are here for one or the other of those topics, please let me know.

Richard Seltzer (199.3.129.189) - 12:04pm -- Several people sent email with concerns/ questions related to "new kinds of money". They hoped we would discuss that topic, but all of them (unfortunately) said that they would be unable to connect today. To get that discussion going, we need live folks who wish to ask/answer questions. The alternative is the impact of search engines. My book The AltaVista Search Revolution was just published by Osborne/ McGraw-Hill, and I have a lot to say on that topic if the audience here is interested.

Tom Dadakis DadaCom tomdadak@ix.netcom.com (207.93.47.66) - Thu, Nov 7, 12:06pm -- Hi,joining in a few minutes late.

Richard Seltzer (199.3.129.189) - 12:08pm -- Welcome, Tom -- are you mainly interested in continuing the discussion on "new kinds of money" or the question of the impact of search engines on Web page/site design?

Bill Wendel (140.186.92.2) - 12:08pm -- I'm interested in your second topic: the impact of search engines on Web site design and would like to know more about your book,too. I love AltaVista and it has revolutionized the way I do research!

Warren Agin - Law Solutions (199.3.142.80) - 12:15pm -- Hello, joining in a little late and can't stay long.

todd (192.208.46.249) - 12:16pm -- hi. I'm more interested in the search engines/directories topic.

Jim Dorval (jimd@vivo.com) (199.171.52.30) - 12:29pm -- Hi, sorry I am late, just extinguishing the last flames. Vivo Software. Multimedia over the internet.


MICROPAYMENTS

barbara (199.93.126.34) - 12:04pm -- I'd also like to know if any company has tried the small money approach?

Richard Seltzer (199.3.129.189) - 12:10pm -- Barbara -- Regarding the "small money" approach, I'm expecting that Russ Jones, a Millicent expert from Digital, will be joining us shortly. He'd be the best one to point you to examples. In the interim I'll try to get started on the search questions.


SEARCH ENGINES -- DEFINITIONS

barbara (199.93.126.34) - 12:02pm -- Richard, I'm interested in your second topic about search engines on the internet. How do you know which ones to submit to or do you just submit to everything you can?

Tom Dadakis DadaCom tomdadak@ix.netcom.com (207.93.47.66) - Thu, Nov 7, 12:10pm -- Most of the search engines are interconnected now but you have to know whether there is a robot searching the site or there is some human intervention, like at yahoo. But Richard would be the expert in describing how the search engines work. Richard, why don't you explain how they work.

Richard Seltzer (199.3.129.189) - 12:07pm -- Barbara -- First, let's distinguish between "directories" and search engines. Yahoo is a directory. The information in their database comes from "submissions" -- it is categorized by the person doing the submission and checked (by hand) by folks at Yahoo. A full-text search engine like AltaVista gets its information by sending out a robot program, which fetches Web pages and brings them back to be indexed. Over time AltaVista should retrieve data on any public Web site that has links to others and that doesn't include commands to exclude robots. You can, however, make sure that your site/page is indexed sooner rather than later by "submitting" -- connecting to the site, clicking on Add URL, and providing your Web address.

Richard Seltzer (199.3.129.189) - 12:13pm -- Tom -- The robot programs (spiders) appear to a Web site like just another user. They retrieve information which then is indexed. AltaVista indexes every single word -- some others deal only with key words. Also, the robot that AltaVista uses ("Scooter") sends out dozens of "threads" simultaneously -- in other words it acts as if it were dozens of users connecting to dozens of different Web sites simultaneously. This speeds up the fetching process -- which otherwise could take far too long, considering there are over 30 million pages on the Web today. Is that clearer?


TARGETED, INDUSTRY-SPECIFIC SEARCHES

barbara (199.93.126.34) - 12:06pm -- I work for an electronic commerce company and am publicizing their Web site over the Internet. Are there any special web directories for business of electronic commerce?

barbara (199.93.126.34) - 12:15pm -- I still would like to know if there are specific search engine sites or directories. They all seem so general and non-focused.

Warren Agin - Law Solutions (199.3.142.80) - 12:17pm -- Barbara, I know there is a site which searches websites with a legal orientation.


INTELLIGENT AGENTS AND "META-SITES"

Bill Wendel (140.186.92.2) - 12:14pm -- What is the difference between a search engine and an intelligent agent? How do I decide which to use and when? I am particularly interested in learning how both can be used in real estate where there are now over 15,000 websites which threaten to break up the cartel-like hold that realtors have had on inventory in the past. That is, if way can be developed to search all 15,000 sites with a single search for matching properties.

Richard Seltzer (199.3.129.189) - 12:20pm -- Bill -- The biggest barrier to doing what you are interested in doing is the fact that most of those sites hold their information in databases rather than as plain HTML pages. Robots, like AltaVista, simply stop at the front-page of such sites. So a search tailored to find homes for sale, say, in Vancouver would only find those pages that aren't in databases and aren't block in other ways. Intelligent agents could probably be programmed to go through the registration and/or query process for a few such sites. But it would be a daunting task to program one for all the many different real estate sites today.

Bill Wendel (140.186.92.2) - 12:24pm -- Richard, thanks for your answer. Can you elaborate on: "Intelligent agents could probably be programmed to go through the registration and/or query process for a few such sites. But it would be a daunting task to program one for all the many different real estate sites today." If voluntary standards were set to facilitate such searches, it would be a great public service and probably a good business opportunity, too.

Warren Agin - Law Solutions (199.3.142.80) - 12:25pm -- Bill, sounds sort of like "Submit-it" which uses the URL to complete the submission forms for several different sites.

Bill Wendel (140.186.92.2) - 12:29pm -- Warren Agin: Thanks for your suggestion. I just retrieved 8,000 references to submit-it using Alta Vista. I peruse some of them later.

Todd (192.208.46.249) - 12:30pm -- FYI, there's a free service out there, I think it's "submit" or "submit it" or something" that posts your page to a dozen or so directories at once.

Richard Seltzer (199.3.129.189) - 12:32pm -- Bill -- The nature of the problem you raise depends largely on how many sites you want to be able to search. If it's a dozen or maybe even a few dozen, then, as Warren suggests, an approach like Submit-It make work. But I believe the number of such sites is escalating rapidly. Possible solution -- enter into some kind of business arrangement with the main sites -- encouraging them to use standard methods (that's a tough one) or at least to cooperate with you (that's also tough). You could aim to create a "meta-site" one site that is an entrypoint for the whole industry. But a lot of other folks would probably also like to play that role. No guarantees of victory there.

barbara (199.93.126.34) - 12:34pm -- Todd, there are a lot of services like that, but, basically, once you type the information in their main directory, you usually have to individually make sure the information is submitted to their list of search engines. They are a good starting point, but it's not a large time saver. It does give you good lists to work from, though.

Richard Seltzer (199.3.129.189) - 12:37pm -- Back to the real estate question -- I had an interesting experience with job search sites recently. My wife was looking for a job. So we connected to about a dozen different sites. They all were set up very differently with different cascades of menus and search methods. We found nothing. Then, out of the blue, my wife got a call from a headhunter, who had found her resume at our little Web site using AltaVista. Think about it -- if you were a headhunter, would you rather do dozens of different kinds of searches at dozens of different sites, or would you rather just do an AltaVista search and find the folks who had sense enough to post their resumes as plain HTML files (as opposed to putting them in the databases of job search sites, where search engines could never index them).

Todd (192.208.46.249) - 12:40pm -- Bill, Richard- Home Scout attempts to be such a site. (http://homescout.com). It's rather rough at this stage though.

Richard Seltzer (199.3.129.189) - 12:41pm -- Todd -- Thanks for the pointer. I'll take a look at homescout. I'm particularly interested in how people set up "meta-sites". I feel it's an interesting new business opportunity. It's too much hassle going to the sites of every individual company when your purpose is to find a particular kind of product or information that could be anywhere.

Warren Agin - Law Solutions (199.3.142.80) - 12:43pm -- Richard - I have a metasite geared toward bankruptcy professionals. A pretty small segment.

Warren Agin - Law Solutions (199.3.142.80) - 12:43pm -- Basically, I just provide lists of resources available on the Internet, and make an effort to organize them in a usable fashion.

Richard Seltzer (199.3.129.189) - 12:48pm -- Warren -- Sound like a step in the right direction. The kinds of features that move it up a notch would be 1) forum/chat -- any way to let users interact at your site on the focus topic 2) something more than just hyperlinks -- for instance specialized search focused at the key sites in your area. You mentioned LawCrawler, which uses AltaVista. You also could put together a kludge -- compose specific complex searches using AltaVista and limited to the sites (host:) in your "industry". Then bookmark the search results and cut and paste those URLs on your pages and anyone clicking on that link would automatically make that complex search of just those sites. (NB -- this only works well when the target sites have their main info in HTML pages, rather than databases.)

Warren Agin - Law Solutions (199.3.142.80) - 12:50pm -- I do have a broad search tool which allows users to identify the pages in the web which contain the desired key word. I also have a discussion group which, much to my amazement, actually attracts some postings. I also have some links which check out various databases like AltaVista for key words like "bankruptcy"

Richard Seltzer (199.3.129.189) - 12:52pm -- Warren -- Sounds good. What's the URL? (I know you've posted it in previous chat sessions -- just so the folks today can see it.)

Warren Agin - Law Solutions (199.3.142.80) - 12:53pm -- http://www.agin.com/lawfindKeep in mind that this is pretty low budget. I get about 50-100 hits a day. Not quite Alta Vista.

Bill Wendel (140.186.92.2) -- 1:08pm -- Thanks for addressing real estate searches. While you're checking homescout, try IRED.com, and REIPA (Real Estate Information Providers Association). bwendel@realestatecafe.com We are overhaling our website (http://www.realestatecafe.com) and would like to incorporate searches and intelligent agents to help consumers save money! The Consumer Federation of America says consumers could save $10 billion per year if they knew more aobut how real estate agents get paid. I firmly believe that reforms in the RE industry will be one of the Internet's first multi-billion paybacks for consumers!


IMPACT OF SEARCH ENGINES ON WEB PAGE/SITE DESIGN -- GENERAL CONSIDERATIONS

Richard Seltzer (199.3.129.189) - 12:17pm -- With regard to search engines, there are about 8 points to keep in mind from the point of view of the information provider: 1) you no longer control the context of your customer's experience 2) you can use the search engine to learn about your own site and its problems 3) You can learn who links to your pages (and who links to pages that are old or have moved or have expired) 4) how a search engine like AltaVista ranks information -- hence how to optimize how high your pages will appear on a search list 5) what kinds of things prevent a robot from retrieving your pages at all (e.g., registration/passowrds, databases, cookies) 6) the "flypaper" strategy for getting folks to come to your site 7) metatags -- their use and abuse 8) how and why to exclude robots from certain directories and files. Please let me know what of these you'd like me to attack first.

Bill Wendel (140.186.92.2) - 12:20pm -- I am interested in #4) how a search engine like AltaVista ranks information and hence how to optimize how high your pages will appear on a search list; and #8) how and why to exclude robots from certain directories and files.


a

OPTIMIZING RANKING -- USE AND ABUSE OF METATAGS

Richard Seltzer (199.3.129.189) - 12:23pm -- Okay -- let's focus for now on how to optimize. First rule -- use lots of text and state clearly what you are doing. That's the simplest and most direct way to be findable. Pay special attention to the HTML title and the first few lines of text. The words that appear there are crucial. That's what will put you quickest at the top of the list. Metatags are a crutch -- a way to patch up poorly designed pages.

Tom Dadakis DadaCom tomdadak@ix.netcom.com (207.93.47.66) - Thu, Nov 7, 12:18pm -- I think people would be more interested in how they get listed and how they can improve their positioning on the list by using keywords or meta tags in their html pages listing certain words (sometimes their competitors names) to come up under certain searches.

barbara (199.93.126.34) - 12:19pm -- Tom, isn't using competitor's names against intenet netiquette?

Tom Dadakis DadaCom tomdadak@ix.netcom.com (207.93.47.66) - Thu, Nov 7, 12:21pm -- Barbara, you are absolutely correct but wouldn't Digital like to show up everytime someone searches for IBM :->

barbara (199.93.126.34) - 12:28pm -- What one would like to do and what one with a conscience would actually do are different. I don't work for Digital, but I'm sure they are too ethical to put a competitors name on their page.

Richard Seltzer (199.3.129.189) - 12:25pm -- Regarding Metatags -- AltaVista recognizes two kinds -- (details can be found in the on-line Help) One kind gives you the opportunity to state exactly the words that you want to appear as the description in the results list. (Keep in mind that the default is the first couple lines of text that appear on your page. If your page clearly states what you are up to, there is no need to mess with metatags. And if it doesn't, you should take another look at your page design.)

Richard Seltzer (199.3.129.189) - 12:28pm -- Second kind of metatag allows you to add "key words" -- in other words, if key terms that are important to your business and that people are likely to search for do not appear in the text of your page, you can add them in a metatag. But once again, if those words don't appear on the page, you should think twice about maybe editing the text to include them. Some people reportedly have begun to abuse this variety of metatag by including their competitors' names as key words. That practice is of questionable value and also could lead to legal complications if your intent is to confuse or mislead the user.

Todd (192.208.46.249) - 12:35pm -- Richard - I have a page with JavaScript that metatags would help on. Right now it shows the beginning of the script as the description. Also the second kind of metatag would be good for synonyms.
Richard Seltzer (199.3.129.189) - 12:40pm -- Todd -- Yes, metatags or descriptors can let people know about the content of pages that are primarily graphics or Java etc. That's a good use of them. Also, synonyms make sense as keywords. Keep in mind too that if someone's name is important, you might want to enter variants of it as key words as well -- these search engines don't recognize that Robert is the same as Bob or that Seltzer, Bob is the same as Bob Seltzer unless the person entering the query follows a strict format and enters all the likely alternatives (which doesn't happen very often.)


ROBOT EXCLUSION

Richard Seltzer (199.3.129.189) - 12:44pm -- Regarding "exclusion" keep in mind that just because you have no links to a Web page that's under construction doesn't guarantee that it won't be fetched and indexed. Some server software has a "directory indexing feature". In that case a robot like "Scooter" can sniff out absolutely everything on your Web server -- with or without links. To avoid that, you should either disable that directory indexing feature or add a robot exclusion command that keep robots away from the directory or file that isn't ready for public viewing.

Warren Agin - Law Solutions (199.3.142.80) - 12:45pm -- Or files you don't want the public to find. I often post stuff such as pictures for friends, but don't link it to my primary webpage. 


IS ALTAVISTA REACHING ITS LIMITS?

Todd (192.208.46.249) - 12:24pm -- Richard - I've been using AltaVista for a while and am generally very happy with it. Lately, however, it seems to be missing what I'm looking for occationally. For instance, I was looking for a friend whom I eventually found at Marquette U., but Alta Vista wasn't finding him. I don't believe his page is new. Is AV having trouble covering the Web at this point?

Todd (192.208.46.249) - 12:46pm -- Richard - I had post a question earlier that got buried in the heat of battle: why might AV miss a page other than that it's too new? Are you confident that it is covering the web, or has it grown to the point where that's no longer feasible? How do you know how well AV is doing?

Richard Seltzer (199.3.129.189) - 12:51pm -- Todd -- To the best of my knowledge, AltaVista is still fetching everything that's public. They keep throwing more and more hardware at the project, and it's amazing how fast the response rate is for over 21 million hits a day. Keep in mind how the robot works -- it fetches a pages, sniffs out all the links on the page and then goes to all those links. So if a page is not pointed to from other pages, or has very few pages anywhere in the world that link to it, it may take a while for Scooter to randomly get around to it. That's why it's a good idea to submit the URLs of your new pages. That way Scooter retrieves the pages immediately and they are fully indexed in about a day.


HOTBOT

Warren Agin - Law Solutions (199.3.142.80) - 12:31pm -- Richard, how about HotBot. I hear it has more material now than AltaVista but my experience is that it is deadly slow. How is that engine structured?

Richard Seltzer (199.3.129.189) - 12:33pm -- Warren -- I haven't tried HotBot yet, and hence don't know how it's structured. (Once you get hooked on one search engine and are comfortable using it to find what you want, it takes something major to induce you to change.)

barbara (199.93.126.34) - 12:49pm -- Todd, I just looked for something under hotbot and got no response. I did the same with Alta Vista and got 10,000. True it's too much to sift through for what I want, but they did give me some answers while hotbot did not.


FLYPAPER

Richard Seltzer (199.3.129.189) - 12:55pm -- Quick thought -- keep in mind the "flypaper" strategy. There's an article on it at my site, in the latest issue of my newsletter Internet-on-a-Disk, http://www.samizdat.com/news18.html

Richard Seltzer (199.3.129.189) - 12:58pm -- Briefly about "flypaper" -- I was getting 2-3 email messages per week from old acquaintainces I hadn't heard from in 20-30 years. It finally occurred to me that they weren't looking for me (any more than I'd be likely to do a search for them). On inquiring, I discovered that they were looking for themselves. In fact, that is by far the most common thing that people do when they connect to a search engine like AltaVista -- they look for themselves. I had so much text at my site, covering so many of the things I've been involved in in my life, that these people were mentioned there -- they found me by looking for themselves. Just keep that psychology in mind when designing Web pages -- make sure that your target audience/customers can find themselves or the topics that are of most interest to them somewhere at your site.


WRAPUP

Richard Seltzer (199.3.129.189) - 12:54pm -- Quick check -- I got lots of email from people saying that they wanted to talk about "new kinds of money", but they don't seem to have showed up today. It feels like there's a fair amount to say about search engines and design, so I'm inclined to continue that next week (while remaining open to further "virtual money" questions). Reactions?

anonymous (199.3.134.192) - 12:52pm -- Hi...I think it's crucial as web designers that we understand how the various search engines are constructed. That means reading all the FAQs and Help Files...and even understanding what librarians know so well-- how to find info

barbara (199.93.126.34) - 12:58pm -- Anonymous has a good point. You do have to read up on the individual search engines and directories to know what they specifically want and how to submit to it. It took me a long time to get my submission to Yahoo in the right categories.

Carol J. Snyder (snyderinfo*Web Design) (199.3.134.192) - Thu, Nov 7, 12:55pm -- Hi, Richard, et al. Definitely! The topic of Search engines have a lot to discuss.

Tom Dadakis DadaCom tomdadak@ix.netcom.com (207.93.47.66) - Thu, Nov 7, 12:57pm -- I think people will have to learn to do more selective searches so they can find the information they are looking for. A list that returns 10,000 entries doesn't do anyone any good. How can we make more selective searches?

Carol J. Snyder (snyderinfo*Web Design) (199.3.134.192) - Thu, Nov 7, 12:59pm -- People will need to understand Boolean logic (AND, OR, NOT) in doing searches and how to limit...all things that Information Specialists/Librarians do everyday... or there will be a market for info providers...(as search engines grow, they're too unwieldly...)

Richard Seltzer (199.3.129.189) - 1:02pm -- Carol -- I try to demystify that AND/OR complexity in the book The AltaVista Search Revolution. I can get into it more next week if there is interest.

Carol J. Snyder (snyderinfo*Web Design) (199.3.134.192) - 1:01pm -- www.snyderinfo.com (cjsnyder@snyderinfo.com) (was anonymous in this discussion) cheering for "knowing how to think like InfoSpecialists/Librarians think"! Info is Power

Tom Dadakis DadaCom tomdadak@ix.netcom.com (207.93.47.82) - 1:01pm -- see you next week

Richard Seltzer (199.3.129.189) - 1:00pm -- All -- Time's running out. Thanks to all for joining us. Please post your email addresses and URLs for followup.

Also, please send me email seltzer@samizdat.com with your followup messages/questions and check for the transcript of this session later today http://www.samizdat.com/index.html#chat

Thanks to all for your participation. Richard


Previous transcripts and schedule of upcoming chats -- www.samizdat.com/chat.html

To connect to the chat room, go to www.samizdat.com/chat-intro.html

The full text of Richard Seltzer's books The Social Web, Take Charge of Your Web Site, Shop Online the Lazy Way, and The Way of the Web, plus more than a hundred related articles are available on CD ROM My Internet: a Personal View of Internet Business Opportunities.

Web Business Boot Camp: Hands-on Internet lessons for manager, entrepreneurs, and professionals by Richard Seltzer (Wiley, 2002). No-nonsense guide targets activities that anyone can perform to achieve online business
success. Reviews.

a library for the price of a book.

This site is Published by B&R Samizdat Express, 33 Gould St., West Roxbury, MA 02132. (617) 469-2269. seltzer@samizdat.com


Return to B&R Samizdat Express


<


Internet Business Showcase: