Search Engine Review for Development Researchers
 
Cecilia Hogan, Development Researcher
University of Puget Sound
Association of Professional Researchers for Advancement Conference
San Antonio, Texas      ~     May 1998
 ================================
"The days when you could rely on a single search site for all you needs are gone."
- PC Magazine, December 2, 1997
 
First, a bit of drivers ed:

What's going on here?
This presentation is about relational or text-based database searching on the Internet.  Flat files (phone directories) and directories (like Yahoo) will not be covered.

Choose the right tool for the job.
This is the toughest part --- pick the right engine for your research task.  The best tool for the job isn't always the biggest or easiest to use.  It isn't always the one everyone else - especially the general public - is using.  Windweaver tells you which engine to use based on your search goals at
http://www.windweaver.com/searchguide.htm#WHICH

Keep the toolbox handy.
As PC Magazine notes, one tool won't do the job anymore.  Develop a collection of engines you understand and use them.

Map your course before you leave home.
Experienced researchers talk about the complex searches executed through fee-based electronic services.  Borrow some of this good research behavior by making a plan for your Internet search before you begin.

Go ahead.  Kick the tires.
It's tough to keep up on all the engine news, so run your own tests periodically.  Or, even better, subscribe to Search Engine Watch, the premier tire-kicker, at
http://www.searchenginewatch.com

Take 'er in for a diagnostic:  Engine Surveys
PC Magazine: Your Complete Guide to Searching the Net (12/02/97) is the best yet at
http://www.zdnet.com/pcmag/features/websearch/edchoice.htm

Terry Mertz collects engine reviews, tips and more at
http://www.library.carleton.edu/staff/terry/websearch/

The surveys usually reflect the general public's engine interests on the Internet.  How do the results of these surveys fit your more sophisticated professional needs?  Of the features rated, what really matters to you in your work?


 Pit Pass: Comparing the Top Engines for Prospect Researchers
 a brief outline of the features of the major players
 
         . AltaVista Excite HotBot Infoseek Lycos Northern      
Light
Size (1) /(2) 100M/110M 55M/33M 110M/100M 50M/35M 30M/NA 67M/NA
Portal     
Status
Quotes     
Categories
Free email     
     
     
 
Quotes     
News     
Categories 
Weather     
Channels     
Personalize 
    ---- Quotes     
News     
Channels     
Worldwide 
     
 
Quotes     
Categories 
Weather     
News     
Personalize
Free 
email/page
  ---
Search
Area
     
Options
Entire Web 
Business     
People     
Usenet     
    
  
 
Entire Web     
Directory     
News     
city.net     
Usenet    
  
 
Entire 
Web    
 
New Sites 
People     
Top New Sites    
Categories    
 
Entire Web 
Directory
(Companies)

News     
Newsgroups    
  
 
Entire Web 
Image/Sound 
Directory     
 Top Sites    
City Guide    
  
 
Entire 
Web    
 
All sources     
Special      
 Collection 
     
 
Boolean
& Other 

Advanced
Search
Options
AND   &  
OR   | 
AND NOT ! 
NEAR  ~ (10) 
* = wildcard 
(precedence)
"exact 
phrase" 
+/-      
 
 
 

  

AND 
OR 
AND NOT 
(precedence) 
+/-  
 
 
 
 
 
 
 
 
  

   

Graphical: 
AND   &    
OR  | 
NOT  ! 
All words 
Any words  
(precedence) 
"exact 
 phrase"    
+/-    
 
 
  

 

AND 
OR  
AND NOT 
"exact 
 phrase" 
+ two  
   words     
    = maybe 
Comma 
+/-     
 
 
  

 

AND   &    
OR   |    
NOT   !    
NEAR (25)  
FAR (25+)     
BEFORE     
ADJ 
O:prepend 
ADJ 
NEAR/# 
(precedence)
"exact 
phrase"  
+/-  
AND 
OR 
NOT 
(precedence)
"exact 
phrase"  
+/- 
 
 
 
 
  

    
  

Other       
Refining  
Options
Case 
sensitiv.      
Ranking      
Date range   
 dd/mmm/yr  
Language  
Multinat char. 
   --- Date range     
   after, 
before,    
   within 
dates    
Page title    
Links to url  
Person
Case 
sensitiv.     
Person     
     
     
     
 
Pro w/ 
Power:     
drop downs  
for any words,
natural lang.,
exact phrase

Language 
   ---
Special     
Function 
Searches
ANCHOR:  
DOMAIN:  
HOST:      
IMAGE:      
LINK:      
TEXT:      
URL:
   --- Domain     
Continent     
feature:     
linkdomain:
title:     
     
 
link:     
site:     
title:     
url:     
     
     
 
title:    
url:    
    
    
    
    
 
TITLE:     
PUB:     
COMPANY:
TICKER:     
TEXT     
URL:     
 
Special     
Features
Refine 
with     
 
Cow9     
     

International
Search 
Sites    
 
     
     
     
 

ICE: Infer link
btwn words. 

International
Search 
Sites   
 

Power 
search    
 
(w/o Boolean)     

Search wizard

Super 
Search   
 
    
    
    
    
    
    
    
    
 
ESP:  Extra
Search P
recision   
 

Search by 
country    
    
    
    
    
 

Search 
Expressions Guide   
 

International
Search 
Sites   
 
    
    
    
 

Enterprise  
     Accts.     
     
Glossary of terms     
     
     
     
     
   
 
Results     
Options
Format     
Language 
Detail      

 

By 
relevance    
 
By Web 
site   
 
More like
  this

 

Page depth
Time 
period    
 
# of results
Presentation
 
Presentation
    
    
    
 
Ranking by:
word, 
depth,  
 
frequency, 
order, 
location   
 
More like 
  this
Folders:    
Subject   
Type   
Source   
Language 
Other     
Features
   --- Excite Direct    ---    ---    ---    ---
       . AltaVista Excite HotBot Infoseek Lycos Northern     
Light
 
(1) Reported (2) AltaVista research, all dated March-April 1998


Places to go, people to see (and some mapping help):

A few search tips . . .

Dyno-Power!  Run more than one engine at once!
So, we have to use more than one engine for each search.  Why not turn to the meta-searchers, engines using several engines at once?  If this is your philosophy, be sure to check out PC Magazine's take on this.  Search operators vary from engine to engine, so a meta-searcher must have the capacity to dynamically convert syntax to be effective.  PC Magazine outlines which ones do this at

http://www.zdnet.com/pcmag/features/websearch/ms.htm

Visit the Engine Collectors:
Some people have done amazing jobs of collecting search engines.  Here are a few:

Internets.com - http://www.internets.com
Search "1000's of databases" here.  Internets mission is to be the "largest, fastest and most recognized global link to online resources including archives, libraries, research databases, catalogs and statistical data.  Access MedNets here.

Web-Search - http://www.web-search.com
"Why choose between search engines when you can have them all?" Web-Search asks.  Despite the lofty come-on, Web-Search's approach might interest researchers.  It's indexing the search engines on individual and corporate sites.

Brain Forest - http://www.brainforest.com/search.htm
No descriptions here but there are more engines than we can ever count.  The purpose of some can be inferred.  Others will take a slow day to explore.

Windweaver - http://www.windweaver.com/searchtools.htm
Windweaver has a modest collection of engines but outlines a few directories and libraries, too.

Beaucoup - http://www.beaucoup.com
Beaucoup links to thousands of engines, directories and indices.  Go to the Big List for a collection of international engines.

And your specialty is . . . ?
Looking for regional, international or specialty engines?  Go to Amanda Harizan's  collection at
http://web.ntu.edu.sg/library/specialframe.htm

Visit IFINDIT's collection of regional search engines at
http://www.ifindit.com/regional_seaches.html

The Exotics:
Here are a couple of search sites that put a different twist on searching --

Human Search - http://www.humansearch.com
With an index now of questions it has answered, Human Search is even more intriguing.  Touted as the "human-based search engine," Human Search answers questions posed by visitors.

Ask an Expert - http://www.askanexpert.com/askanexpert
Ask an Expert gives you a search engine and then categories as the first places to look for answers.  If that fails, you email their stable of experts.  For example, you may ask the Cuba Expert, Banking Expert or even the Fundraising Expert.

What's next?
At this writing, the advent of portals are the rage.  Lots of deals being closed, plenty of alliances being made.

The claim is that the focus is shifting from indexing the entire Web to providing the kind of content which attracts visitors.  This is bad news for researchers because of course we want it all indexed, right?  Their content choice won't be our content choice.

The latest?  Stickiness - how long visitors stay at a site.  It's a concept that is affecting the price of ads.

Oh, and rumor has it that Microsoft is going to come out with something bigger and better . . . ..

                              "Warning - optimism ahead."

                     - Jesse Berst, ZDNet Anchor Desk, May 1998

Jesse Berst's ZDNet Anchor Desk interviewed Dr. Joseph Bordogna, Acting Deputy Director of the National Science Foundation on May 21, 1998.  NSF got us where we are today, you might say.  Dr. Bordogna talked about the future of the Internet as a time when "knowledge will be available to anyone, anywhere, anytime." He uses the term "knowledge" instead of information to "extend the concept beyond mere data to relevant information in a useful form." (Researchers may wish to stand and cheer now).

Headers, Spoilers and Dual Exhausts
Downloadable software products which will improve searching -- they're here.  Many of you are already using the Electronic Monk (http://www.electricmonk.com) or Alexa (http://www.alexa.com). Keep your eyes open for more tools like these.

Time for the Zen of researcher-type Web searching.
So, for the time being, we researchers relax and ride along with the current state of Web searching.  We accept that this is not the world of fee-based searching. We are happy to be living in such an exciting time electronically.  Hmmm . . .

But we keep our good research eyes on the issues facing the evolution of Web searching.  What influence will the marketers have on searching?  We know they want to bump their sites up in the indexing by pointing robots to their sites for quicker indexing.  You know about 'stickiness,' now a measurement for advertising revenues and perhaps driving the development of portals, channels and other features.

How functional portals will be for prospect researchers is still to be determined.  Personalizing a portal to meet the needs of professional researchers isn't likely to be in the lineup at this early phase.  But as this concept develops, we may begin to approach the kind of search capability researchers seek.
 

"There is often relief in change . . . . As I have often found traveling in a stagecoach,
it is a comfort to shift one's position, and be bruised in a new place."

                                         - Washington Irving (from Stephen Stanoudis' Quotations)


Glossary (of Some of the Terms Used in this Presentation/Web Site)

Boolean Operators:  a search language which allows you to indicate how your search terms relate to one another.  Relational concepts include nesting, truncation, proximity searching, inclusion and exclusion and others.

Database:  a computer file or system of data organized in records or fields for fast retrieval and ease of updating. The two types of databases researchers encounter on the Internet are flat files (phone books) and text or relational (two or more databases linked) databases.

Directory:  a hierarchical database organized by subject.  Directories are not machine-indexed but are divided into categories instead.

Dynamism (Dynamically, Dynamic): a term used to describe a machine replicating a human-like relational leap.

Library:  a directory or directories of online databases or files.

Portal:  an entrance to the Web;  a starting-off point for Internet users.

Robots (or spiders):  agents which crawl the Web searching for new or updated Web pages.  These agents record the entire text of every page within a Web site they visit.  The agent then visits all the external links noted at the site.

Search Engine:  a program that retrieves links to documents and other information on the World Wide Web based on a query posed by a user.

Stickiness: a measurement of market appeal, this term refers to the site characteristic of keeping visitors for a length of time.  It forms a point of gauging advertising fees.

Supersite: an integration of Internet, Intranet, Extranet and E-commerce sites to support business and communication networks. (For an example, go to http://warnerchappell.com/)

Syntax Adaptation: an application of dynamism where meta-searchers convert a single search string to the specifications of two or more search engines so that the engines may effectively complete the search.

Web site rating services: companies whose business it is to measure Web sites on a variety of scales, compare them and report.  Rating services are having an affect on advertising revenues.