This is probably not the primer you are expecting. So don’t skip this, as what you read hear is quite interesting.
I’m going to explain a little known fact about the internet so you can become a much more efficient researcher. This little fact is about how Google is useless 90% of the time. Why? Because 90% of the internet is Deep Web and Google can’t index it to show you resources you need. So let me say that again because it is an important concept to you, Google can not find deep web resources.
I’ll be covering both the Surface Web and the Deep Web. If you’ve read books before about searching using the internet, this will be a new twist for you. I’m not going to hold back any punches, I’m just going to tell it like it is.
A lot of How-To-Research tutorials simply cover the “Well, type in these keywords into Google and…” which doesn’t get you very far. We have to go beyond that and really dive deep into the research assets of the internet.
To be fair, the shallow web does hold some minor resources we need. The Deep Web, however, has many more and they have higher ‘authority rank’ with us. I’ll explain authority rank in depth later.
What is the Deep Web?
Google just doesn’t cut it for good research. It is a Surface Web search engine. If you have no idea what surface web is, you’ve come to the right place.
The internet is divided into two parts, Deep Web and Surface Web. If a web page can be seen by an indexing bot, for example, Google, then that is considered Surface Web. If Google can’t index it, therefore can’t display it as a result, it therefore is Deep Web.
” Deep Web resources are higher quality, higher authority, and a boon to researchers.”
Why should you care? Because Deep Web resources are usually higher quality, higher authority, and a boon to researchers, be they high energy physicists or treasure troving cache hunters.
Maximizing Shallow web
There are many search engines, but they all do the same thing. They crawl the web and index what they find. There are a few big ones, like Google, Yahoo, Bing, and Dogpile. Personally, I’m fond of Zuula which searches the big ones and gives the result in one page.
There are many more smaller boutique search engines that specialize, like Pipl, used to locate facts on people, and Deeperweb.com, and others. Each has a strength, these should be found and added to your Source Book.
Almost every search engine uses some form of boolean logic to help you narrow your search:
- If you wanted to find just keywords on a single site on Google, say the word ‘books’ on treasurenet.com use [books site:treasurenet.com] (but without the brackets)
- If you wanted keywords in a specific order ie “Black and Tan” when looking for beer use quotes, like [“black and tan” beer]
- If you don’t want certain words in your results use a minus sign, no space, and the word. ie, we don’t want results for tanning salons so we use [“black and tan” beer -tanning]
- If you want university results put an ‘inurl:’ like [Montana historical map inurl:”.edu”]
