Inktomi believes it has a solution with a new “cluster” of European search servers. The cluster is hosted in the UK by BT, which will market Inktomi’s search engine as part of its package of Internet services.If Inktomi can provide better Web searches, it will find a fertile market: research carried out for the company this month by Mori revealed that half of all UK Internet users were unhappy with the performance of their preferred search engine. The most common complaint is that search engines fail to produce relevant “hits”.”If a first-time user goes to the Internet and wants to look for a company, they may not get the company website right away,” Peterschmidt admits. “But more and more the algorithms and the way that we help people find information mean they should be able to find it even if they are relatively new to the Internet.”Running a search engine is a complicated and increasingly automated process. In the early days, directories such as Yahoo! were created largely by hand. Human editors hunted on the Net for interesting sites and added the ones they liked to the listings. The sheer scale of the Web means that is no longer an option.

Inktomi, along with other search companies, relies on automated “Web crawlers” that search the Net for sites and list their contents. The search engine then uses this information to build a database using keywords and mathematical algorithms. This database then responds to Web users’ search requests with a list of documents, in descending order of relevance.A search site’s effectiveness depends on the efficiency of the Web crawler and of the indexing software. Inktomi’s Web crawlers are designed to visit each site at least every 30 days. Ideally, says Peterschmidt, they should go back every 10 days. This is no easy task.”There is a lot more content all the time,” he explains.

“The challenge is finding and indexing that, but maintaining relevancy. It has to bring you documents that are germane to what you are looking for. That requires continual research and development.”Users are becoming more experienced, and technology more sophisticated, leading Peterschmidt to believe that the Internet is becoming both more useful and easier to use.”There are better search results than 12 months ago, even though the Web’s content is four times larger,” he says. “We are getting much better at finding the fine grain of information. We are also learning a lot from popularity: what are the most frequently visited sites? That helps us understand what users want, and how they want it to look and feel.”Inktomi’s challenge goes beyond keeping ahead of the Web’s incredible growth. Search engines are also locked in a constant struggle with website developers, who want their clients’ pages to appear at the top of visitors’ search results – whether or not the content merits it.For David Peterschmidt, it is an ongoing game of hi-tech double bluff. His information scientists and software engineers develop ever more sophisticated ways to search and rank sites, and Web designers respond with tricks, ranging from pages with nothing but keywords to hidden “meta-tags” to fool the crawlers into putting their sites top of the list.”There is a constant gaming that goes on between our technologists and the people who write the content,” he says “The people writing content …