At first, the thought of developing a custom search engine might sound like a colossal mistake. Competing with Google’s almost limitless engineering resources can’t end well. However, there are search tasks where a custom search engine might be a better fit than Google. While Google controls much of the web search, people often prefer to find more focused answers to their queries than just a list of links. A person who is shopping for home supplies may search directly on an online shopping site instead of first visiting Google, because the results on the web shop are more targeted. Indeed, according to The New York Times, Amazon controls a larger share of shopping searches than Google.
Topical search sites specialize in providing the best possible answers on one specific topic. They aim to offer a better user experience, for example, by pruning nonessential results. A movie search engine only lists recommended movies to watch, a people search engine helps find contact information for friends, and a news search shows the latest news items arranged by theme. On a general web search engine information may get buried among irrelevant links.
A topical search engine is often able to offer superior search tools that are targeted for the task. On a hotel search engine, the results can be filtered by price, by rating, and by the distance to nearby attractions. A similar level of filtering is next to impossible on a web search engine where one can only input text. Sometimes data may lend itself to be explored through maps, timelines or other advanced visualizations. A customized search interface is often more suited for this kind of structured data.
Topical search engines usually have diverse filtering options.
Many mobile apps are essentially topical search engines. Apps can search for nearby restaurants or book hotel rooms. On mobile, the need for an effective search interface is perhaps even more pronounced that on desktop, because of less available space and generally clumsier interface for switching between apps and web pages.
Another reason for building a custom search engine might be that the data is not publicly available on the web; for example, searching in user’s private email or chat logs, or querying data that is made available only to paying customers.
Building your own search engine
If you decide to build a search engine, you don’t have to start from scratch. Many search companies offer enterprise search for organizations. An enterprise search service automatically indexes intranet and other organizational resources and provides a single search box for all the data. Google Custom Search Engine and Yahoo Search BOSS make it easy to build focused search engines that return results just from hand-picked sites. For example, a search engine that only covers a few reliable music review sites can guarantee to return only relevant results for somebody interested in reading music reviews.
Often more control over the search is needed. In that case, search platforms, such as Elasticsearch and Solr, are better alternatives. They can be used to integrate customizable advanced search features into any application. Both Elasticsearch and Solr offer fine-level control of how results are ordered, versatile querying and filtering capabilities, autocompleting of search terms, spelling correction and lots of other features that are expected of a modern search engine.
Future and conclusions
As the amount of data produced by humans, integrated sensors and gadgets continues to grow, there will certainly be an ever increasing need for search engines to make sense of it all. Therefore, it is worth paying attention to advancements in search technology.
Future trends might be consolidating different topical search engines into a meta-search engine. Instead of having separate apps for different kinds of searches, there could be one site that infers which topical search engine is best able to answer the query and relays the query to it. Google is already starting to do this when they are able to infer the intent from the search terms. For example, searching for “flight Helsinki to New York” on Google will show a timetable of departing flights above the organic results.
Google shows special result view for certain types of queries
Not all search problems are solved by general web search engines. A custom search engine is useful if it can provide a better user experience for a specialized tasks or when the data is private. Developing a custom search engine is feasible thanks to available tools that hide many of the inherited complexities.
Antti AjankiLead Data Scientist
