To get Wikipedia Miner up and running, you have to download and host a huge amount of information. We've created the services below to make things a bit more convenient.

The Search service allows you to treat Wikipedia as a gigantic thesaurus, for describing everything from nanotechnology to Barbie dolls. The Wikipedia articles this service locates provide a wide array of useful linguistic information, including definitions, synonyms, translations, and related topics. The search vocabulary is extensive (5 million or more terms and phrases), and encodes both synonymy and polysemy.

The Compare service allows you to compare terms and concepts to measure how strongly they relate to each other. From this you can tell that nanotechnology doesn't have much to do with Barbie dolls, but that it does have a lot in common with engineering.

The details of how this works (and an evaluation) can be found in this paper:

• Milne, D. and Witten, I.H. (2008) An effective, low-cost measure of semantic relatedness obtained from Wikipedia links. In Proceedings of the first AAAI Workshop on Wikipedia and Artificial Intelligence (WIKIAI'08), Chicago, I.L.

  This paper describes a new technique for obtaining measures of semantic relatedness. Like other recent approaches, it uses Wikipedia to provide structured world knowledge about the terms of interest. Our approach is unique in that it does so using the hyperlink structure of Wikipedia rather than its category hierarchy or textual content. Evaluation with manually defined measures of semantic relatedness reveals this to be an effective compromise between the ease of computation of the former approach and the accuracy of the latter.

  

The Wikify service automatically augments either snippets of text or entire web pages with links to relevant Wikipedia topics. It doesn't just use Wikipedia as a source of information to link to, but also as training data for how best to do it. In other words, it has been trained to make the same decisions as the people who edit Wikipedia.

This paper describes how the wikifier was implemented and evaluated:

• Milne, D. and Witten, I.H. (2008) Learning to link with Wikipedia. In Proceedings of the ACM Conference on Information and Knowledge Management (CIKM'2008), Napa Valley, California.

  This paper describes how to automatically cross-reference documents with Wikipedia: the largest knowledge base ever known. It explains how machine learning can be used to identify significant terms within unstructured text, and enrich it with links to the appropriate Wikipedia articles. The resulting link detector and disambiguator performs very well, with recall and precision of almost 75%. This performance is constant whether the system is evaluated on Wikipedia articles or "real world" documents.

  This work has implications far beyond enriching documents with explanatory links. It can provide structured knowledge about any unstructured fragment of text. Any task that is currently addressed with bags of words—indexing, clustering, retrieval, and summarization to name a few—could use the techniques described here to draw on a vast network of concepts and semantics.