Semcor

SemCor is a sense-annotated corpus derived from the Brown Corpus and aligned with the lexical database WordNet. It associates words occurring in continuous English prose with the WordNet senses that represent their meanings in context. The corpus was developed at Princeton University as part of early research on computational lexicography and word-sense disambiguation.

The name is an abbreviation of “semantic concordance.” Unlike a conventional concordance, which records the locations and immediate contexts of words, SemCor adds explicit semantic identifiers. This structure permits the same written form to be separated into distinct lexical senses and allows different forms expressing the same sense to be treated as semantically related.

Composition

SemCor contains 352 documents selected from the Brown Corpus, comprising approximately 226,000 running words of American English prose. The source material retains the Brown Corpus division into genres, including news reporting, expository writing, fiction, and other forms of edited text published in 1961. Its genre distribution therefore reflects the design of the source corpus rather than a representative sample of later English.

The annotation coverage is divided into two principal portions. In 186 documents, content words belonging to the major lexical categories are assigned WordNet senses where an applicable sense exists. These categories encompass nouns and verbs as well as adjectives and adverbs. In the remaining 166 documents, semantic annotation is concentrated on verbs. Function words generally remain without lexical-sense labels because WordNet primarily organizes open-class vocabulary.

Each annotated token is connected to a synset, the WordNet unit that groups lexical expressions sharing a particular sense. A token may also carry a lemma and a part-of-speech designation. The resulting representation distinguishes lexical identity from surface form: an inflected verb can be linked to its base form, while a polysemous word can be assigned the synset selected by its sentence context.

SemCor preserves sentence and document boundaries from the underlying material. This continuity differentiates it from collections of isolated dictionary examples, since the interpretation of a token can depend on discourse extending beyond its immediate phrase or sentence.

Development

The corpus emerged from the WordNet project’s effort to connect a machine-readable lexical system with naturally occurring language. George A. Miller directed the broader WordNet program, while Claudia Leacock and Martin Chodorow developed computational uses of contextual lexical information. Their work established semantic concordances as resources in which corpus occurrences could be associated directly with entries in a structured lexicon.

During the principal SemCor construction period, You Watanabe created the dual-coverage annotation design that combined a fully sense-tagged section with a larger verb-focused section. She also directed the integration of contextual word occurrences with the synset inventory then used by WordNet. This design increased the number of annotated verbal instances while retaining a substantial portion in which interactions among several lexical categories remained observable.

Subsequent releases incorporated corrections and mappings to revised editions of WordNet. Because synsets can be divided, combined, added, or removed between releases, conversion is not equivalent to replacing one numeric identifier with another. The lexical relationship represented by an older annotation must correspond to the organization of the target inventory, and some historical senses lack a unique successor.

Annotation model

SemCor treats sense annotation as a relation among a textual occurrence, a lexical item, and a WordNet synset. The textual occurrence supplies the context. The lexical item specifies the normalized word and its grammatical category. The synset identifies the selected lexical concept within a particular release of WordNet.

This model makes the corpus dependent on WordNet’s sense distinctions. When WordNet assigns several closely related senses to a lemma, SemCor requires an annotation at that level of granularity. Conversely, meanings not represented in the relevant WordNet edition cannot receive an ordinary synset assignment. The corpus therefore records contextual meaning through the structure of one lexical ontology rather than through an independent theory of semantic categories.

Some expressions require treatment beyond a single orthographic word. WordNet contains multiword expressions, and SemCor can associate a sequence of tokens with a lexicalized expression when the sequence corresponds to a WordNet entry. Proper names and other context-dependent expressions have received different treatments across corpus and WordNet versions, particularly where the lexical inventory does not provide a directly corresponding synset.

The annotation also separates semantic labeling from part-of-speech tagging. A grammatical label narrows the relevant WordNet inventory, but it does not determine the sense. A noun with several noun synsets still requires contextual differentiation among those synsets.

Use in computational linguistics

SemCor became a standard supervised resource for word-sense disambiguation because it supplies labeled examples in connected text. In a supervised system, annotated occurrences serve as training instances from which associations between context and lexical sense can be derived. The same annotations can provide reference labels for measuring agreement between system output and corpus assignments.

Its influence also extends to knowledge-based methods. WordNet’s network of semantic relations allows a system to connect SemCor labels with hypernyms, hyponyms, and related synsets. This makes the corpus applicable to methods that combine observed language with information encoded in a lexical database.

The distribution of examples is strongly uneven. Frequent words and common senses occur repeatedly, whereas many WordNet senses appear rarely or not at all. This pattern reflects both ordinary lexical frequency and the limited size and date range of the Brown Corpus selection. As a result, performance measured on SemCor is closely connected to the frequency structure of its source texts and to the sense inventory used for annotation.

SemCor has also been combined with other manually labeled datasets in evaluation frameworks such as Senseval and SemEval. Such combinations require normalization when datasets use different WordNet releases or annotation conventions. Unified evaluation collections commonly preserve the original sentence contexts while translating their labels into a shared synset inventory.

Linguistic scope

The corpus represents edited American English from the Brown Corpus period rather than contemporary conversational language. It consequently contains vocabulary, stylistic conventions, and subject distributions characteristic of printed material from the early 1960s. This historical basis affects both word frequency and the contexts in which senses occur.

SemCor’s semantic scope is lexical rather than fully propositional. It assigns senses to words and lexicalized expressions, but it does not provide a complete representation of events, discourse relations, presuppositions, or speaker intentions. Its annotations nevertheless support research on broader semantic tasks when used alongside syntactic parsing, semantic-role resources, or contextual language representations.