Applications Of N-Grams
Cultural Trends
One of the first applications of n-grams was research into cultural trends through the large-scale analysis of printed material from a large corpus from over five million books obtained from the Google Books project. The work was performed by a collaborative team of researchers from Harvard University, Google, and elsewhere. The corpus contains over 500 billion words in seven languages – English, German, Hebrew, Chinese, French, Spanish, and Russian.
Subsequent analysis was performed on the resulting n-grams. The mean percentages of overall frequency for words and phrases by year, based on different criteria were computed and used in subsequent statistical analysis. The authors demonstrated this approach with the word “treaty”. The mean percentage of overall frequency against time (increasing years), relative to treaty signature, was plotted and analyzed. The same approach was used to analyze heads of state relative to their accession to power, as well as changes in the names of countries over time (Michel et al., 2011).
Word Usage in Novels and Published Literature
In the article Digital Humanities, Big Data, and Ngrams, University of California, Berkeley sociologist Claude S. Fischer reports on some recent studies, including one by historian Marc Egnal analyzing the evolution of the American novel (Egnal, 2013), employing n-grams to research into the dynamic (time-varying) patterns of word usage and how they are reflected in cultural trends.
Some interesting findings are that the word “gentleman” appears approximately 130 times more than the word “guy” in books from the early part of the twentieth century. By 2000, however, there was a shift: the word “gentleman” was only used about one-third as often as “guy”. Another observation was that the word “I” at the start of a sentence, indicating the subject of the ensuing action, was about 70% of that of the words “He” or “She” in both 1970 and 2000. However, there was a drop in the ratio of “I” to “You” at the start of a sentence over the same period, possibly indicating that “I-absorption” is not as common as previously thought. Additionally, the word “vampire” occurs about 10 times more frequently in 2008 than in 1950, the cultural implications of which Fischer leaves open.
However, as Fischer points out, n-gram analysis comes with some challenges. First, there is the issue of which material to use. This human curation of sources, as explained in earlier sections, always involves a subjective element. Beyond curation, the books that survived, and therefore could be scanned and subjected to quantitative analysis, are not the entirety of books that were published. As a large component of the material scanned by the Google Books project is contributed from libraries, the curation of those library materials affects the results of any n-gram analysis. For instance, many libraries did not acquire books that contain crude language or obscenities, and therefore, th