Keyword (or keyterm) extraction is the task of automatically identifying the terms that best describe the contents of a document. It is the most important step in generating the Subject Indexes that you can find in the back of many of our publications.

Up until now we’ve been using Sketch Engine to accomplish this. Sketch Engine is proprietary software that offers many pre-compiled corpora to compare your own texts against. For example, we would upload the raw text of a new book and compare it against the British National Corpus, an extensive collection of various English language texts from the late 20th century. Since linguistic publications usually have a vocabulary that is quite different from everyday language, Sketch Engine did a good job at filtering out the terms that were unique to our book – resulting in a list of keyterms such as language acquisition or vowel harmony.

So why fix something that isn’t broken? Because we are committed to using free and open software wherever possible, and also to making things easier wherever possible. We wanted our own, local, open source keyword extraction software. As an intern with Language Science Press, I was tasked with developing that software. It should work just as well while being easier to use and possibly more tailored to the linguistic domain.

Keyword extraction models

The following paragraphs will demonstrate different approaches to keyword extraction. For illustration purposes we’re going to look at terms extracted from a forthcoming volume of papers from the 49th Annual Conference on African Linguistics. This book represents a wide variety of linguistic disciplines while adhering to the latest LangSci standards, and since I was involved in typesetting I’m somewhat familiar with its contents and possible good key terms.

Sketch Engine

As you can see on the right, the top 20 results are generally good. Some terms need manual editing (e.g. complete countable mass to countable mass noun, count noun and mass noun – see chapter 14) while others should be removed, namely parts of linguistic examples (dā gbáŋ and buy book) and TeX commands (tex localbibliography) that appear even though we detex’ed the document!

YAKE!

YAKE! (Yet Another Keyword Extractor) is a keyword extraction algorithm developed by Campos et al. in 2018. There is a web interface and a Python package, among others, and using it couldn’t be simpler. Just paste your text on the website or type a few lines of code and out comes your keyterm list.

It theoretically works on both TeX code and detexed text, but using it on TeX code obviously results in a very cluttered keywords list. Therefore I compiled a stop words list that filters out the most common TeX commands.

To the right you can see the top 20 keyterms extracted from the raw TeX code (not detexed), using the default settings and my custom stop words list.

TF-IDF

This cryptic acronym stands for term frequency–inverse document frequency. I won’t go into the mathematical details here (see Wikipedia or a myriad of articles detailing the Python implementation for more info), but basically TF-IDF counts how many times each term occurs in a document as compared to other terms in the same document (term frequency) and weighs that number by how many other documents in the corpus also contain that term (inverse document frequency). Thus, a term that appears many times within our book but is rarely found in the reference corpus would receive a high TF-IDF score.

Here’s the result (using scikit-learn’s TfidfVectorizer with parameters adjusted to our needs, since the default values are useless here):

Simple maths

The last method I’m going to introduce is the benign-sounding simple maths method. Developed by the late Adam Kilgarriff in 2009, it is the basis for Sketch Engine’s keyword extraction model. It uses simple frequency ratios, i.e. how common a term is in one document versus another, but demands a computationally intensive normalization step that involves keeping track of all appearances of a term within a document.

Since first experiments with this approach did not yield more promising results than the models described above, I decided to leave it to the Sketch Engine servers and focus on the more efficient YAKE and TF-IDF.

Corpus

Before I go into details about the software I developed, I’d like to introduce the corpora that were generated to train the TF-IDF model. We decided to exclude some publications like dictionaries for which the keyword extraction method is generally not suited, as well as some non-English books. This selection is not final and may well be improved in the future.

The result is a compressed file ready for use with the software, as well as a detexed version of it, and can be found here.

The original detex software was quickly replaced with a custom version that is capable of mostly filtering out linguistic examples and tables. Though not perfect (it tends to “swallow” some paragraphs), it proved good enough for our purposes. Those interested can find the source code here.

The langscikw software

With the appropriate models selected, my task now was to piece them together to form usable software. I came up with a three-steps model:

1. Since YAKE provided the best results in and of itself, it should be the first step. This way, it is possible to use the software without a training corpus.
2. In a second step, the results of a TF-IDF model are added. We now have twice the number of keywords.
3. Finally, a different TF-IDF model trained on the detexed corpus extracts even more terms. These are compared against a corpus of 10,000 keywords that appeared in published Language Science Press books and are thus assumed to be reliable, since each keywords list is edited by the author or volume editor. Only those terms that have been selected as key terms before are kept.

The software then deduplicates the list and performs some cleaning steps like removing non-alphabetical results. This works so well that despite extracting many more terms than we need, the returned list is approximately the desired length. We usually extract a list of about 300 keywords, which is manually reduced to about 100–200 terms. This is especially important here because the software deliberately only extracts bigram keyterms – in our experience single keywords are best determined by human intelligence.

Now we have an open-source Python package as well as an easy-to-use command-line interface that operate locally, don’t require registration or a license, can be used on raw TeX code – in fact, no preprocessing is necessary at all –, and extracts keyterms from a 200,000 word document in about two minutes.

Evaluation

langscikw outputs an alphabetically sorted list for technical reasons, so we can’t show any top results. Instead, I told both programs to extract 300 keywords and manually edited the lists like we do when constructing the Subject Index.

I ended up with 92 keywords for Sketch Engine (31%) vs. 107 keywords for langscikw (34% – I received 313 terms). Both lists had 46 terms in common (52% and 45%, respectively).
You can view the spreadsheet with the original and cleaned lists at the end of this post.

While Sketch Engine introduces more noise in the form of linguistic examples (up to 25%), it actually requires less editing because these are easy for humans to spot and remove. langscikw oftentimes requires the user to supply antonyms (e.g. nonfinite clauses when finite clauses is given), Sketch Engine finds these quite reliably. Also, Sketch Engine has a habit of including unigram, bigram and trigram versions of the same term, which I avoided by only working with bigrams.

Of course, my models could be tweaked to achieve even better results. For example there could be some sort of part-of-speech tagging that ensures that only noun phrases are selected as keyterms as there is a quite a bit of noise and non-sensical results.
In the future it would be nice to have a formal evaluation of the model performance and perhaps more fine-tuning of the parameters. An improvement of the detex software or internal pre- and postprocessing could also deliver better results.

All in all, I believe langscikw is a worthwhile alternative to Sketch Engine for our purposes.

[You can find the published package on GitHub and PyPI]

Appendix: Key terms extracted via the different approaches