Keyword assignment with python

Keyword extraction is tasked with the automatic identification of terms that best describe the subject of a document (Wikipedia, 2021). Generally, we can split keywords extraction methods in two:

• Keyword assignment (where keywords are assigned from a controlled pool of words)
• Keyword extraction (where the assigned keywords are chosen from the text)

In this brief article I am going to explore a simple keywords assignment algorithm using Python. The reason for this post is that while trying to find a valid imputer for the missing keywords in the NYT articles (GitHub) I have stumbled upon multiple solutions that most of the time worked inconsistently if not quite poorly. The main target for me was that the keyword extractor could have been capable of using candidate keywords generated by other keyword algorithms or retrieved from a select list of possible keywords/keyphrases since I was dealing already with a large number of different keywords (~4.5k).

I ended up using KeyBERT (more resource and insights of different algorithms can be found here ). KeyBERT is a minimal and easy-to-use keyword extraction technique that leverages BERT embeddings to create keywords and keyphrases that are most similar to a document (Grootendorst, 2021) built upon BERT. BERT stands for Bidirectional Encoder Representations from Transformers and it's a language representation model designed to pre-train deep bidirectional representation from unlabelled text by jointly conditioning on both left and right context in all layers (Jacob Devlin, 2018). More about BERT can be found here.

About keyBERT

Pro of keryBERT:

• High Accuracy
• Possibility to choose from a pre-defined pool of keywords
• Highly customizable trough different parameters

• Slow compared to other algorithms
• Highly customizable trough different parameters

To demonstrate the potential of KeyBERT I will use a snippet of code from my study and let you play with it. But first here's a brief description of the parameters:

• ngram_range: Length, in words, of the extracted keywords/keyphrases - we will set this to 1 considering that we will use "candidates" parameter
• stop_words: Stopwords to remove from the document
• top_n: Return the top n keywords/keyphrases - we'll set this to 6 considering the mean keywords list len is of 6
• use_mmr: Whether to use Maximal Marginal Relevance (MMR) for the selection of keywords/keyphrases. We will set this to True. "MMR considers the similarity of keywords/keyphrases with the document, along with the similarity of already selected keywords and keyphrases. This results in a selection of keywords that maximize their within diversity with respect to the document." (Maarten Grootendorst, 2021)
• diversity: The diversity of the results between 0 and 1 if use mmr is set to True: We will set this value to 0.2 trying to find most similar results.
• candidates: Candidate keywords/keyphrases to use instead of extracting them from the document(s). We will use a smaller subset of our keywords since the article's keywords list has more than 4.5k unique values. This strongly affect keyBERT extraction time.

Extracting keywords from following text:

"The Biden administration plans to require most foreign visitors to be vaccinated. Biden Plans New Policy Requiring That All Foreign Travelers to U.S. Be Vaccinated The Biden administration is developing plans to require all foreign travelers to the United States to be vaccinated against Covid-19, with limited exceptions, according to an administration official with knowledge of the developing policy.Officials say the new policy is being readied in the event that the United States eases its travel rules, which isn't expected soon."

Example Keyword Extraction Results

Using KeyBERT with the following parameters on the sample text:

• diversity: 0.2 (to find similar results)
• top_n: 6 (number of keywords to extract)
• use_mmr: True (Maximal Marginal Relevance enabled)
• candidates: Selected from the keyword pool with cardinality > 60

Extracted Keywords:

Based on the sample text about Biden's vaccination policy for foreign travelers, KeyBERT would extract keywords such as:

Note: In the original Flask application, users could interactively adjust these parameters and see real-time keyword extraction results. This static version shows example output.

References

• Wikipedia, 2021. Keyword extraction. Available at: https://en.wikipedia.org/wiki/Keyword_extraction
• Grootendorst, M., 2021. KeyBERT: Minimal keyword extraction with BERT. Available at: https://maartengr.github.io/KeyBERT/
• Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K., 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
• 10 Popular Keyword Extraction Algorithms in Natural Language Processing. Available at: https://medium.com/mlearning-ai/10-popular-keyword-extraction-algorithms-in-natural-language-processing-8975ada5750c
• A Primer in BERTology: What We Know About How BERT Works. MIT Press. Available at: https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00349/96482/A-Primer-in-BERTology-What-We-Know-About-How-BERT
• BERT Explained: State of the Art Language Model for NLP. Available at: https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270