Text mining and natural language processing (NLP) are common tasks in data science, which can be performed in R using various packages such as tm and tidytext. The tm package is specifically designed for text mining, whereas the tidytext package is part of the tidyverse ecosystem and focuses on tidy data principles.
The tm package provides tools for reading in text data, preprocessing, cleaning, and transforming text, and creating term-document matrices for modeling. For example, the following code reads in a corpus of text files and performs preprocessing steps such as removing stop words and stemming:
library(tm)
data("crude")
crude_corpus <- VCorpus(VectorSource(crude))
crude_corpus <- tm_map(crude_corpus, removeWords, stopwords("english"))
crude_corpus <- tm_map(crude_corpus, stemDocument)Once the text has been preprocessed, the next step is to create a term-document matrix (TDM) that represents the frequency of each term (word) in each document. This can be done using the DocumentTermMatrix() function:
crude_tdm <- DocumentTermMatrix(crude_corpus)The resulting TDM can then be used for various types of analyses, such as clustering or topic modeling.
The tidytext package, on the other hand, provides tools for working with text data in a tidy format, which is more amenable to the tidyverse data manipulation tools. This includes functions for splitting text into words, creating n-grams, and performing sentiment analysis. For example, the following code reads in a data frame of customer reviews and calculates the frequency of each word using the unnest_tokens() function:
library(tidytext)
data("yelp")
yelp_words <- yelp %>%
unnest_tokens(word, text) %>%
count(word, sort = TRUE)The resulting data frame can then be used for further analysis or visualization.
In addition to these packages, there are many other tools and techniques available in R for text mining and NLP, such as regular expressions, word embeddings, and deep learning models.