Feature engineering for text data is a crucial step in the process of building machine learning models. It involves transforming raw text data into numerical features that can be used as input for a model. There are a variety of techniques that can be used to do this, including text preprocessing, feature extraction, and feature selection.
Text preprocessing is a critical step that normally done before feature engineering. In text preprocessing, the text data is transformed into a more manageable format. Common techniques used in text preprocessing include:
1. **Tokenization:** This is the process of breaking up the text into individual words or phrases, also called tokens. Tokenization is often the first step in any text analytics process. It is usually done using white space as the delimiter. However, there are more advanced methods like named entity recognition (NER), sentence segmentation, etc that can be used.
2. **Lowercasing:** Text data is converted into lowercase to ensure all words are at the same level (i.e., "cat" and "Cat" would be considered as the same token).
3. **Stopword removal:** Stopwords are common words that do not hold much meaning or value, e.g., "the," "and," "or," "is," etc.. Removing these words, also called stopword removal, can reduce the size of the data set and make the analysis more efficient. A common list of stopword is NLTK( Natural Language Toolkit) stopwords list.
4. **Stemming and Lemmatization:** These are techniques to convert words to their base form, which is useful for combining words into their root form (i.e., “running” and “ran” are both reduced to “run”). This can be done using tools like the Porter Stemmer, Snowball Stemmer, or lemmatizers like WordNetLemmatizer in NLTK.
Once the text is preprocessed, there are several common techniques that can be used for feature extraction:
1. **Bag of Words:** The Bag-of-Words (BoW) model is a simple way to represent text data as numeric information. A document is represented as a bag, i.e., an unordered set, of words or phrases, and only the frequency of each term is used.
For example, suppose we have the following two sentences "The dog chased the cat" and "The cat ran away." A BoW representation of the two can be;
| | the | dog | chased | cat | ran | away |
|----|-----|-----|--------|-----|-----|------|
| S1 | 2 | 1 | 1 | 1 | 0 | 0 |
| S2 | 1 | 0 | 0 | 1 | 1 | 1 |
| | | | | | | |2. **Term Frequency-Inverse Document Frequency (TF-IDF):** TF-IDF is a modification to the BoW technique. It takes into account the importance of a word in a document and across all documents. The TF-IDF score for a term is calculated as its term frequency multiplied by its inverse document frequency.
Consider we have a dataset of 100 documents with 10 of them talking about "machine learning". For a new document, a borderline TF-IDF term is "machine", which can be present in lots of documents. This makes it less unique and less importance for determining similarity between documents. However, "learning" may be present in many documents in the corpus, so "machine learning" can be used for document classification.
3. **Word embeddings:** This is a more advanced technique of representing words in a high dimension space. One can use precomputed embeddings such as word2vec, GloVe, BERT, and load the embeddings as the input feature.
In summary, the goal of feature engineering for text data is to convert raw text into numerical features that machine learning models can use for training. Common techniques include text preprocessing with tokenization, lowercasing, stopword removal, stemming, and lemmatization. Feature extraction techniques include Bag-of-Words, TF-IDF, and Word Embeddings.