Level 2

Front 1

Winsorizatiom

Back 1

Outlier replacement with max and min value from data set

Front 2

Wrangling vs cleansing

Back 2

Cleansing - basically removing all kinds of error from raw data

Wrangling = removing outliers, or selling with outliers," extraction" (remember not selection) of features, filtration (rows) and selection (columns), conversion (must be stripped out if prefixes and suffixes)

Front 3

Feature extraction vs feature engineering

Back 3

Extraction is the process of creating that is extracting new variables from existing ones in the data. Eg: ratio of two existing features.

Engineering is the process of creating new features by changing a transform in existing one

Front 4

Text processing

Back 4

Unstructured to structured - through cleansing and preprocessing (wrangling)

Html removal, number removal, punctuations removal, white space removal

Front 5

Regex

Back 5

Regular Expression - basically search function - used to search for patterns. Series that contain characters in certain order. Finding whitespace. Numbers, punctuations.

Front 6

Number removal

Back 6

When numbers are present in the text they should be removed or substituted with an annotation /number/

Front 7

Text wrangling (preprocessing)

Back 7

Token is equivalent to word.

Tokenization - process of splitting a given text into separate tokens. In other words, text is considered to be collection of tokens (word)

-Lowercasing

-Stopwords (the, a, is)

-Stemming (analysing, analysed -> analys) / lemmanization (analysing, analysed --> analys'e'). Stemming is more common

Front 8

N gram vs bow

Back 8

N-gram = set of n words that occur in that order

Bow = unigram.

Adv of n gram is that they can be used in the same way as unigram to build a bow. Eg: "Man_went_to_mall" is a n-gram but also a unigram (BOW).

Front 9

Text cleansing when to add annotation?

Back 9

In oujctuauoon, sometime currency signs, % signs are imp. So to not lose the sense add them in annotations /%/

Front 10

When some words appear vary in frequently in a textual database techniques that mein adress the risk of training highly complex models include

Back 10

Stemming

Front 11

Text cleansing vs text preprocessing (wrangling)

Back 11

Cleaning - remove punctuations remove numbers remove white spaces remove HTML tags

Wrangling - lower casing stopwords stemming lemmatization

Front 12

One hot encoding

Back 12

How to combine two categorical features? Through one hot encoding - it converts categorical variables into binary 0 or 1. One hot encoding is an example of feature engineering

Front 13

Feature selection (unstructured)

Back 13

Frequency- can be used for vocabulary pruning to remove noise features a filtering the tokens with very high and low frequency. Document frequency is defined as the number of documents (texts) that contain the respective tokens divided by the total number of documents (texts)

Chi-square test- it tests independence of two event - Occurrence of tokens and Occurrence of the class. Latest ranks the tokens buy the usefulness to each class in text classification problem. tokens with highest chi square test values occur more frequently in text associated with a particular size and therefore can be selected as a feature of ML

Mutual info - measures how much information is contributed by a token to a class of text. If the value is zero the meaning contributed is almost the same in all classes if the value is 1, the token contributes more meaning to a particular class

Front 14

Class imbalance

Back 14

Particular class is significantly larger than other class this could be due to over sampling. Aur under sampling of mini data

Front 15

Precision

Back 15

Useful when cost of causing type I error is high. Rejecting true null hypothesis. Eg. Rejecting quality of the car, when it is perfectly fine.

Ratio of correctly predicted positive classes by total predicted positive classes.

High precision indicates that the class. Labeled as +'ve is indeed positive. Small FP

Front 16

Recall

Back 16

Ratio of correctly predicted positive class by actual positive class.

useful in situation when Type ii error cost is high. Eg: car passes qulirt check, but is actually faulty.

High recall indicates that the class is correctly recignzed. (Small FN)

Front 17

Accuracy and f1 scire

Back 17

Trading off precision and recall is business model choice. Thus two overall performance matrices are used:

Accuracy: percentage of corre got predicted classes out if total prediction. Limitation is that it assumed equal cost/damage for both kind of error.

F1: Harmonic mean if precision and call.

F1 is more useful than accuracy when unequal class distribution is present in the database and it is necessary to measure equilibrium of precision and recall. f1 value is always closer to the lowest value of either recall or precision.

Front 18

Receiver Operating Characteristics (ROC)

Back 18

Higher the Area under curved better the model oerfoencae. The more convex the curve the better the model. Test A is better

Front 19

Grid search

Back 19

systematically training in machine learning model by using various combination of hyper parameter values across validating is model in determining which combination of hyperparametere results into best model

Front 20

Cross validation

Back 20

Technique for estimating out-of-sample error directly by determining the error in validation sample