Logistic Regression

Logistic Regression is a supervised learning algorithm that predicts the probability of an instance belonging to a particular class. It's particularly useful for binary classification problems, where the outcome is one of two classes (e.g., spam or not spam). It can also be used for multiple classification.

Logistic model for binary classification

Denote by $X$ the input and $Y$ the class input $X$ belongs to. Denote also by:

p (X) = P (Y = 1 | X)

The logistic model is:

p (X) = \frac{\exp (β_{0} + β_{1} X)}{1 + \exp (β_{0} + β_{1} X)}

It can be rewritten as (simple calculus):

(E_{1}) : \log (\frac{p (X)}{1 - p (X)}) = β_{0} + β_{1} X

Estimators ${\hat{β}}_{0}$ and ${\hat{β}}_{1}$ are chosen to maximize a likelihood function, which is equivalent to minimizing the binary cross-entropy function

- \frac{1}{n} \sum_{i = 1}^{n} (y_{i} \log (p_{i}) + (1 - y_{i}) \log (1 - p_{i}))

Logistic model for multiple predictors

With the same notations, assuming $X = (X_{1}, \dots, X_{p})$ , the model is defined as (see equation $(E_{1})$ ):

(E_{2}) : \log (\frac{p (X)}{1 - p (X)}) = β_{0} + β_{1} X + \dots + β_{p} X_{p}

Multinomial Logistic Regression

Denote by $β_{0 k}, \dots, β_{p k}$ the coefficients for the $k$ -th class. Assuming $K$ categories, the model is typically expressed as:

P (Y = k | X) = \frac{\exp (β_{0 k} + β_{1 k} x_{1} + \dots + β_{p k} x_{p})}{\sum_{j = 1}^{K} \exp (β_{0 j} + β_{1 j} x_{1} + \dots + β_{p j} x_{p})}

Logistic Regression

Logistic model for binary classification

Logistic model for multiple predictors

Multinomial Logistic Regression

See also