Logistic Regression

Logistic Regression, a machine learning algorithm, is a statistical analysis methodology used to predict a binary outcome based on a set of independent variables. It is a binary classification algorithm that estimates the probability of an event occurring. In other words, it's a predictive modelling technique used when the dependent variable is categorical.

How Logistic Regression works

Logistic regression works by using the logistic function or sigmoid function, which is an S-shaped curve that maps any real-valued number into a value between 0 and 1. This curve is used to convert any linear combination of the independent variables into a range that can be used to predict the probability of the occurrence of the event.

Firstly, a linear model is created, just like in linear regression, which is then put through the logistic function. The resulting number between 0 and 1 is the probability that the dependent variable is a "success" (in the binary case, for example, whether an email is spam (1) or not-spam (0)). The predicted probabilities are then transformed into binary values in order to actually make the prediction.

For making this binary decision, a threshold value or decision boundary is defined. If the probability is above this value, the event is considered to happen, else not. These binary predictions are then compared to actual outcomes to train the model, using a method such as gradient descent, and the process is repeated until the error rate is minimized.

Finally, once a logistic regression model has been trained it is often valuable to present the results by showing the impact of the variables using odds ratios, which are calculated using the coefficients from the logistic regression.