Bayesian classification

Bayesian classification is a statistical technique used to classify the data based on probabilistic reasoning. It is a type of probabilistic classification that uses Bayes’ theorem to predict the probability of a data point belonging to a certain class.

Bayesian classification is a statistical method in data mining to data classification that uses Bayes’ theorem to make predictions about a class of a data point based on observed data.

The basic idea behind Bayesian classification is to assign a class label to a new data record based on the probability that it belongs to a particular class.

The Bayesian classification is a powerful technique for probabilistic inference and decision-making and is widely used in various applications such as medical diagnosis, spam classification, fraud detection, etc.

Naïve Bayesian classifier

Bayesian classification is based on Bayes’ theorem, a simple Bayesian classifier known as the naïve Bayesian classifier.

Bayes’ theorem

Bayes’ theorem is named after Thomas Bayes, an 18th-century British mathematician who first formulated it.

In data mining, Bayes’ theorem is used to compute the probability of a hypothesis (such as a class label) for given some observed event (such as a set of attributes or features).

Bayes’ theorem is a technique for predicting the class label of a new tuple / instance based on the probabilities of different class labels and the observed attributes / features of the instance.

Bayes’ Theorem consists of two types of probabilities: the posterior probability (P(H/X)) and the prior probability (P(H)), where X represents the data tuple and H represents a hypothesis.

The formula for Bayes’ Theorem is as follows:

Let X is a data tuple. In Bayesian terms, X is considered “evidence.” As usual, it is described by measurements made on a set of n attributes. Let H be some hypothesis, such as that the data tuple X belongs to a specified class C.

For classification problems, we want to determine P (H/X), the probability that the hypothesis H holds given the observed data tuple X. In other words, we are looking for the probability that tuple X belongs to class C, given that we know the attribute description of X.

Posterior probability

P(H/X) is the posterior probability or a posteriori probability of hypothesis H conditioned on data tuple X.

For example, suppose we have data tuples from the customers database described by the attributes age and income, respectively, and that X is a 30-year-old customer with an income of $40,000. Suppose that H is the hypothesis that our customer will buy a computer. Then the P(H/X) represents the probability of customer X purchasing a computer, considering the customer’s age and income information.

Similarly, P(X/H) is the posterior probability of X conditioned on hypothesis H. That is, it is the probability that a customer, X, is 30 years old and earns $40,000, given that we know the customer will buy a computer.

Prior probability

P(H) is the prior probability, or a priori probability, of H.

For example, this is the probability that any given customer will buy a computer, regardless of age, income, or any other information, for that matter. i.e., The prior probability P(H), is independent of data tuple X.

Similarly, P(X) is the prior probability of X. Using our example, it is the probability that a person from our data set of customers is 30 years old and earns $40,000.

Naive Bayes Classifier Algorithm

It is a classification technique based on Bayes’ Theorem with an independence assumption among predictors. In simple terms, a Naive Bayes classifier assumes that the presence of a particular feature in a class is unrelated to the presence of any other feature.

For example, a fruit may be considered to be an apple if it is red, round, and about 3 inches in diameter. Even if these features depend on each other, all of these properties independently contribute to the probability that this fruit is an apple and that is why it is known as ‘Naive’.

Working of Naïve Bayes’ Classifier

Example:

Suppose we have a dataset of weather conditions and corresponding target variable “Play”. So, using this dataset we need to decide that whether we should play or not on a particular day according to the weather conditions. So, to solve this problem, we need to follow the below steps:

• Convert the given dataset into frequency tables.
• Generate Likelihood table by finding the probabilities of given features.
• Now, use Bayes theorem to calculate the posterior probability.

Problem: If the Weather is Sunny, then the Player should play or not?

Solution: To solve this, first consider the below dataset:

First, Convert the above table into frequency table as follows,

Next, Generate Likelihood table by finding the probabilities of given features as follows,

Applying Bayes’ theorem

If the data tuple (X) is Sunny, and the hypothesis (H) is Yes, then the probability of Play is:

Where,

P(Sunny|Yes) = 3/10 = 0.3

P(Yes) = 10/14 = 0.71

P(Sunny) = 5/14 = 0.35

Let us substitute the values in the above formula,

= 0.60

If the Weather is Sunny, then the probability of Play (Yes) is 0.60

If the data tuple (X) is Sunny, and the hypothesis (H) is No, then the probability of Play is:

Where,

P(Sunny|No) = 2/4 = 0.5

P(No) = 4/14 = 0.29

P(Sunny) = 5/14 = 0.35

Let us substitute the values in the above formula,

= 0.41

If the Weather is Sunny, then the probability of Play (No) is 0.41.

So, as we can see from the above calculation that P(Yes|Sunny) > P(No|Sunny)

Hence on a Sunny day, Player can play the game.

Exercise

Problem: If the Weather is Sunny and Humidity is High, then the Player should play or not?

Solution:

If the data tuple (X) is (Sunny, High), and the hypothesis (H) is Yes, then the probability of Play is:

If the data tuple (X) is (Sunny, High), and the hypothesis (H) is No, then the probability of Play is: