TechVani: DIP and Pattern Recognition Important Questions with Answers: Chapter 2 Statistical Pattern Recognition

Chapter 1: Introduction Chapter 2: Statistical PRChapter 3: Image Preliminaries All Units (chapter index)

Every question from the Chapter 2 question bank (statistical pattern recognition), answered exam-style: definition first, key terms in bold, formulas and worked numbers where they earn marks. Very short answers are one to two lines, short answers five to eight lines, and long answers are structured with headings.

A. Very Short Answer Questions

Question	Answer
1. Define Bayesian Decision Theory.	Bayesian Decision Theory is the statistical framework that chooses the class with the lowest expected risk (or highest posterior probability), combining prior probabilities, likelihoods and loss values.
2. What is a prior probability?	The prior P(w) is the probability of a class before seeing any measurement - for example "1% of emails are spam".
3. What is a posterior probability?	The posterior P(w \| x) is the probability of a class after seeing the measurement x, computed with Bayes' theorem.
4. What is likelihood in Bayesian classification?	The likelihood p(x \| w) is the probability of observing measurement x given that the true class is w - how typical x is for that class.
5. State Bayes' theorem.	P(w \| x) = p(x \| w) . P(w) / p(x), where p(x) = sum over classes of p(x \| wj)P(wj).
6. What is a classifier?	A classifier is a rule or function that maps a feature vector x to a class label.
7. What is a discriminant function?	A discriminant function gi(x) scores each class i; the classifier picks the class with the highest score.
8. What is a linear discriminant function?	A discriminant that is linear in x: g(x) = w.x + b (weight vector w, bias b). Its decision boundary is a hyperplane.
9. What is a decision boundary?	The decision boundary is the surface in feature space where two classes' scores are equal (gi(x) = gj(x)); crossing it changes the decision.
10. What is a class-conditional probability or density?	p(x \| w) - the probability density of feature vector x within class w; it describes how class w's patterns are distributed.
11. Define Maximum Likelihood Estimation (MLE).	MLE estimates parameters by choosing the values that maximize the probability of the observed training data.
12. What is normal density estimation?	Assuming each class's p(x \| w) is a Gaussian and estimating its mean and covariance from training samples.
13. What is a Gaussian (normal) distribution?	A bell-shaped continuous density fixed by its mean (center) and variance (spread); in d dimensions by mean vector mu and covariance matrix Sigma.
14. What is a decision function?	Another name for the rule that converts scores into a final class decision - usually "choose the class with maximum gi(x)".
15. What is a covariance matrix?	The matrix Sigma = E[(x - mu)(x - mu)^T] holding every pair of features' covariance; diagonal = variances, off-diagonal = correlations.
16. What is minimum-error-rate classification?	Choosing the class with the highest posterior P(w \| x); this minimizes the probability of a wrong decision.
17. Define a loss function.	A loss function L(decision, true class) states the cost of each possible mistake, for example missing a disease costs more than a false alarm.
18. What is conditional risk?	Conditional risk R(action \| x) = sum over classes of L(action, wj) . P(wj \| x) - the expected loss of an action for this particular x.
19. What is a likelihood function?	The likelihood function is p(data \| parameters) viewed as a function of the parameters; MLE maximizes it.
20. What is a weight vector in a linear classifier?	The vector w in g(x) = w.x + b; it is perpendicular to the decision hyperplane and points toward the higher-scoring class.
21. Define a hyperplane in classification.	A hyperplane is a flat (d-1)-dimensional surface in d-dimensional space - a line in 2-D, a plane in 3-D - that separates two class regions.
22. What is the zero-one loss function?	Zero-one loss: 0 for a correct decision, 1 for any error. With it, minimum-risk classification becomes minimum-error-rate (pick max posterior).
23. Why is the Bayes classifier considered optimal?	Because, given the true probabilities and loss, no other classifier can achieve lower average risk/error - it uses all available information optimally.
24. What is the significance of the threshold or bias in a linear discriminant function?	The bias b shifts the hyperplane's position from the origin (w sets its orientation, b sets its offset); it encodes priors and class imbalance.
25. What happens to the decision boundary when class covariance matrices are equal?	Equal covariances make the quadratic terms cancel, so the Bayes boundary becomes a straight line / hyperplane (a linear classifier).
26. Differentiate between prior probability and posterior probability.	Prior P(w): belief about the class before measuring x. Posterior P(w \| x): updated belief after measuring x, via Bayes' theorem.
27. Compare Bayes risk with actual classification error.	Bayes risk is the best achievable expected loss using true probabilities; actual error of a real classifier is greater or equal, because it is built from estimated, imperfect probabilities.
28. Define risk in Bayesian decision theory.	Risk is the expected loss of a decision rule; the Bayes rule minimizes overall (average) risk over all inputs.
29. What is evidence in Bayes' theorem?	The evidence p(x) = sum_j p(x \| wj)P(wj) - the total probability of the measurement; it normalizes the posteriors to sum to 1.
30. Name one parameter estimated using maximum likelihood estimation.	The mean (equivalently the variance/covariance) of a class's Gaussian density.

B. Short Answer Questions

1. Explain Bayesian Decision Theory and the Bayes decision rule.

Bayesian Decision Theory treats classification as a decision under uncertainty using probabilities.
For each class we know the prior P(wi) and the likelihood p(x | wi).
Bayes' theorem combines them into the posterior P(wi | x) = p(x | wi)P(wi) / p(x).
Bayes decision rule (minimum error): choose the class with the highest posterior.
With a loss function, the rule generalizes to: choose the action with the lowest conditional risk.
It is optimal: given true probabilities, no rule can beat its average error.

2. Explain prior, likelihood, and posterior, and show how Bayes' theorem relates them.

Prior P(w): class probability before measuring - from base rates.
Likelihood p(x | w): how typical measurement x is inside class w.
Posterior P(w | x): class probability after measuring x.
Bayes' theorem: posterior = likelihood x prior / evidence, with evidence p(x) = sum_j p(x | wj)P(wj).
Intuition: posterior grows if x is typical for the class (likelihood) AND the class is common (prior).
Example: rare disease, positive test - high likelihood but tiny prior, so the posterior stays modest.

3. Describe how a classifier uses a decision boundary to classify patterns.

Training divides the feature space into decision regions, one per class.
The surface between two regions is the decision boundary, where the competing scores are equal.
To classify a new pattern: compute its feature vector, find which region it falls in, output that region's label.
Points near the boundary are the most uncertain (scores nearly equal).
Example: for g(x) = w.x + b, the boundary is the line w.x + b = 0; g > 0 is class 1's side, g < 0 is class 2's.

4. Explain minimum-error-rate classification and the role of loss functions.

Minimum-error-rate rule: decide the class with maximum posterior P(w | x) - this minimizes P(error).
It treats every mistake as equally costly.
A loss function L(ai, wj) assigns a cost to choosing action ai when the true class is wj.
Then we minimize conditional risk R(ai | x) = sum_j L(ai, wj)P(wj | x) instead of error count.
Example: in medical screening, missing a disease (say loss 50) costs more than a false alarm (loss 1), so the threshold shifts toward flagging more positives.
Zero-one loss (0 correct, 1 wrong) recovers minimum-error-rate as a special case.

5. Explain the basic concept and purpose of Maximum Likelihood Estimation (MLE).

Problem: a class density has a known form (say Gaussian) but unknown parameters theta (mean, covariance).
MLE idea: choose theta that makes the observed training samples most probable.
Formally: maximize the likelihood L(theta) = product of p(xk | theta) over samples (or its log).
Steps: write the log-likelihood, differentiate with respect to theta, set to zero, solve.
Purpose: turns raw samples into the numbers a Bayes classifier needs (mu, Sigma per class).
For a Gaussian, MLE gives the familiar sample mean and sample covariance.

6. Explain normal density estimation and its importance in classification, with its mathematical representation.

Assumption: each class-conditional density is multivariate normal.
Representation: p(x | wi) = 1/((2 pi)^(d/2) |Sigma_i|^(1/2)) . exp(-1/2 (x - mu_i)^T Sigma_i^-1 (x - mu_i)).
Estimation: set mu_i and Sigma_i to the sample mean and covariance of class i's training data (the MLE).
Importance: two numbers per class (mu_i, Sigma_i) replace the whole training set at decision time.
Bayes' rule then becomes comparing simple closed-form scores - fast and analytically clean.
Accuracy depends on how truly Gaussian the features are; goodness of fit should be checked.

7. Differentiate between linear and non-linear discriminant functions.

Form	Linear: g(x) = w.x + b	Non-linear: includes x^2 terms or other transforms
Boundary	Hyperplane (line in 2-D)	Curved surface (quadratic: ellipse, parabola, hyperbola)
Parameters	d + 1 (weights + bias)	Many more (quadratic: about d^2/2)
When it appears	Gaussian classes with equal covariance	Unequal covariances or complex class shapes
Risk	May underfit curved classes	May overfit with little data

8. Explain how discriminant functions help select a class in a multiclass problem.

Define one discriminant function gi(x) per class i = 1..c, for example gi(x) = log p(x | wi) + log P(wi).
Rule: compute all c scores and choose the class with the maximum gi(x).
This converts a c-class problem into c score evaluations plus one comparison - no pairwise voting needed.
Where two scores tie, gi(x) = gj(x), lies the decision boundary between regions i and j.
Example: digits 0-9 - ten discriminant scores; the unknown digit gets the label of the largest.

9. Explain the linear discriminant function and the roles of weights and bias.

Form: g(x) = w.x + b = w1x1 + ... + wdxd + b.
Decision: two-class case - class 1 if g(x) > 0, class 2 if g(x) < 0; boundary is the hyperplane w.x + b = 0.
Weight vector w: sets the orientation of the boundary; w is perpendicular to the hyperplane, pointing to the positive side. The size of each wi says how much feature i matters.
Bias b (threshold): sets the boundary's distance from the origin; it absorbs class priors and imbalance.
Geometric read: g(x) is proportional to the signed distance of x from the boundary - larger |g| means more confidence.

10. Explain the meaning of mean and variance in a Gaussian distribution.

The Gaussian density p(x) = 1/(sqrt(2 pi) sigma) exp(-(x - mu)^2 / (2 sigma^2)) has exactly two parameters.
Mean mu: the center - the most likely value and the balance point of the bell.
Variance sigma^2: the spread - average squared distance from mu; sigma is the standard deviation.
Small sigma: tall narrow bell (values cluster tightly). Large sigma: flat wide bell (values vary widely).
About 68% of samples lie within mu +/- sigma, 95% within mu +/- 2 sigma - a quick exam fact.
In PR, mu locates the class in feature space and sigma^2 controls how far a point can be and still look typical.

11. Explain how risk is minimized in Bayesian decision making.

Each possible action ai (assign a class, or reject) has a loss L(ai, wj) for every true class wj.
For input x, the conditional risk of action ai is R(ai | x) = sum_j L(ai, wj) P(wj | x) - expected loss given what we see.
Rule: for every x, choose the action with the smallest conditional risk.
Averaging this best choice over all x gives the minimum overall risk, called the Bayes risk.
Intuition: weigh the cost of each mistake by how probable that mistake is for this x, then pick the cheapest option.

12. Discuss the role of loss functions in Bayesian classification.

The loss function L(ai, wj) encodes how bad each decision outcome is - domain knowledge, not data.
It converts "maximize posterior" into "minimize expected cost" via conditional risk.
Zero-one loss: all errors equal - recovers minimum-error-rate classification.
Asymmetric loss: different errors cost differently - for example, in cancer screening a missed case costs far more than a false alarm, shifting the decision threshold.
A loss can also price a reject option (refuse to decide when uncertain).
Role summary: losses decide where boundaries sit; probabilities decide their shape.

13. Compare the Bayes classifier with maximum-likelihood-based classifiers.

Uses priors?	Bayes: yes - P(w) multiplies the likelihood	ML classifier: no - picks max p(x \| w) only
Optimality	Minimum error/risk when probabilities are true	Optimal only when classes are equally likely
Imbalanced data	Handles it correctly via priors	Biased toward the class with wider density
When used	Priors known or estimable	Priors unknown or uniform

Note: "MLE" also names a parameter-estimation method; a Bayes classifier usually uses MLE internally to get mu and Sigma. The comparison here is the decision rule: max posterior vs max likelihood.

14. Analyze the effect of unequal covariance matrices on the shape of a decision boundary.

For Gaussian classes the discriminant contains the quadratic term -1/2 x^T Sigma_i^-1 x.
Equal Sigma_1 = Sigma_2: the quadratic terms cancel - boundary is a line/hyperplane.
Unequal Sigma: quadratic terms remain - boundary becomes a quadratic curve (ellipse, circle, parabola or hyperbola depending on the matrices).
Example: one tight and one wide class - the boundary curves around the tight class, because the wide class's points stay plausible farther away.
Cost: quadratics need more parameters and more data; with little data, forcing equal covariance (a linear model) can generalize better.

15. Differentiate between generative and discriminant approaches to classification.

Models	Generative: p(x \| w) and P(w), then Bayes	Discriminant: the boundary or P(w \| x) directly
Learns	How each class generates data	Only how classes differ
Example	Gaussian Bayes	Logistic regression, perceptron
Strength	Works with little data; handles missing values	Often more accurate with lots of data
Weakness	Wrong density assumption hurts	Needs more data; no data model

16. Analyze the relationship between conditional risk and Bayes risk.

Conditional risk R(a | x) is the expected loss of action a for one fixed input x: sum_j L(a, wj) P(wj | x).
Overall risk R of a decision rule is the average of R(a(x) | x) over all possible x (integral over p(x)).
Since p(x) >= 0 and each conditional risk is non-negative, the overall risk is minimized by minimizing the conditional risk separately for every x.
The resulting minimum overall risk is the Bayes risk - the best any classifier can do.
So: pointwise minimization (conditional) → global optimality (Bayes risk). This is the core justification of the Bayes rule.

17. Compare linear and quadratic discriminant functions.

Formula	Linear: g(x) = w.x + b	Quadratic: g(x) = x^T A x + w.x + b
Boundary	Hyperplane	Conic surface (ellipse, parabola, hyperbola)
Gaussian case	Equal covariance matrices	Unequal covariance matrices
Parameters	d + 1	About d^2/2 + d + 1
Data need	Works with modest data	Needs much more data to avoid overfitting

18. Apply the Bayes decision rule to classify a sample using given priors and likelihoods.

Given: two classes, P(w1) = 0.7, P(w2) = 0.3; for x, p(x | w1) = 0.2, p(x | w2) = 0.4.
Step 1 - joint scores: p(x|w1)P(w1) = 0.14; p(x|w2)P(w2) = 0.12.
Step 2 - evidence: p(x) = 0.14 + 0.12 = 0.26.
Step 3 - posteriors: P(w1|x) = 0.14/0.26 ≈ 0.54; P(w2|x) = 0.12/0.26 ≈ 0.46.
Decision: choose w1 (higher posterior).
Note: although x looks more typical of w2 (higher likelihood), the stronger prior of w1 wins - for minimum error, comparing joint scores without dividing by p(x) is enough.

19. Apply maximum likelihood estimation to compute the mean and variance of a data sample.

Data: one feature, samples {4, 6, 8}.
Step 1 - MLE mean: mu = (1/n) sum xk = (4 + 6 + 8)/3 = 6.
Step 2 - deviations: -2, 0, +2; squares: 4, 0, 4.
Step 3 - MLE variance: sigma^2 = (1/n) sum (xk - mu)^2 = 8/3 ≈ 2.67 (sigma ≈ 1.63).
Exam note: MLE divides by n; the unbiased sample variance divides by (n - 1), giving 4 - mention if the examiner distinguishes them.

20. Explain the sequential steps involved in the maximum likelihood method.

Choose a model: fix the density form p(x | theta), for example Gaussian with theta = (mu, Sigma).
Collect samples: assume training samples x1..xn are drawn independently from it.
Write the likelihood: L(theta) = product of p(xk | theta).
Take the log: l(theta) = sum of log p(xk | theta) - products become sums, easier to differentiate.
Differentiate and set to zero: grad of l(theta) = 0.
Solve for theta: the solution is the MLE (for a Gaussian: sample mean and sample covariance).
Verify: check it is a maximum (second derivative / boundary check) when required.

21. Describe the main properties and geometric interpretation of a linear discriminant function.

Form: g(x) = w.x + b; boundary H: w.x + b = 0 is a hyperplane.
w is normal to H: it points toward the class with g > 0.
Signed distance: the distance of any point x from H is g(x)/||w|| - so g measures confidence, not just side.
Bias role: distance from origin to H is |b|/||w||; b shifts H without rotating it.
Properties: linear in x (fast to compute), parameters d + 1, convex decision regions, cannot separate classes that need curved boundaries (for example XOR-shaped data).

22. Summarize the role of discriminant functions and decision surfaces in multiclass pattern recognition.

One discriminant gi(x) per class scores "how much x belongs to class i".
The classifier assigns the argmax - the label of the highest score.
The set where gi(x) = gj(x) is the decision surface between classes i and j.
Decision surfaces partition the whole feature space into c decision regions, each one connected in the linear/Gaussian case.
Summary: discriminants turn classification into arithmetic (score and compare); surfaces are the geometric picture of the same rule.

23. Show the calculation structure for posterior probability using prior probability and likelihood.

Structure: P(wi | x) = p(x | wi) P(wi) / p(x), with p(x) = sum over all classes of p(x | wj) P(wj).
Step 1: for each class, multiply likelihood x prior (the unnormalized score).
Step 2: add all scores to get the evidence p(x).
Step 3: divide each score by the evidence; the posteriors now sum to 1.
Worked shape: scores 0.14 and 0.12 → evidence 0.26 → posteriors 0.54 and 0.46.
Exam shortcut: for decisions only, skip step 3 - the largest score is already the answer.

24. Design a two-class Bayesian classifier for a practical classification problem.

Problem: factory bottle inspection - class OK vs class DEFECT from one measured feature: fill level x (ml).
Model: assume each class is Gaussian. From 200 labeled bottles per class, MLE gives OK: mu1 = 500, sigma1 = 4; DEFECT: mu2 = 488, sigma2 = 6. Priors from production: P(OK) = 0.95, P(DEFECT) = 0.05.
Rule: decide DEFECT if p(x | DEFECT) P(DEFECT) > p(x | OK) P(OK), with the Gaussian formula plugged in.
Loss tuning: shipping a defect (loss 20) costs more than rechecking a good bottle (loss 1), so the threshold moves toward flagging more bottles as DEFECT than the pure 0-1 boundary would.
Evaluation: report error and the two mistake types on a separate test set.

C. Long Answer Questions

1. Explain Bayesian Decision Theory in detail and derive the Bayes decision rule for minimum-error-rate classification.

Setup: an input x (feature vector) must be assigned to one of classes w1..wc. Known: prior P(wi), likelihood p(x | wi), and a loss L(ai, wj) for taking action ai when the truth is wj.

Step 1 - posterior via Bayes: P(wj | x) = p(x | wj) P(wj) / p(x), with p(x) = sum_j p(x | wj)P(wj).

Step 2 - conditional risk: the expected loss of action ai for this x is R(ai | x) = sum_j L(ai, wj) P(wj | x).

Step 3 - Bayes rule: choose the action minimizing R(ai | x). Because the overall risk is the average of conditional risks and p(x) >= 0, minimizing risk pointwise minimizes total risk - the Bayes risk, the best achievable.

Derivation for minimum error rate: use zero-one loss - L = 0 when correct, 1 when wrong. Then R(ai | x) = sum of posteriors of all classes except wi = 1 - P(wi | x).

Minimizing 1 - P(wi | x) means maximizing the posterior: decide wi if P(wi | x) > P(wj | x) for all j. Since p(x) is common to all classes, this equals choosing the largest p(x | wi)P(wi).

Interpretation: the rule weighs how typical x is for each class by how common the class is; it is optimal - no decision rule can achieve lower average error given the same information.

2. Discuss minimum-error-rate classification and loss functions with suitable examples.

Minimum-error-rate classification: decide the class with the maximum posterior P(w | x). This minimizes the probability of error, P(error | x) = 1 - max P(w | x).

Example: posteriors 0.54 vs 0.46 - choose the first class; the error probability at that x is 0.46.

Why loss functions: counting errors treats all mistakes alike, which real problems reject. The loss function L(ai, wj) prices each (decision, truth) pair.

Generalized rule: minimize conditional risk R(ai | x) = sum_j L(ai, wj)P(wj | x).

Examples:

Zero-one loss: 0 correct, 1 wrong - recovers max-posterior.
Medical test: missing a disease = loss 50, false alarm = loss 1 - the threshold shifts so more borderline cases are flagged.
Reject option: a moderate "refuse to decide" loss means borderline x (posteriors near equal) are sent for manual review instead of a risky guess.

Summary: posteriors carry the data's information; losses carry the application's values; the Bayes rule merges both into the cheapest average decision.

3. Explain discriminant functions, decision functions, decision regions, decision surfaces, and decision boundaries for two-class and multiclass classification, with examples.

Discriminant function gi(x): a score assigned to class i; the standard choice is gi(x) = log p(x | wi) + log P(wi).

Decision function / rule: assign the class with the maximum gi(x) (argmax rule).

Decision region Ri: all points x where gi is largest; the classifier labels everything in Ri as class i.

Decision surface / boundary: the set where two scores tie, gi(x) = gj(x); it separates Ri and Rj.

Two-class example: define one function g(x) = g1(x) - g2(x). Decide class 1 if g(x) > 0. For Gaussians with equal covariance, g(x) = w.x + b - a linear boundary (hyperplane).

Multiclass example: digit recognition with c = 10: compute ten scores, output the largest; pairwise boundaries gi = gj cut the space into 10 regions.

Geometry note: linear discriminants give polygonal regions; unequal-covariance Gaussians give curved (quadratic) surfaces. The same rule thus looks algebraic in computation and geometric in space.

4. Explain MLE, its basic steps, and its importance in statistical pattern recognition. Derive the ML estimates of the mean and covariance of a normal density.

Concept: Maximum Likelihood Estimation picks the parameters theta that maximize the probability of the observed data: theta* = argmax L(theta), L(theta) = product of p(xk | theta).

Steps: fix model → write likelihood → take log (l = sum log p(xk | theta)) → differentiate → set gradient to zero → solve.

Derivation for a Gaussian:

log-likelihood: l = -(n d /2) log(2 pi) - (n/2) log|Sigma| - 1/2 sum_k (xk - mu)^T Sigma^-1 (xk - mu).

Mean: differentiating with respect to mu gives sum_k Sigma^-1 (xk - mu) = 0, so sum xk = n mu, hence

mu*  = (1/n) sum_k xk            (sample mean)

Covariance: substituting mu* and differentiating with respect to Sigma (using matrix derivative rules) yields

Sigma* = (1/n) sum_k (xk - mu*)(xk - mu*)^T   (sample covariance)

Importance in PR:

Converts raw training samples into the mu and Sigma a Gaussian Bayes classifier needs.
Consistent: estimates converge to the true values as n grows.
Simple closed forms make training fast.
Limitation: with few samples in high dimensions, Sigma* can be poor or singular (curse of dimensionality).

5. Explain normal density estimation and derive discriminant functions for a multivariate normal distribution.

Normal density estimation: assume p(x | wi) is multivariate normal and estimate (mu_i, Sigma_i) by MLE from class i's samples.

Start from Bayes: choose max gi(x) where gi(x) = log p(x | wi) + log P(wi) (log is monotone, so it changes nothing).

Substitute the Gaussian p(x|wi) = (2 pi)^(-d/2) |Sigma_i|^(-1/2) exp(-1/2 (x-mu_i)^T Sigma_i^-1 (x-mu_i)):

gi(x) = -1/2 (x - mu_i)^T Sigma_i^-1 (x - mu_i)  -  (d/2) log(2 pi)  -  1/2 log|Sigma_i|  +  log P(wi)

Case 1 - Sigma_i = sigma^2 I (features independent, equal spread): expanding shows gi(x) = -||x - mu_i||^2 / (2 sigma^2) + log P(wi) - a minimum-distance (nearest-mean) classifier; boundaries are linear.

Case 2 - Sigma_i = Sigma (equal covariances): the quadratic x^T Sigma^-1 x is the same for all classes and cancels; gi(x) = w_i . x + b_i with w_i = Sigma^-1 mu_i - still linear.

Case 3 - arbitrary Sigma_i: nothing cancels; gi(x) = x^T A_i x + w_i . x + b_i - quadratic boundaries (ellipses, parabolas, hyperbolas).

Significance: one formula family covers the three most common exam cases and explains exactly when a Bayes classifier is linear.

6. Discuss linear discriminant functions in detail: mathematical formulation, geometric interpretation, roles of weights and bias, and learning methods.

Formulation: g(x) = w.x + b = sum wi xi + b. Two-class rule: class 1 if g(x) > 0 else class 2. Multiclass: one gi per class, choose argmax.

Geometric interpretation:

The boundary H = {x : w.x + b = 0} is a hyperplane.
w is the normal vector of H; g(x)/||w|| is the signed distance of x from H - magnitude measures confidence.
b sets the offset: distance of H from the origin is |b| / ||w||.

Roles: weights encode each feature's importance and the boundary's orientation; the bias absorbs class priors and threshold choices.

Learning methods:

From Gaussians: equal-covariance Bayes gives w = Sigma^-1 (mu1 - mu2) directly.
Perceptron rule: cycle through samples; if a sample is misclassified, update w ↔ w + (correct-side adjustment). Converges when classes are linearly separable.
Least squares / LMS: choose w to minimize squared output error over samples.

Limitation: cannot separate classes needing curved boundaries (for example XOR); remedy is features transforms or quadratic/neural discriminants.

7. Compare Bayesian classifiers under equal and unequal covariance assumptions. How do these affect linear or quadratic decision boundaries?

Recall the Gaussian discriminant: gi(x) = -1/2 (x - mu_i)^T Sigma_i^-1 (x - mu_i) - 1/2 log|Sigma_i| + log P(wi).

Equal covariance (Sigma_i = Sigma for all classes):

The quadratic term x^T Sigma^-1 x is identical for every class and cancels from comparisons.
Result: gi(x) = w_i . x + b_i with w_i = Sigma^-1 mu_i - a linear discriminant; boundaries are hyperplanes.
Special sub-case Sigma = sigma^2 I: nearest-mean (Euclidean) classifier.
Needs few parameters; robust with limited data.

Unequal covariance (each class its own Sigma_i):

The quadratic terms differ and survive; gi(x) = x^T A_i x + w_i . x + b_i - a quadratic discriminant.
Boundaries become conic surfaces: the boundary can curve around the tighter class.
More expressive but needs roughly d^2/2 parameters per class - easy to overfit small datasets.

Exam example: class A tight (small Sigma), class B spread out (large Sigma) - the boundary bends so that far points still belong to B; equal-covariance forcing would draw a straight line and misclassify such regions.

8. Apply Bayesian decision theory to design a complete multiclass classifier for a given dataset.

Problem: classify fruits into 3 classes (apple, banana, orange) from features x = [weight, redness].

Step 1 - data: 150 labeled samples per class; split 100 train / 25 validation / 25 test per class.

Step 2 - model choice: assume a Gaussian per class; estimate mu_i and Sigma_i by MLE on the training part. Priors P(wi) = class frequencies (here 1/3 each).

Step 3 - decision rule: for a new fruit compute gi(x) = -1/2 (x - mu_i)^T Sigma_i^-1 (x - mu_i) - 1/2 log|Sigma_i| + log P(wi) for i = 1, 2, 3 and output the argmax.

Step 4 - worked check: for x = [150 g, 0.8 redness], the apple score comes out highest because 150 g and red is typical of apples - classify as apple.

Step 5 - losses: all confusions cost the same here, so zero-one loss (maximum posterior) is used.

Step 6 - evaluation: measure accuracy and the confusion matrix on the test set; if banana and orange confuse most, consider adding a feature (for example length) and retrain.

Step 7 - deployment and adaptation: re-estimate parameters monthly as new labeled batches arrive.

9. Evaluate the performance and limitations of maximum likelihood estimation when training data is limited.

What MLE does: chooses parameters maximizing the probability of the observed samples - for a Gaussian, the sample mean and sample covariance.

Strengths with limited data: simple, closed-form, no tuning; unbiased for the mean.

Limitations when data is scarce:

High variance of estimates: a few unlucky samples pull mu and Sigma far from the truth.
Singular covariance: with n < d (fewer samples than features), Sigma* is not invertible, so the Gaussian discriminant cannot even be computed.
Overfitting: the model fits noise; training error looks great, test error is poor.
Biased variance: MLE variance divides by n and slightly underestimates spread (unbiased uses n - 1).

Remedies:

Reduce features (PCA, feature selection) - fight the curse of dimensionality.
Regularize the covariance (shrink toward diagonal: Sigma + epsilon I).
Assume shared covariance across classes (linear instead of quadratic boundaries).
Use priors/ Bayesian parameter estimation to blend data with prior knowledge.

Verdict: MLE is the right default, but with limited data its raw output must be simplified or regularized before trusting a classifier built on it.

10. Design and justify a complete statistical pattern classifier for a real-world application such as medical diagnosis.

Application: diabetes screening from two features x = [fasting glucose, BMI]; classes: HIGH-RISK vs LOW-RISK.

1. Data: 1000 labeled patient records per class from a hospital study; split 60/20/20 into train/validation/test.

2. Model (justified): each class modeled as a bivariate Gaussian - medical measurements are approximately bell-shaped, and only mu, Sigma per class are needed, which suits moderate data. Estimate by MLE.

3. Priors: from population screening rates, P(HIGH) = 0.2, P(LOW) = 0.8.

4. Loss function (critical): missing a high-risk patient (loss 40) costs far more than a false alarm (loss 1). With asymmetric loss, decide HIGH-RISK when P(HIGH | x) > 1/41 instead of > 0.5 - the threshold moves to catch more true cases.

5. Rule: compute posteriors with Bayes' theorem; apply the risk-based threshold; optionally reject (send for lab test) when posteriors are near the threshold.

6. Validation: report sensitivity (missed cases - must be high) and false-alarm rate on the test set, not just accuracy.

7. Adaptation: re-estimate parameters per clinic, since populations differ.

11. Analyze the performance evaluation methods for different classifiers under Bayesian Decision Theory.

1. Bayes risk / Bayes error: the theoretical minimum error, computed by integrating P(error | x)p(x). It is the reference floor - no classifier can beat it; comparing a classifier's error to the Bayes error shows how much room remains.

2. Confusion matrix: rows = true classes, columns = predicted; shows which classes get confused, guiding feature design.

3. Error-rate estimate on a test set: errors / test samples - the honest headline number, valid only if the test set was never used in training or tuning.

4. Per-class costs: with asymmetric losses, report expected loss (sum of confusion entries x their losses), not plain accuracy - a classifier with more errors can be cheaper overall.

5. Cross-validation: rotate which fold is held out and average - tighter estimates on small datasets.

Analysis: the right metric mirrors the loss function; accuracy fits zero-one loss, expected loss fits asymmetric costs, and the Bayes error tells whether a poor result means a bad classifier or an intrinsically hard problem.

Answers are original study material written for exam preparation. Definitions and derivations follow Duda, Hart and Stork, Pattern Classification (standard for statistical pattern recognition). Worked numbers were computed and checked. Institution names from the source question bank are not reproduced.

Chapters: Chapter 1: Introduction Chapter 2: Statistical PR Chapter 3: Image Preliminaries All Units index