Fundamentals of Machine Learning | Semester 5 Core Notes
Semester 5 - Page 4: Fundamentals of Machine Learning. 27 numbered topics, highlighted rules, worked arithmetic, answered exam questions, pipeline diagrams, comparison tables and four complete Python demos. General common-core guide, not an official syllabus or a validated prediction service.
Topic map
- Learning problems and generalization
- Features, labels and prediction time
- Baselines and task framing
- Train, validation and test splits
- Leakage and preprocessing boundaries
- Missing values, encoding and scaling
- Linear regression and least squares
- Loss and gradient descent
- Logistic regression and thresholds
- K-nearest neighbors and distance
- Naive Bayes and conditional independence
- Decision trees and split criteria
- Support vector machines and kernels
- Ensembles, bagging and boosting
- Neural networks and backpropagation
- Overfitting, underfitting and regularization
- Cross-validation and hyperparameters
- Confusion matrix and classification metrics
- ROC, precision-recall and calibration
- Regression metrics and uncertainty
- K-means and cluster interpretation
- Other clustering and dimensionality reduction
- Feature engineering and selection
- Reinforcement learning basics
- Deployment, drift and reproducibility
- Privacy, fairness and decision costs
- Exam and experiment checklist
1. Learning problems and generalization
Machine learning fits patterns from examples to use on new data. Supervised learning uses target labels; unsupervised learning searches for structure. Prediction does not establish causation.
Answer: No. It may reflect memorization, leakage or an unrepresentative sample.
2. Features, labels and prediction time
Features are inputs available when predicting; labels are the target outcome. Define unit, time, source and permission for each input.
Answer: Yes. It may be unavailable at prediction time or inappropriate to use.
3. Baselines and task framing
Choose a measurable task and a simple baseline before adding complexity. Regression predicts numeric targets; classification predicts categories; clustering has no supplied target classes.
Answer: No. Data quality, uncertainty, costs and deployment conditions still matter.
4. Train, validation and test splits
Training fits parameters; validation guides model/threshold choices; test estimates held-out performance after choices. Keep the test set out of tuning.
Answer: They adapt the model to that test set and bias the reported score.
5. Leakage and preprocessing boundaries
Fit learned preprocessing only on training data, including inside each cross-validation fold. Apply the fitted transformation unchanged to held-out data.
Answer: No. Scaling, imputation, PCA and feature selection can leak information too.
Split by unit/time Training -> fit preprocessing + model Validation -> transform + choose Test -> frozen transform + evaluate
6. Missing values, encoding and scaling
Handle missingness deliberately; encode nominal categories without inventing order; scaling matters for many distance/gradient methods. Preserve the training transformation at inference.
Answer: Not generally; preprocessing needs depend on the estimator and data.
7. Linear regression and least squares
A linear model predicts a weighted sum plus intercept. Least squares minimizes squared residuals; a perfect toy fit says nothing about future data.
Answer: They can expose systematic errors, outliers and model mismatch.
Least-squares line and training-only scaling
from math import sqrt
x, y = [1.0, 2.0, 3.0], [2.0, 4.0, 6.0]
xmean, ymean = sum(x)/len(x), sum(y)/len(y)
denominator = sum((v-xmean)**2 for v in x)
slope = sum((a-xmean)*(b-ymean) for a, b in zip(x, y))/denominator
intercept = ymean - slope*xmean
print("slope:", slope, "intercept:", intercept)
print("prediction at 4:", slope*4+intercept)
# Population scale for this toy training set, not fit on held-out data.
scale = sqrt(sum((v-xmean)**2 for v in x)/len(x))
print("held-out 100 scaled:", format((100-xmean)/scale, ".6f"))
actual, predicted = [2, 4, 6], [2, 5, 5]
errors = [a-b for a,b in zip(actual,predicted)]
print("MAE:", format(sum(abs(v) for v in errors)/len(errors), ".6f"))
print("MSE:", format(sum(v*v for v in errors)/len(errors), ".6f"))
Checked output:
slope: 2.0 intercept: 0.0 prediction at 4: 8.0 held-out 100 scaled: 120.024997 MAE: 0.666667 MSE: 0.666667
8. Loss and gradient descent
Loss quantifies error; gradient descent steps opposite the gradient of a differentiable objective. Learning rate controls step size; objective normalization changes gradient constants.
Answer: No. It may oscillate or diverge.
Gradient descent on a small linear-regression objective
x, y = [1.0, 2.0, 3.0], [2.0, 4.0, 6.0]
w, b, rate = 0.0, 0.0, 0.05
for _ in range(20000):
errors = [w*a+b-target for a,target in zip(x,y)]
dw = 2*sum(e*a for e,a in zip(errors,x))/len(x)
db = 2*sum(errors)/len(x)
w, b = w-rate*dw, b-rate*db
print("slope:", format(w, ".6f"))
print("intercept:", format(b, ".6f"))
print("prediction at 4:", format(w*4+b, ".6f"))
print("MSE:", format(sum((w*a+b-target)**2 for a,target in zip(x,y))/len(x), ".6f"))
Checked output:
slope: 2.000000 intercept: 0.000000 prediction at 4: 8.000000 MSE: 0.000000
9. Logistic regression and thresholds
Binary logistic regression transforms a linear score with a sigmoid and is commonly fitted using log loss. A threshold turns a score into a class; probability calibration is not automatic.
Answer: No. Its common binary use models class scores/probabilities.
10. K-nearest neighbors and distance
KNN predicts using nearby labeled training points. Distance, scaling, K and tie handling matter. Prediction can require expensive neighbor search.
Answer: No. KNN uses labels for prediction; K-means clusters without labels.
11. Naive Bayes and conditional independence
Naive Bayes uses Bayes' rule with conditional feature-independence assumptions given the class. Smoothing handles unseen events in appropriate models.
Answer: No. The simplifying assumption can still support useful baselines.
12. Decision trees and split criteria
Trees recursively split data using feature tests; criteria such as impurity reduction guide classification splits. Depth/minimum-sample constraints control complexity.
Answer: No. They depend on the model, data and feature relationships.
13. Support vector machines and kernels
SVMs learn a separating decision rule with margin/regularization tradeoffs; kernels represent suitable nonlinear similarities. Scaling and hyperparameter choices matter.
Answer: No. Greater flexibility can also overfit.
14. Ensembles, bagging and boosting
Ensembles combine models. Bagging fits models on resampled data; random forests add feature randomness. Boosting builds an ensemble in successive fitting stages.
Answer: No. Data, model errors and training design determine the result.
15. Neural networks and backpropagation
Neural networks compose parameterized transformations and activations. Backpropagation computes gradients by the chain rule; optimization updates weights.
Answer: No. Architecture, learning rate and training design remain choices.
Input -> hidden transformations -> score Score + target -> loss Loss gradient -> backpropagate -> update
16. Overfitting, underfitting and regularization
Underfitting misses relevant structure; overfitting fits training detail without generalizing. Regularization and complexity limits introduce constraints/tradeoffs.
Answer: No. Excessive constraint can underfit.
17. Cross-validation and hyperparameters
Cross-validation evaluates repeated held-out folds under a stated split strategy. Hyperparameters are selected outside ordinary fitted parameters. Tuning needs separation from final evaluation.
Answer: No. Future-to-past leakage can make the estimate unrealistic.
18. Confusion matrix and classification metrics
TP, FP, FN and TN depend on the declared positive class. Precision asks correctness among predicted positives; recall asks coverage among actual positives.
Answer: No. A majority-only predictor may miss the positive group.
Confusion metrics with explicit zero-denominator handling
def metrics(tp, fp, fn, tn):
divide = lambda a, b: a / b if b else 0.0
return (divide(tp + tn, tp + fp + fn + tn),
divide(tp, tp + fp), divide(tp, tp + fn),
divide(2 * tp, 2 * tp + fp + fn))
for counts in [(30, 10, 5, 55), (0, 0, 5, 95)]:
values = metrics(*counts)
print("counts:", counts)
for name, value in zip(["accuracy", "precision", "recall", "F1"], values):
print(name + ":", format(value, ".6f"))
Checked output:
counts: (30, 10, 5, 55) accuracy: 0.850000 precision: 0.750000 recall: 0.857143 F1: 0.800000 counts: (0, 0, 5, 95) accuracy: 0.950000 precision: 0.000000 recall: 0.000000 F1: 0.000000
19. ROC, precision-recall and calibration
Threshold curves describe score tradeoffs; precision-recall is useful in many imbalanced settings. Calibration concerns whether predicted probabilities agree with observed frequencies.
Answer: No. Ranking and calibration measure different properties.
20. Regression metrics and uncertainty
MAE averages absolute error; MSE squares error; RMSE returns target units. R-squared compares residual error to variation under defined conventions. Scores depend on split and sample.
Answer: No. It can be negative; constant-target cases require care.
21. K-means and cluster interpretation
K-means alternates nearest-center assignment and center means to reduce its within-cluster squared-distance objective. Initialization, K, scaling and empty clusters need handling.
Answer: No. Interpret with domain evidence and safeguards.
Initial centers -> assign nearest center -> recompute nonempty means -> repeat to stated stop rule
Deterministic one-dimensional K-means teaching demo
points, centers = [1.0, 2.0, 8.0, 9.0], [1.0, 9.0]
for step in range(20):
labels = [min(range(len(centers)), key=lambda j: (abs(p-centers[j]), j)) for p in points]
new = []
for j in range(len(centers)):
group = [p for p,label in zip(points,labels) if label == j]
new.append(sum(group)/len(group) if group else centers[j])
if new == centers:
break
centers = new
print("centers:", centers)
print("labels:", labels)
print("squared error:", sum((p-centers[j])**2 for p,j in zip(points,labels)))
Checked output:
centers: [1.5, 8.5] labels: [0, 0, 1, 1] squared error: 1.0
22. Other clustering and dimensionality reduction
Density/hierarchical methods have different assumptions from K-means. PCA finds variance directions for a linear projection; it is not supervised feature selection by target usefulness.
Answer: No. Useful prediction features require task-specific evaluation.
23. Feature engineering and selection
Features can express meaningful transformations or interactions, but their definition must respect prediction time and fold boundaries. Selection is part of the fitting workflow.
Answer: It exposes held-out information to model choices.
24. Reinforcement learning basics
Reinforcement learning concerns an agent choosing actions in an environment using reward/feedback. States, actions, policy and delayed outcomes differ from ordinary static-label supervision.
Answer: No. Poor reward design can encourage unwanted behavior.
25. Deployment, drift and reproducibility
Preserve feature/preprocessing/model versions, evaluate serving inputs and monitor relevant drift/outcomes. Seeds help repeat experiments but do not prove future performance.
Answer: No. Monitoring, data quality, errors and responsible use continue.
26. Privacy, fairness and decision costs
Access permission, sensitive information, group effects and intervention costs matter beyond an aggregate score. Human review and contestability can be necessary.
Answer: No. Proxy features and unequal data quality can preserve harmful differences.
27. Exam and experiment checklist
State task, unit, available features, split, baseline, pipeline, metric and limitations. Demonstrate arithmetic and test assumptions rather than promising accuracy.
Answer: Dataset/split, metric definition, fitting/tuning procedure, uncertainty and scope of use.
Evaluation boundaries
| Stage | Use | Avoid |
|---|---|---|
| Training | Fit parameters and transforms | Using held-out labels |
| Validation / CV | Select model/threshold | Random folds for every time/group task |
| Final test | Evaluate frozen choices | Repeated tuning on test score |
| Deployment | Monitor actual serving behavior | Assuming train score persists |
Confusion matrix
| Actual / predicted | Positive | Negative |
|---|---|---|
| Actual positive | TP = 30 | FN = 5 |
| Actual negative | FP = 10 | TN = 55 |
Metric arithmetic
| Metric | Calculation | Result |
|---|---|---|
| Precision | 30 / (30+10) | 0.75 |
| Recall | 30 / (30+5) | 0.857143 rounded |
| Accuracy | (30+55) / 100 | 0.85 |
| F1 | 60 / (60+10+5) | 0.80 |
| MAE example | (0+1+1) / 3 | 0.666667 rounded |
Learning method comparison
| Method | Target/structure | Key check |
|---|---|---|
| Linear regression | Numeric target | Residuals, extrapolation |
| Logistic / classifier | Class score/category | Threshold, calibration, positive class |
| KNN | Labeled neighbors | Scaling, K, ties |
| K-means | Unlabeled centers | Initialization, K, empty clusters |
| PCA | Variance projection | Train-only fit, target relevance |
| Reinforcement learning | Actions and rewards | Safe exploration, reward mismatch |
Revision and experiment checklist
Define unit and prediction time, establish baseline, split safely, fit preprocessing only inside training boundaries and use a metric linked to decision costs. A perfect toy training fit is not a deployment promise. Check uncertainty, privacy, group effects and drift.
Resources and sources
Official teaching and estimator documentation support the concepts. All demos here are original standard-library examples and do not depend on the version of scikit-learn documentation.
Comments
Post a Comment