Big Data Analytics | Semester 7 Complete Core Notes

Semester 7 - Page 2: Big Data Analytics. 25 numbered topics, highlighted rules, worked cases, answered exam questions, diagrams, comparison tables and three complete Python demos. General common-core guide, not an official syllabus. Verify required framework versions/algorithms against your course outline.

Previous: Semester 7 - Page 1, Image ProcessingSemester indexNext: Semester 7 - Page 3, Entrepreneurship

Run each Python block separately: Python 3, standard library only; finite synthetic data in local memory. These verify calculation/policy semantics, not Spark, Hadoop, Kafka or distributed fault-tolerance behavior. Exactly-once claims need a named boundary and compatible source/sink design.

Topic map

1. Workload-relative big data and the Vs
2. Volume, velocity and sizing assumptions
3. Distributed architecture and failure models
4. Partitioning and data locality
5. Replication versus backup
6. HDFS concepts and block storage
7. Map, shuffle and reduce
8. Combiners and algebraic correctness
9. Spark RDDs and lineage
10. DataFrames, schema and SQL planning
11. Lazy transformations and actions
12. Shuffle, joins and skew
13. Caching, memory and driver limits
14. Batch, streams and ETL/ELT
15. Event time and processing time
16. Tumbling/sliding windows and boundaries
17. Watermarks and late-data policies
18. Retries, checkpoints and deduplication
19. Delivery semantics and end-to-end effects
20. Formats, schema evolution and quality
21. NoSQL models and query tradeoffs
22. Analytics types and causal caution
23. Evaluation, time splits and numerical metrics
24. Privacy, access and operational evidence
25. End-to-end pipeline and exam checklist

1. Workload-relative big data and the Vs

Big data describes workloads whose storage, memory, processing or freshness constraints make one-machine handling impractical. Distribution adds coordination cost, not automatic speed.

Worked example: Ten million 500-byte records are five billion raw bytes; event rate and required freshness also matter.

Exam question: Is there one universal file-size cutoff for big data?
Answer: No. Workload, hardware and constraints determine the problem.

2. Volume, velocity and sizing assumptions

Estimate payload, ingestion rate, retention, indexes, replicas and overhead separately. Use explicit decimal/binary units and measured variation.

Worked example: At 10000 events per second, ten million events represent 1000 seconds of arrival; this excludes transport/storage overhead.

Exam question: Does raw payload equal the final storage bill?
Answer: No. Replication, metadata, logs and compression change it.

3. Distributed architecture and failure models

Separate clients, coordination, workers, storage and sinks. State what failures/network conditions the design tolerates rather than promising zero downtime.

Worked example: Losing one replica may still require detection/recovery and can interact with network partitions or quorum rules.

Exam question: Does adding nodes eliminate coordination problems?
Answer: No. Distribution creates new failure and communication paths.

4. Partitioning and data locality

Partitioning spreads records/work by hash, range or another policy. Locality can reduce transfer; cross-partition joins and uneven ranges cost more.

Worked example: Hashing user keys spreads ownership under its distribution assumptions, while one dominant key can remain hot.

Exam question: Does partitioning guarantee equal worker load?
Answer: No. Key frequencies and workload matter.

5. Replication versus backup

Replication maintains copies for availability/fault tolerance under a system policy; backup preserves recoverable historical state. They address different failures.

Worked example: A mistaken delete can propagate to every current replica, so a protected earlier copy and restore test remain needed.

Exam question: Can replicas alone replace recovery history?
Answer: No. They may share logical corruption/deletion.

6. HDFS concepts and block storage

HDFS separates filesystem metadata coordination from data-block storage and uses replication for fault tolerance. Layout and configured policy affect real capacity/access.

Worked example: A replicated block needs available copies and metadata service behavior; do not infer every deployment setting from a diagram.

Exam question: Does a generic HDFS description prove a live cluster's replication factor?
Answer: No. Inspect its actual configuration/state.

6A. Hadoop YARN: Resource Manager, Node Manager and ApplicationMaster

YARN (Yet Another Resource Negotiator) separates cluster resource management from application-specific task coordination. HDFS stores data; YARN manages the resources used to run applications. The ResourceManager, NodeManager and ApplicationMaster have different scopes.

Role of the Resource Manager (RM)

The ResourceManager is the cluster-level resource authority. It allocates available resources, such as memory and CPU, among applications according to the configured scheduling policy. It has two main components:

• Scheduler: allocates resources to applications, subject to capacity, queues and resource requirements. The Scheduler itself does not track application progress or restart failed tasks.
• ApplicationsManager: accepts application submissions, arranges the first container for the application's ApplicationMaster, and provides the service for restarting the ApplicationMaster container on failure.

Role of the Node Manager (NM)

The NodeManager is the per-machine agent on a YARN worker node. It launches and manages containers on that machine, monitors resource usage such as CPU, memory, disk and network, and reports node/container information to the ResourceManager. It does not decide the resource allocation policy for the whole cluster.

Role of the ApplicationMaster (AM)

Each application has an ApplicationMaster that negotiates resources from the ResourceManager and works with NodeManagers to execute and monitor that application's tasks. Application-specific progress and task coordination belong here, rather than in the ResourceManager's Scheduler. An application may be a single job or a directed acyclic graph of jobs.

Component	Scope	Main responsibility	Do not confuse it with
ResourceManager	Cluster	Resource allocation and application admission	NameNode, which manages HDFS metadata
NodeManager	One worker machine	Container lifecycle and local resource monitoring	DataNode, which stores HDFS blocks
ApplicationMaster	One application	Request resources and coordinate/monitor tasks	The cluster-wide Scheduler

High availability note: A simple deployment can use one ResourceManager. In a ResourceManager HA setup, one RM is active at a time and one or more RMs can be standby, ready to take over. Therefore, "exactly one ResourceManager process in every cluster" is not a general rule. Failover behavior depends on the HA configuration.

Worked example: a word-count application

1. The client submits the application to the ResourceManager.
2. The ApplicationsManager arranges the first container for the ApplicationMaster.
3. The ApplicationMaster requests resources for the application's tasks.
4. The Scheduler allocates suitable resources; the ApplicationMaster asks the relevant NodeManagers to launch task containers.
5. NodeManagers manage those local containers and report their status/resource use. The ApplicationMaster tracks the application's task progress.

Conclusion: The RM decides how cluster resources are allocated. Each NM runs and monitors containers on its machine. The AM coordinates the tasks of one application. A container represents allocated execution resources, not necessarily a Docker container.

Short exam questions and answers

Q1. State the role of the ResourceManager.

It is the cluster-level resource authority in Hadoop YARN. Its Scheduler allocates resources among applications, while its ApplicationsManager accepts submissions and arranges the ApplicationMaster's initial container.

Q2. State the role of the NodeManager.

It is the per-machine YARN agent. It launches and manages containers, monitors their resource usage, and reports node/container information to the ResourceManager.

Q3. How is the ApplicationMaster different from the NodeManager?

The AM coordinates one application's tasks across nodes. An NM manages containers on one machine, which may run work from several applications.

Q4. Does the ResourceManager Scheduler restart failed tasks?

No. It allocates resources. Application-specific task monitoring and coordination belong to the ApplicationMaster; the ApplicationsManager provides the service for restarting a failed AM container.

Q5. Can a YARN cluster have more than one ResourceManager?

Yes. HA deployments use active/standby ResourceManagers, with one active at a time.

Official references: Apache Hadoop YARN architecture and ResourceManager high availability.

7. Map, shuffle and reduce

Map emits intermediate key/value data, shuffle brings matching keys together, and reduce combines each key's group. Transfer and skew often dominate.

Worked example: Map cat dog cat to counts, group cat values together and sum to two; this local teaching analogy is not a cluster benchmark.

Exam question: Can a local word count demonstrate distributed fault tolerance?
Answer: No. It illustrates aggregation semantics only.

Input records
 -> map: intermediate key/value
 -> shuffle: matching keys together
 -> reduce: one result per key
 -> output under sink policy

Shuffle communication and hot keys can dominate cost.

8. Combiners and algebraic correctness

A combiner is optional and cannot be assumed to run once or at all. Combining must preserve the final result under legal regrouping.

Worked example: Average partition means are wrong when partition sizes differ; carry sum and count, then divide once.

Exam question: Is mean-of-means always the global mean?
Answer: No. Weight by counts or combine sufficient statistics.

Sum/count combiner versus unweighted means

partitions = [[2,4], [10]]
stats = [(sum(part), len(part)) for part in partitions]
mean = sum(s for s,n in stats) / sum(n for s,n in stats)
wrong = sum(s/n for s,n in stats) / len(stats)
assert mean == 16/3 and wrong == 6.5
print("partition sum/count:", stats)
print("global mean:", format(mean,".6f"))
print("unweighted mean-of-means:", wrong)
print("Local algebra demo, not a Hadoop combiner execution claim.")

Checked output:

partition sum/count: [(6, 2), (10, 1)]
global mean: 5.333333
unweighted mean-of-means: 6.5
Local algebra demo, not a Hadoop combiner execution claim.

9. Spark RDDs and lineage

RDDs are distributed collections with transformation dependencies. Lineage supports recomputation under relevant assumptions; persistence policy and source availability still matter.

Worked example: A map/filter chain creates dependency structure; executor loss is not proof that every external side effect can safely be repeated.

Exam question: Does lineage make arbitrary external writes idempotent?
Answer: No. Side effects need their own design.

10. DataFrames, schema and SQL planning

DataFrames represent named typed columns and enable SQL-like optimization. Validate schemas/units and inspect plans when performance or semantics matter.

Worked example: Filter amount>=50 then group customer sums gives a different meaning from filtering grouped totals at 50.

Exam question: Can optimizer language excuse changing the requested query meaning?
Answer: No. Valid rewrites must preserve semantics.

11. Lazy transformations and actions

Transformations commonly build execution dependencies/plans; actions trigger work. Some APIs include analysis/eager behaviors, so inspect the actual operation.

Worked example: select/filter/groupBy define a plan; count requests a result. Do not time only plan construction as processing speed.

Exam question: Does an unevaluated plan prove output correctness?
Answer: No. Run the action and check actual results.

12. Shuffle, joins and skew

Redistribution groups data across workers and can cost network, disk and memory. Join strategy and skew depend on data sizes/distribution and actual plan.

Worked example: One anonymous-user key can send most records to one partition even with many workers.

Exam question: Does more parallelism necessarily solve a single hot key?
Answer: No. The key's partitioning behavior remains a bottleneck.

13. Caching, memory and driver limits

Caching is a deliberate reuse choice; data may spill/evict or be recomputed. collect brings results to the driver, which can become a memory bottleneck.

Worked example: A huge table should not be collected merely to inspect it; use bounded samples/aggregates with clear limitations.

Exam question: Does Spark mean all data always fits permanently in RAM?
Answer: No. Storage levels and execution determine behavior.

14. Batch, streams and ETL/ELT

Batch processes bounded input; streams handle ongoing input with state/time policies. ETL transforms before target loading; ELT transforms after loading.

Worked example: A daily purchase report and continuously updated dashboard have different freshness/reprocessing needs.

Exam question: Is streaming automatically zero-latency?
Answer: No. Triggers, sources, state and sinks affect latency.

15. Event time and processing time

Event time reflects occurrence timestamps; processing time reflects handling/arrival. Their distinction matters for late/out-of-order data.

Worked example: An event arriving 10:12 stamped 10:04 belongs to the 10:00-10:05 event-time window if accepted by the policy.

Exam question: Does arrival order equal occurrence order?
Answer: Not generally. Delays and retries reorder events.

16. Tumbling/sliding windows and boundaries

A tumbling window partitions time into non-overlapping intervals; sliding windows overlap. Define units, interval endpoints and timestamp parsing explicitly.

Worked example: With five-minute half-open windows, minute 4 belongs to[0, 5) and minute 5 to[5, 10).

Exam question: Does an endpoint belong to both adjacent half-open tumbling windows?
Answer: No. The upper bound is excluded and next lower bound included.

5-minute half-open tumbling windows:
[0,5): minute 0,1,2,3,4
[5,10): minute 5,6,7,8,9

Event timestamps assign windows; lateness acceptance is a separate policy.

17. Watermarks and late-data policies

Watermarks/state cleanup express system-specific assumptions about progress/lateness. Accepted/drop/update behavior depends on actual query and configuration.

Worked example: A late event may update an earlier result or be omitted under the configured state policy; do not promise acceptance from timestamp alone.

Exam question: Can a generic watermark definition establish exact sink output?
Answer: No. Query mode, state and runtime rules matter.

18. Retries, checkpoints and deduplication

Retries can repeat processing/delivery. Stable event IDs, checkpoints and idempotent writes address different boundaries; retention horizons limit deduplication.

Worked example: Two deliveries of order 42 should not blindly add twice. Conflicting payloads under one ID need explicit reconciliation.

Exam question: Does deduplication by an ID prove conflicting payloads are resolved correctly?
Answer: No. A conflict policy is separate.

Stable-ID deduplication with explicit conflict rejection

from collections import defaultdict
# Event times are integer minutes; finite replay, no watermark implementation.
events = [("a",1,10),("b",4,20),("a",1,10),("c",5,30)]
seen = {}; windows = defaultdict(int)
for key, minute, amount in events:
    payload = (minute,amount)
    if key in seen:
        if seen[key] != payload: raise ValueError("conflicting payload")
        continue
    seen[key] = payload
    start = minute//5*5
    windows[start] += amount
assert dict(windows) == {0:30,5:30}
print("unique IDs:", len(seen))
print("window totals:", sorted(windows.items()))
conflict = ("a",1,99)
print("conflicting a detected:", seen[conflict[0]] != conflict[1:])
print("Finite in-memory policy demo; no distributed checkpoint or late-data guarantee.")

Checked output:

unique IDs: 3
window totals: [(0, 30), (5, 30)]
conflicting a detected: True
Finite in-memory policy demo; no distributed checkpoint or late-data guarantee.

19. Delivery semantics and end-to-end effects

At-most-once, at-least-once and exactly-once claims need a named boundary. Source, processor and sink must cooperate for a particular end-to-end guarantee.

Worked example: A transactional internal write does not automatically include a separate email/payment side effect outside that transaction.

Exam question: Can a processor's exactly-once claim cover every external system?
Answer: No. Verify the scope and sink behavior.

20. Formats, schema evolution and quality

Typed schemas, field units, null/error policies and evolution rules prevent silent meaning changes. Columnar formats can help selective analytics but are not universal fixes.

Worked example: Changing amount from cents to currency units without a schema/version rule multiplies reported value by 100.

Exam question: Does a successful parse guarantee valid business data?
Answer: No. Semantic validation is separate.

21. NoSQL models and query tradeoffs

Key-value, document, wide-column and graph models support different access patterns. Choose by queries, consistency/failure behavior and operational limits.

Worked example: A graph traversal and a single-key lookup have different needs; do not choose a store solely because it is labeled NoSQL.

Exam question: Does NoSQL imply every system lacks transactions or schemas?
Answer: No. Features depend on the actual system.

22. Analytics types and causal caution

Descriptive, diagnostic, predictive and prescriptive analyses answer different questions. Correlation does not identify causation without an appropriate design.

Worked example: Two metrics may move together because of a common cause; a dashboard trend alone does not prove an intervention worked.

Exam question: Does a correlated feature establish causal effect?
Answer: No. Confounding and design need consideration.

23. Evaluation, time splits and numerical metrics

Use unseen relevant data and time-aware boundaries for forecasting; fit preprocessing on training only. State metric counts/units and compare baselines.

Worked example: Actual[10, 20, 30], predicted[12, 17, 34] have absolute errors[2, 3, 4], so MAE is 3.

Exam question: Does low training error establish future generalization?
Answer: No. Evaluate against the intended future/unseen setting.

Grouped filtering and exact MAE calculation

from collections import defaultdict
orders = [("A",100),("B",30),("A",50)]
totals = defaultdict(int)
for customer, amount in orders:
    if amount >= 50: totals[customer] += amount
actual = [10,20,30]; predicted = [12,17,34]
errors = [abs(a-p) for a,p in zip(actual,predicted)]
assert dict(totals) == {"A":150} and errors == [2,3,4]
print("filtered customer sums:", sorted(totals.items()))
print("absolute errors:", errors)
print("MAE:", sum(errors)/len(errors))
print("Local finite-data calculation, not a Spark execution or generalization result.")

Checked output:

filtered customer sums: [('A', 150)]
absolute errors: [2, 3, 4]
MAE: 3.0
Local finite-data calculation, not a Spark execution or generalization result.

24. Privacy, access and operational evidence

Minimize sensitive fields, enforce access, protect logs/secrets and document retention. Observe lag, throughput, errors and restore/replay behavior without overstating live state.

Worked example: A teaching dataset uses synthetic customers; pseudonymous real records can still identify people in context.

Exam question: Does distribution remove privacy duties?
Answer: No. Copies and pipelines can increase disclosure surfaces.

25. End-to-end pipeline and exam checklist

Trace ingestion, validation, partitioning, processing, output and recovery boundaries. Show correct units/results, retry policy and limitations.

Worked example: The local demos below verify aggregation/windows/dedup rules on finite toy inputs; they do not prove Spark/Kafka/HDFS runtime behavior.

Exam question: What makes a sound big-data answer?
Answer: Explicit workload/failure assumptions, correct semantics, worked calculations and scoped evidence.

Source -> validate/schema -> partition
 -> process/checkpoint -> permitted sink
 -> reconciliation + replay/restore evidence

One internal guarantee does not cover every external side effect.

Storage distinctions

Mechanism	Purpose	Limit
Partition	Spread records/work	Skew and cross-partition costs
Replication	Current copies/availability	Logical deletion can propagate
Backup	Recover historical state	Actual restore test needed
Cache	Reuse chosen computation/data	Eviction/spill/recompute possible

Time and processing models

Term	Meaning	Check
Batch	Bounded run input	Replay/input version
Stream	Ongoing input/state	Checkpoint and sink policy
Event time	Occurrence timestamp	Parsing and late-data rule
Processing time	Handling/arrival clock	Not occurrence order
Tumbling window	Non-overlapping intervals	Endpoint/unit convention

Semantic and evaluation pitfalls

Mistake	Better evidence
Mean of unequal partition means	Combine sums and counts
Counting retry twice	Stable ID plus conflict/idempotency policy
Exactly-once without scope	Name source/processor/sink boundary
Timing plan construction as full work	Run actions and measure actual result
Huge collect to inspect data	Bounded samples/aggregates with stated limits
Training score as forecast proof	Relevant unseen/time-separated evaluation

Revision checklist

Check units, raw versus replicated storage, partition skew, shuffle, lazy actions and driver limits. Explain event versus processing time, window endpoints and retry boundaries. State privacy and restore assumptions. Local toy output is not evidence of a live cluster guarantee.

Resources and sources

Original sources are preserved alongside official framework references. Runtime guarantees/settings depend on version, query and source/sink configuration.

• Apache Spark SQL and DataFrames guide
• Apache Spark RDD programming guide
• Apache Spark Structured Streaming guide
• Hadoop MapReduce tutorial
• HDFS user guide
• Spark RDD guide HTML

Previous: Semester 7 - Page 1, Image ProcessingSemester indexNext: Semester 7 - Page 3, Entrepreneurship