Big Data Analytics Exam Prep

Unit 1Unit 2Unit 3

1. Big Data and the 5 Vs

1 mark: What is Big Data?

• Big Data is data whose scale, arrival rate or complexity requires more scalable handling than the available conventional tools can provide.

4 marks: Explain the five Vs with examples.

1. Volume: A retailer stores years of purchases. Partition storage and process parts in parallel.
2. Velocity: Payment events arrive continuously. Use a buffered ingestion path and bounded-latency processing.
3. Variety: Tables, JSON events, images and notes need different parsing and storage methods.
4. Veracity: Duplicate orders or missing sensor values require validation, cleaning and traceable sources. Value: A useful result, such as fewer failed deliveries, must justify the cost of collecting and processing data.

6 marks: A retailer has purchases, clickstreams and returns. Explain how the five Vs affect its system.

Volume + variety	Partition data; parse different inputs
Velocity + veracity	Buffer events; validate and deduplicate
Value	Measure a real outcome against cost

1. Volume: retain historical purchase and event partitions in scalable storage; size for growth and retention, not only today's files.
2. Velocity: buffer bursts so event producers are not tied directly to slow analytics. Fraud alerts may need seconds, while weekly reports can run in batches.
3. Variety: give purchases a defined schema, parse JSON clickstreams, and store large media separately with identifiers linking them to records.
4. Veracity: validate transaction IDs, deduplicate retries, standardize time zones and record missing fields. Preserve raw input for audit.
5. Value: measure an outcome, such as a reduction in stock-outs, rather than treating more stored bytes as success.
6. Trade-off: more replicas, fresher results and longer retention cost money. Choose each from business need and test result quality. Large volume alone does not prove Hadoop is necessary.

Common mistake: The five Vs describe challenges, not five software products. Do not define Big Data only as "very large data".

Back to units ↑

2. Data types and storage choices

1 mark: What is semi-structured data?

• Semi-structured data uses flexible keys or tags rather than a fixed table schema, as in JSON events.

4 marks: Compare structured, semi-structured and unstructured data.

Swipe the table to see all columns →

Point	Structured	Semi-structured	Unstructured
1. Organization	Fixed fields and types	Keys/tags; optional fields	No fixed table schema for the analysis
2. Example	Order table	JSON event	Photo, audio, free text
3. Processing	Validate columns; query rows	Parse keys; handle changes	Extract text/features as needed
4. Caution	Schema must be maintained	Still needs rules to interpret data	Not meaningless; can have metadata

6 marks: Should a growing retailer replace its RDBMS with Hadoop?

Live orders	RDBMS: updates and transactions
Historical logs	Distributed batch scan if justified
Decision	Combine tools when workloads differ

• Not automatically. Compare workload requirements before selecting a system.

1. Keep transactions safe: order entry and payments need constraints, updates and transaction guarantees. A relational database remains a strong fit.
2. Separate analytics: copy permitted historical and event data into an analytical store so heavy scans do not block live purchases.
3. Assess the bottleneck: measure data size, query latency, throughput and growth. Improve indexes, schemas or partitioning before assuming a distributed cluster is required.
4. Choose distributed work when justified: HDFS plus batch processing can spread large sequential scans across machines, but adds operations and coordination costs.
5. Handle variety deliberately: retain raw files with metadata, then create clean queryable datasets. "Schema-on-read" still requires a schema when interpreting data.
6. Conclude from cost and guarantees: a hybrid architecture is often reasonable. RDBMS products can also scale out; Hadoop is not inherently real-time, and HDFS is not a transactional database.

Back to units ↑

3. Analytics, lifecycle and decision-making

1 mark: What is Big Data Analytics?

• Big Data Analytics is the use of computational and statistical methods to find patterns, explain outcomes or support decisions from data that needs scalable handling.

4 marks: Compare descriptive, predictive and prescriptive analytics.

Swipe the table to see all columns →

Point	Descriptive	Predictive	Prescriptive
1. Question	What happened?	What may happen?	What should we do?
2. Input	Observed past events	Past data plus available current features	Predictions, costs and constraints
3. Example	Late-delivery count	Delay probability	Choose a route
4. Limit	Does not explain cause by itself	Prediction can be wrong	Action needs validation

6 marks: Explain the analytics lifecycle using delivery-delay prediction.

1. Define the decision and success measure

↓

2. Collect permitted order, route and GPS data

↓

3. Validate, clean and join records

↓

4. Explore and build time-aware features

↓

5. Train and test on separate time periods

↓

6. Serve results, act and monitor

1. Start with a measurable goal: reduce late deliveries without raising costs beyond a stated limit.
2. Collect only needed fields, with access controls and retention limits. Keep IDs and timestamps consistent across systems.
3. Check duplicate deliveries, impossible coordinates and missing values. Keep a record of transformations.
4. Construct features available at prediction time, such as distance and recent traffic. Do not use the actual arrival time as an input.
5. Test on later unseen deliveries and compare with a simple baseline. Data leakage can make an ineffective model look accurate.
6. Use predictions in a dispatch workflow and track errors, drift and actual delivery outcomes. A dashboard alone is not business value.

Back to units ↑

4. Typical analytical architecture and bottlenecks

1 mark: What is a data warehouse?

• A data warehouse is an integrated store organized for historical reporting and analysis, usually combining cleaned data from multiple operational sources.

4 marks: Explain a typical traditional analytical architecture.

1. Sources: operational systems record sales, finance and application events.
2. ETL: extract, clean and transform data before loading it into the analytical store.
3. Warehouse: combine historical data under consistent definitions; data marts serve a narrower subject.
4. Analysis: queries, OLAP and dashboards present results; freshness depends on the loading schedule.

6 marks: Analyze the limitations of a traditional warehouse-only design for Big Data.

Operational sources: sales, finance, applications

↓

ETL: extract, clean, transform and load

↓

Central warehouse → subject data marts

↓

Queries, OLAP analysis and dashboards

Nightly load	Old information for urgent decisions
Single bottleneck	Limited scan throughput
New data shape	Extra parsing and schema work

1. Ingestion delay: a nightly ETL job can leave a fraud report hours behind events. Match refresh intervals to the decision deadline.
2. Schema friction: new event fields and unstructured content may need preprocessing before fitting the warehouse schema.
3. Central bottlenecks: one storage or compute tier can limit concurrent scans. Scaling up has cost and hardware limits.
4. Data movement: repeatedly copying large datasets through ETL can consume bandwidth, space and time.
5. Operational contention: poorly separated analytics may compete with transaction processing. Isolate analytical workloads.
6. Balanced conclusion: these are limitations of a particular design, not a claim that every modern warehouse fails. Distributed or cloud warehouses, partitioning and incremental loading can address some limits; use a new architecture only where requirements justify it.

Common mistake: Do not write that all traditional databases cannot store Big Data. State the particular scale, latency or data-model constraint.

Back to units ↑

5. Modern analytical architecture and a streaming case

1 mark: Why is a new analytical architecture required?

• A new design is needed when the existing system cannot meet required scale, variety, freshness, resilience or cost.

4 marks: List four requirements of a modern Big Data architecture.

1. Scalable ingestion and storage: absorb growth and traffic bursts without losing traceability.
2. Appropriate processing latency: separate time-sensitive streams from heavy historical batch jobs.
3. Resilience and quality: recover from failures, validate records and handle duplicate or late events.
4. Governance and useful serving: restrict access, track lineage and expose results through a service, query engine or dashboard that supports the decision.

6 marks: Design a high-level architecture for real-time fraud detection.

Payment events + device signals

↓

Durable ingestion buffer

↓

Stream processing: validate, deduplicate, enrich

↓

Low-latency risk serving → alert or review

↓

Raw history → batch analysis → model evaluation

1. Ingest: assign event IDs and event timestamps. A durable buffer absorbs bursts and allows replay.
2. Validate: reject or quarantine malformed records; protect sensitive payment identifiers.
3. Enrich: combine the event with permitted recent-account features. Define handling of late or out-of-order data.
4. Score and act: calculate a risk score within the required latency and route high-risk cases to a controlled review or decision path.
5. Retain history: keep governed raw and cleaned datasets for audit and later batch analysis. HDFS-based MapReduce suits offline history, not a millisecond decision loop.
6. Measure and recover: monitor lag, failures, duplicate handling, false positives and detection rate. Replay and actions safe to repeat reduce duplicate effects, but exactly-once behavior must be designed end to end.

Back to units ↑

6. Challenges, distributed frameworks and trade-offs

1 mark: What is horizontal scalability?

• Horizontal scalability means increasing capacity by adding machines and distributing work or data across them.

4 marks: Explain four major challenges and one remedy for each.

1. Storage and processing: Partition data and process in parallel. Network transfer and skew can still limit speed.
2. Data quality: Validate, deduplicate and record provenance. Cleaning cannot recover facts never captured.
3. Security and privacy: Use least-privilege access, encryption and retention controls. Encryption alone does not make unauthorized use acceptable.
4. Scalability and failure: Add capacity, replicate data and retry failed work. Replication uses space and does not replace separate backups.

6 marks: Why are Big Data frameworks needed? Compare Hadoop MapReduce and Spark.

MapReduce	Large batch transformations
Spark	Dataflow and reusable cached work
Small simple workload	A database or simpler tool may be enough

• A distributed framework coordinates work that would otherwise require custom partitioning, scheduling, recovery and aggregation code.

1. Partitioning: divide a large dataset into manageable pieces. Correct partition keys help balance load.
2. Parallel execution: schedule tasks near data where possible. More nodes help only when work is parallelizable and coordination does not dominate.
3. Recovery: detect failed tasks and recompute work from recoverable input. This is fault tolerance, not immunity to every failure.
4. Hadoop MapReduce: uses map/shuffle/reduce stages and saved output between jobs; useful for large batch transformations, but repeated multi-job passes can add disk I/O and latency.
5. Spark: supports multi-stage dataflow and caching useful in iterative analysis. It can spill when data does not fit memory; "everything must fit in RAM" is wrong.
6. Selection: evaluate workload latency, iteration, cost, skills and data governance. Hadoop storage and Spark processing can coexist; a small dataset with simple queries may need neither.

Back to units ↑

Quick 1-mark revision

1 mark: What is veracity?

• Veracity is the trustworthiness and quality of data, including accuracy, consistency and completeness.

1 mark: What is value?

• Value is the useful, measurable benefit obtained from data, after considering collection, processing and decision costs.

1 mark: What is batch processing?

• Batch processing handles a collected set of records as a job, rather than processing every event as it arrives.

1 mark: What is stream processing?

• Stream processing handles events continuously as they arrive, with a defined latency and rules for state and event timing.

1 mark: Name two sources of Big Data.

• Application clickstream events and IoT sensor readings are two sources.

1 mark: What is fault tolerance?

• Fault tolerance is the ability to continue or recover useful operation after specified component failures.

1 mark: What is data provenance?

• Data provenance records where data came from and how it was transformed, helping audit and trust the result.

Technical references and scope

Original explanations and examples for revision. Technical behavior was checked against the references below. Hadoop 1 terms are labeled as legacy; Hadoop 2/3 use YARN. Configuration and installed versions can change defaults. Institutional names, personal information and supplied document headers are not reproduced.

• Apache Hadoop HDFS architecture
• Apache Hadoop MapReduce tutorial
• NIST Big Data definition
• Apache Spark dataflow and persistence guide
• IBM Big Data Analytics overview

Unit 2 - Hadoop and HDFS

1. Hadoop components, daemons and Hadoop 1 versus 2

1 mark: What is Hadoop?

• Apache Hadoop is a framework for distributed storage and processing of large datasets across a cluster.

4 marks: Compare Hadoop 1 and Hadoop 2.

Swipe the table to see all columns →

Point	Hadoop 1	Hadoop 2 / YARN
1. Storage	HDFS	HDFS, with added capabilities including high-availability (HA) support
2. Cluster control	JobTracker manages resources and jobs	ResourceManager allocates resources
3. Worker	TaskTracker runs map/reduce tasks	NodeManager manages containers
4. Job control	JobTracker coordinates jobs	Per-application ApplicationMaster coordinates tasks

6 marks: Explain the major Hadoop daemons and their responsibilities.

Storage metadata	NameNode
Stored blocks	DataNode
Cluster resources	ResourceManager + NodeManager
One application	ApplicationMaster

1. NameNode: maintains the filesystem file/directory names and file-to-block metadata. It decides where blocks should be placed.
2. DataNode: stores block replicas, serves data and reports its health and blocks to the NameNode.
3. SecondaryNameNode, in a non-HA setup: periodically merges filesystem image and edit information into a checkpoint. It is not an automatically promoted hot standby.
4. ResourceManager, in YARN: allocates cluster resources and starts application management through its applications machinery.
5. NodeManager, in YARN: launches and monitors containers and reports node resource status. An ApplicationMaster requests resources and coordinates application tasks.
6. Legacy boundary: Hadoop 1 uses JobTracker and TaskTracker for MapReduce. Do not put JobTracker in a YARN diagram. Hadoop Common supplies shared libraries and utilities; it is not another storage daemon.

Back to units ↑

2. HDFS design and read/write data flow

1 mark: Define NameNode and DataNode.

• The NameNode manages file names and block metadata; DataNodes store and serve the actual block bytes.

4 marks: Explain the design principles of HDFS.

1. Large sequential access: optimize throughput for large files, not tiny random updates.
2. Block-based storage: split a file into blocks distributed across DataNodes; configured replication keeps multiple copies.
3. Failure recovery: detect missing nodes or corrupt replicas, then read another replica and restore replication where possible.
4. Move work toward data: processing frameworks can schedule computation near stored blocks. HDFS supplies storage; it does not itself run all application computation.

6 marks: Explain HDFS file read and write operations with a diagram.

Metadata: Client ↔ NameNode

↓

Write data: Client → DataNode A → B → C

↓

Acknowledgments: C → B → A → Client

↓

Read data: Client ← selected DataNode

1. Read metadata: the client asks the NameNode for block locations and receives available replicas.
2. Read bytes: the client chooses a suitable replica, often a nearby one, and reads directly from that DataNode. It repeats this for later blocks.
3. Read failure: if a replica is unavailable or fails a checksum check, the client can try another and report the bad replica.
4. Create: for a write, the NameNode checks the request and chooses DataNodes according to placement policy.
5. Pipeline: the client sends packets to the first DataNode, which forwards them along the chain of DataNodes storing copies. Acknowledgments return in reverse order; a failed pipeline is reconfigured.
6. Close and metadata: the client completes the file and metadata is finalized. The diagram uses three replicas for illustration; replication is configurable. The NameNode manages placement but does not relay ordinary file data.

Back to units ↑

3. Replication, rack awareness and capacity numericals

1 mark: What is the HDFS replication factor?

• The replication factor is the desired number of replicas of a file's blocks.

4 marks: How does HDFS tolerate a DataNode failure?

1. DataNodes send heartbeats and block reports. The NameNode detects an unavailable node after its configured timeout and failure checks.
2. Clients read surviving replicas instead of depending on the failed node.
3. The NameNode identifies under-replicated blocks and schedules new replicas on healthy nodes when capacity is available.
4. Rack-aware placement reduces correlated failure risk, but cannot recover a block if all valid replicas are lost. HA protects NameNode availability; separate backups address different risks.

6 marks: A 300 MiB file uses 128 MiB blocks and replication factor 3. Calculate its block layout and storage.

• Assumptions: ordinary replicated HDFS storage, no erasure coding, no compression, no metadata or checksum overhead. MiB means 2²⁰ bytes.

Quantity	Calculation
Logical blocks	ceil(300 / 128) = 3
Block contents	128 MiB, 128 MiB, 44 MiB
Block replicas	3 blocks × 3 copies = 9 block replicas
Replicated payload	300 MiB × 3 = 900 MiB

1. The final 44 MiB block does not consume a full 128 MiB of payload storage just because the configured block size is 128 MiB.
2. If one replica is lost, the file can remain readable from another valid copy; the cluster tries to restore the desired count.
3. Placement should span failure domains. With the usual three-replica policy, not every copy is necessarily on a different rack.
4. For a 10 TiB dataset at replication 3, payload alone needs 30 TiB. If the target is at most 75% disk utilization, 30 / 0.75 = 40 TiB usable disk capacity is an illustrative lower planning bound, before other overhead and growth.
5. This arithmetic does not determine node count alone: disk capacity, rack layout, throughput, repair headroom and workload also matter.
6. Increasing block size reduces block metadata for large files, but does not combine millions of separate small files into one file.

Common mistake: Replication is not a substitute for backups: deletion or bad data can affect the logical file and its replicas.

Back to units ↑

4. HDFS commands and filesystem interfaces

1 mark: Which command lists HDFS files?

• hdfs dfs -ls /data lists entries under /data in the configured filesystem.

4 marks: Give four common HDFS commands with their purpose.

1. hdfs dfs -mkdir -p /user/student/bda: create a directory and any missing parents.
2. hdfs dfs -put events.txt /user/student/bda/: copy a local file into HDFS.
3. hdfs dfs -ls /user/student/bda: list entries and file metadata.
4. hdfs dfs -cat /user/student/bda/events.txt: display the stored text.

6 marks: Explain HDFS interfaces and a safe file-transfer workflow.

1. Prepare	Check filesystem, permissions and paths
2. Transfer	Upload to a directory you own
3. Verify	List, read and compare downloaded bytes

1. CLI: the filesystem shell provides file and directory operations through commands such as hdfs dfs -ls.
2. Java API: Hadoop's FileSystem interface lets applications open streams and manipulate paths programmatically.
3. Web interface: WebHDFS exposes filesystem operations over HTTP. Authentication and authorization still apply.
4. Prepare: identify the configured filesystem and use a directory you own. Check whether destination files already exist before copying.
5. Transfer and verify: create a directory, upload the local input, list it and read or download it. For a tiny text fixture, compare the downloaded bytes to the original, not just the filename. To download, use hdfs dfs -get /user/student/bda/events.txt events-copy.txt.
6. Change replication carefully: hdfs dfs -setrep -w 3 /user/student/bda/events.txt requests replication 3 and waits. Completion depends on the available nodes and space; it is not an instant guarantee. Avoid destructive cleanup commands while demonstrating basic transfer.

Back to units ↑

Command examples show syntax only; they were not executed against a live Hadoop cluster. Output depends on paths, content, permissions and configuration.

5. Data ingestion with Flume and Sqoop

1 mark: What is a Flume agent?

• A Flume agent is a process containing sources, channels and sinks that receive, buffer and deliver events.

4 marks: Compare Flume and Sqoop.

Swipe the table to see all columns →

Point	Flume	Sqoop
1. Input	Continuous events/logs	Relational tables
2. Method	Source → channel → sink	Parallel bulk import/export
3. Example	Web logs → HDFS	Sales table → HDFS
4. Caution	Reliability depends on source/channel	Legacy tool; retired in 2021

6 marks: Design a reliable log-ingestion pipeline and explain a database-import alternative.

Web/application events

↓

Reliable Flume source

↓

Durable file channel

↓

HDFS sink → time-partitioned log files

1. Source: choose a source that can track or acknowledge input. A simple unacknowledged tail command can lose events; a reliable-source design matters.
2. Channel: a file channel persists buffered events locally. A memory channel is faster in some workloads but can lose queued data when the agent dies.
3. Sink: send batches to HDFS and roll files by time or size to avoid an excessive number of tiny files.
4. Recovery: Flume transactions help move events between channel and destination. A retry can cause duplicates, so include unique event IDs and deduplicate if required.
5. Control: monitor backlog, disk capacity, failed writes and end-to-end lag. Protect log fields that contain personal data.
6. Structured alternative: for a relational table, a syllabus-level Sqoop import uses a database connection, table/query, target directory and split strategy to create parallel transfers. Verify keys, permissions and row counts; an incremental import needs a defined watermark and update policy, not merely a repeating full copy.

Back to units ↑

6. Hadoop Archives and the small-files problem

1 mark: What is a Hadoop Archive?

• A Hadoop Archive is an immutable .har directory that combines logical files into indexed part files.

4 marks: Why do small files burden HDFS, and how does HAR help?

1. Each file and its blocks require NameNode metadata; millions of tiny files can consume substantial metadata memory.
2. HAR packs content into part files and records each logical file's location in _index and _masterindex.
3. The HAR filesystem resolves a logical path to content inside those part files, reducing the physical filesystem entries needed for the archived copy.
4. HAR is immutable and does not automatically delete the original files or automatically compress their contents. Keeping both originals and archive means the original metadata burden remains.

6 marks: A thumbnail archive causes NameNode memory pressure. Propose and evaluate HAR.

Before	Many separate files and metadata entries
After	HAR index + larger part files
Constraint	Read-only archive; originals still exist

1. Diagnose: confirm that file count, rather than only data volume, is the metadata bottleneck. Count files and measure NameNode memory.
2. Group: choose a stable collection of old thumbnails suitable for read-only access; do not mix frequently updated files into an immutable archive.
3. Build: create a HAR with its index files and data parts, placing it in a separate verified destination. Keep the source until verification is complete.
4. Validate: compare file counts and sample or full content checks through the HAR filesystem. Ensure readers support HAR paths.
5. Retire sources only under an approved retention plan: archiving alone leaves originals unchanged. Removing originals would be a separate administrative decision with backup and verification requirements.
6. Evaluate: HAR can reduce pressure on file metadata for archived small files, but adds lookup overhead and is not ideal for frequent random updates. A different container layout may fit analytics better; block-size tuning alone does not fix the number of files.

Back to units ↑

7. Serialization, Avro, SequenceFile and MapFile

1 mark: What is serialization?

• Serialization converts structured in-memory data into a byte representation for storage or transmission.

4 marks: Compare SequenceFile, MapFile and Avro.

Swipe the table to see all columns →

Point	SequenceFile	MapFile	Avro container
1. Layout	Binary key-value records	Sorted data plus index	Schema header plus data blocks
2. Use	Typed Hadoop records	Key lookups	Cross-language exchange
3. Ordering	Not automatically sorted	Keys must be sorted	Not a key-lookup index
4. Schema	Hadoop types	Hadoop types	Writer/reader schema rules

6 marks: Explain Avro schema evolution with an example.

Old writer	id, amount
New reader	id, amount, currency with valid default
Result	Absent currency uses reader default

• Suppose old records contain id and amount. A new reader also wants currency.

1. Writer schema: describes the data actually stored. An Avro object container includes that schema in its header.
2. Reader schema: describes the fields the consuming program expects. Resolution compares writer and reader schemas.
3. Add with a default: if the new reader defines currency with a valid default such as "INR", that default can supply the field when reading old records where it is absent.
4. Missing without a default: if the reader requires the absent field and has no default, resolution fails. Schema evolution is constrained, not arbitrary.
5. Other changes: unused writer fields may be ignored; compatible type promotions are allowed by Avro's rules. Renaming or incompatible type changes need deliberate compatibility design.
6. Physical structure: data blocks may be compressed and separated by synchronization markers that support splitting. Test old-writer/new-reader and new-writer/old-reader combinations before changing a pipeline.

Back to units ↑

8. Compression, performance and system comparisons

1 mark: What is data compression?

• Data compression represents information using fewer bytes.

4 marks: Explain four compression decisions in Hadoop.

1. Codec: compare speed, compression ratio and installed support. Snappy emphasizes speed; gzip often suits compact archival output.
2. Input splitting: a plain gzip-compressed text file is normally not splittable by TextInputFormat, limiting it to one mapper per file.
3. Map output: compress intermediate data to reduce shuffle traffic, then measure the CPU-versus-network trade-off.
4. Container: a splittable container with sync markers can change the parallel-reading story; judge the codec together with its file format.

6 marks: Compare Hadoop with an RDBMS and traditional grid computing for a large batch workload.

Hadoop	Large independent batch tasks near data
RDBMS	Structured queries, constraints and transactions
Grid	Compute distribution; data may be remote

1. Workload: assume a repeated scan of large historical log files. State this before comparing systems; results differ for payment transactions or tightly coupled simulations.
2. Hadoop: HDFS distributes blocks and MapReduce spreads independent tasks with recovery. Data locality can reduce network movement.
3. RDBMS: relational schemas, SQL, constraints and transactions support structured data and updates. Modern systems can scale out; do not claim all RDBMS products are single-machine.
4. Grid computing: often distributes computation across machines while data may remain in separate shared or remote storage. A data-heavy workload can spend time moving input.
5. Hadoop trade-off: replication, scheduling, serialization and shuffle add overhead. It is poorly suited to many tiny low-latency queries or tightly communicating tasks.
6. Decision: choose from throughput, latency, recovery, cost and existing skills. Compression may help I/O-bound jobs but can slow CPU-bound ones. Validate with representative data instead of asserting "distributed is always faster."

Back to units ↑

Quick 1-mark revision

1 mark: What is the typical HDFS block size?

• A common Hadoop 2/3 default is 128 MiB (134217728 bytes), but the block size is configurable.

1 mark: What is a Hadoop daemon?

• A Hadoop daemon is a background service process, such as NameNode, DataNode, ResourceManager or NodeManager.

1 mark: What is rack awareness?

• Rack awareness uses network topology information in placement and scheduling to balance locality, bandwidth and resilience to rack-level failures.

1 mark: What is Hadoop Common?

• Hadoop Common supplies shared libraries and utilities used by the other Hadoop modules.

1 mark: What is a YARN container?

• A YARN container is an allocated bundle of resources on a node in which an application task or process can run.

1 mark: Is the SecondaryNameNode a hot standby?

• No; it creates checkpoints, while a configured HA standby has the separate role of taking over from a failed NameNode.

1 mark: What does a block report contain?

• A DataNode block report tells the NameNode which blocks the DataNode stores.

1 mark: Does HAR automatically compress files?

• No.

Technical references and scope

Original explanations and examples for revision. Technical behavior was checked against the references below. Hadoop 1 terms are labeled as legacy; Hadoop 2/3 use YARN. Configuration and installed versions can change defaults. Institutional names, personal information and supplied document headers are not reproduced.

• HDFS architecture
• HDFS user guide and checkpoints
• YARN architecture
• Filesystem shell
• Hadoop Archives
• Apache Avro specification
• Flume user guide
• Sqoop retirement notice

Unit 3 - MapReduce

1. Anatomy of a MapReduce job and WordCount

1 mark: What is MapReduce?

• MapReduce is a distributed batch-processing model in which mappers produce intermediate key-value pairs and reducers process grouped values for each key.

4 marks: Explain a job run on YARN in four stages.

1. Submit: the client supplies job configuration, input/output paths and application code. The InputFormat describes logical input splits.
2. Allocate: YARN starts an ApplicationMaster, which obtains containers from the ResourceManager; NodeManagers launch tasks.
3. Process: map tasks read records and emit pairs. The framework partitions, transfers, sorts and groups map output for reducers.
4. Finish: reducers write results through the OutputFormat/RecordWriter. The framework commits successful task output and reports job status.

6 marks: Trace WordCount from two input lines to final output.

• Input: line 1 is red blue red; line 2 is blue green. For a clear manual trace, treat each line as a separate mapper input; real split boundaries are chosen by the InputFormat.

Input records → Map: emit (word, 1)

↓

Optional local combine → partition and shuffle

↓

Sort/group by word → Reduce: sum counts

↓

Output files containing word totals

Stage	Exact trace
Map 1	(red,1), (blue,1), (red,1)
Map 2	(blue,1), (green,1)
Optional combine	Map 1: (red,2), (blue,1); Map 2: (blue,1), (green,1)
Grouped reducer input	blue → [1,1]; green → [1]; red → [2], if the shown combine occurs
Reduced results	blue → 2; green → 1; red → 2

• The totals sum to 5, matching the five input words. Without a combiner the red group is [1,1], with the same result. A combiner is optional, so correctness cannot depend on whether it runs. Several reducers produce separate output partitions; their files are not automatically one globally sorted stream.

Back to units ↑

2. Shuffle, sort, partitioning and bottlenecks

1 mark: What is shuffle?

• Shuffle transfers each mapper's intermediate records to the reducer responsible for their key partition.

4 marks: Explain shuffle and sort in four steps.

1. Map output is buffered, partitioned and sorted by intermediate key; buffers may spill sorted runs to local disk.
2. Each reducer fetches its assigned partition from the successful map outputs over the network.
3. Fetched runs are merged, with sorting and copying overlapping in time.
4. Equal keys are grouped for a reduce call. The reducer output is not automatically sorted again after user code emits it.

6 marks: A shuffle-heavy job runs slowly. Diagnose and improve it.

Too many bytes	Filter fields; valid combine; compress
Too many spills	Check memory and scratch disk
Hot key	Use a correct multi-stage strategy

1. Measure first: inspect bytes shuffled, spill counts, network utilization and task duration. Do not change every parameter at once.
2. Reduce map output: filter irrelevant fields and records early. Use a valid combiner for mergeable aggregates such as counts or sum/count pairs.
3. Compress: map-output compression can reduce network and disk bytes, but adds CPU work. Benchmark with the actual codec and data.
4. Manage memory and disk: reasonable sort buffers reduce excessive spills, while sufficient local scratch space prevents failed merges. Giving all memory to buffers can starve other work.
5. Balance partitions: inspect very frequent keys and skew. More reducers distribute many distinct keys, but do not split one very frequent key under an ordinary key partitioner.
6. Preserve correctness: for associative aggregates, salting a hot key into subkeys plus a later merge can spread work. A join or ordered operation needs a more careful design. Confirm equal results after optimization, not only shorter runtime.

Back to units ↑

3. Combiner correctness and a checked average example

1 mark: What is a combiner?

• A combiner optionally merges compatible map-output values locally to reduce data transfer without changing the final result.

4 marks: Distinguish combiner, partitioner and reducer.

Swipe the table to see all columns →

Point	Combiner	Partitioner	Reducer
1. Role	Local aggregation	Choose destination	Process grouped values
2. Where	Map-output processing	Map-output routing	Reduce task
3. Example	Sum local counts	Route by word	Sum all counts for word
4. Rule	Optional; safe if repeated	Keep grouped keys together	Emit final partition results

6 marks: Why is averaging local averages wrong? Give the correct MapReduce design.

• Dataset: partition A has [10,20,30] and partition B has [100].

Method	Calculation
Wrong: average two local means	A mean = 20; B mean = 100; (20 + 100) / 2 = 60
Correct: preserve totals and counts	A → (60,3); B → (100,1); merged → (160,4)
Final average	160 / 4 = 40

1. Emit (category, (amount,1)) from each valid input row.
2. The combiner merges pairs by adding sums and counts: (s1+s2, n1+n2).
3. The reducer applies the same merge, then divides total sum by total count once.
4. This remains correct when the combiner does not run or runs repeatedly, because the intermediate representation preserves the needed information.
5. Skip or explicitly handle missing amounts rather than silently treating them as zero; a category with no valid rows has no defined mean.
6. The illustrated arithmetic was independently checked locally. It demonstrates aggregation semantics, not an executed Hadoop cluster job.

Back to units ↑

4. Types, InputFormat, RecordReader and OutputFormat

1 mark: What is an InputSplit?

• An InputSplit describes the logical input assigned to one mapper; it is not an HDFS block or a copy of the data.

4 marks: Explain the roles of input and output components.

1. InputFormat: validates input, defines splits and provides a RecordReader.
2. RecordReader: turns the split's bytes into records while respecting record boundaries. TextInputFormat normally provides byte offset as key and line text as value.
3. Types: map input, map output and final output key/value types must match the selected formats and task code. Hadoop commonly uses types such as Text, LongWritable and IntWritable.
4. OutputFormat and RecordWriter: validate the destination and convert emitted pairs into stored output. TextOutputFormat writes text; SequenceFileOutputFormat writes binary key-value records.

6 marks: Choose formats for multiline logs and typed intermediate data.

InputFormat	Describe splits and provide reader
RecordReader	Produce complete records
OutputFormat	Write and validate output layout

1. Simple lines: use TextInputFormat when one line is one record. A mapper receives an offset and line, not automatically parsed database columns.
2. Multiline events: if one logical event spans several lines, define a RecordReader that detects complete event boundaries, including those near split edges.
3. Split policy: decide whether the format is safely splittable. Never split in the middle of a record without a recovery rule.
4. Validation: count malformed records and route them to a separate error output rather than silently shifting fields into the wrong columns.
5. Intermediate storage: SequenceFileInputFormat/OutputFormat can preserve typed key-value records between compatible Hadoop jobs; schema-oriented formats may be preferable for cross-language pipelines.
6. Compression: one large plain gzip text file is normally processed by one mapper with TextInputFormat. A large file on many HDFS blocks does not by itself guarantee many map tasks. Test parsing and expected record counts at boundaries.

Back to units ↑

5. Failures, task execution and speculation

1 mark: What is speculative execution?

• Speculative execution launches an extra attempt of a suspected slow task on another node.

4 marks: Explain task and node failure handling.

1. A failed task attempt can be retried, usually on another suitable node, up to configured limits. Repeated deterministic exceptions may still fail the job.
2. When a node is lost, unfinished tasks can be rescheduled. Completed map output kept only on that node's local disk may need to be recomputed.
3. HDFS replicas supply recoverable input when one DataNode fails; they do not automatically replicate every intermediate map-output file.
4. ApplicationMaster and ResourceManager recovery depend on configured restart or high-availability behavior. Fault tolerance has limits and is not a promise that all jobs survive every coordinator failure.

6 marks: Compare a slow task, a failed worker and a failed JobTracker.

Slow node	Speculative attempt may help
Hot key	Repartition/rethink work; copying is not enough
Lost worker	Retry tasks and recreate lost map output
Lost coordinator	Recovery depends on version and setup

1. Slow task: a straggler may be caused by a weak node. Speculation tries another attempt, at the cost of extra CPU, I/O and network.
2. Skew: if a task is slow because it owns a genuinely huge key group, copying the same work does not remove the imbalance.
3. Worker loss: in Hadoop 1, missed TaskTracker heartbeats lead the JobTracker to reschedule work; lost local map output must be regenerated.
4. Legacy coordinator loss: the Hadoop 1 JobTracker combines cluster and job control and is a central failure risk. State recovery depends on the version and configuration; do not say every job always resumes unchanged.
5. YARN separation: ResourceManager allocates resources, NodeManagers manage containers, and the ApplicationMaster coordinates the application. Configured RM HA/recovery and AM retry mechanisms address different failure scopes.
6. Side effects: retries or speculative attempts can execute user code again. Use task-output commit mechanisms and external effects safe to repeat; sending an email or charging an account directly inside a retried mapper is unsafe.

Back to units ↑

6. Map-side and reduce-side joins with a full trace

1 mark: What is a reduce-side join?

• A reduce-side join emits both datasets keyed by the join key, shuffles matching keys to the same reducer, and joins their tagged records there.

4 marks: Compare map-side and reduce-side joins.

Swipe the table to see all columns →

Point	Map-side join	Reduce-side join
1. Requirement	Small lookup fits in each mapper, or compatible sorted partitions	Both sides keyed and tagged for shuffle
2. Movement	Avoid join shuffle if requirements hold	Shuffle by join key
3. Risk	Lookup memory or incompatible partitions	Hot keys, shuffle cost and memory
4. Choose	Huge orders + small unique customer lookup	Two large unsorted tables

6 marks: Join customers and orders using both strategies.

Order O1, customer C1	Ana, amount 40
Order O2, customer C1	Ana, amount 60
Order O3, customer C3	No match: omitted in this inner join

• Inner-join input: customers are (C1,Ana), (C2,Bo); orders are (O1,C1,40), (O2,C1,60), (O3,C3,10).

1. Map-side lookup: load the customer mapping into each mapper if it fits with safe memory headroom. Read orders and look up the customer ID.
2. Map-side output: emit (O1,Ana,40) and (O2,Ana,60). O3 has no matching customer and is absent from this inner join.
3. Reduce-side mapping: emit customer records as (C1,(C,Ana)), (C2,(C,Bo)); emit orders as (C1,(O,O1,40)), (C1,(O,O2,60)), (C3,(O,O3,10)). Tags identify record origin.
4. Grouping: partition on customer ID. C1 reaches one reducer group containing Ana and both orders; C2 has only a customer; C3 has only an order.
5. Joining: match the C1 records and emit the same two result rows. C2 and C3 produce none for an inner join. Outer joins need an explicit unmatched-row rule.
6. Multiplicity: duplicate customer keys can produce several matches per order. Do not overwrite duplicates in a lookup unless uniqueness is guaranteed. The trace assumes unique customer IDs and was checked independently as relational matching logic.

Back to units ↑

7. Secondary sorting and global-order traps

1 mark: What is secondary sorting?

• Secondary sorting arranges records within a primary-key group by another field, usually by using a composite key, a compatible partitioner and separate sort/grouping comparators.

4 marks: Explain four configuration steps for sorting sales by region and date.

1. Composite key: emit (region,date) as key, with the sale in the value. Include the date in the value too if the reducer needs each record's date.
2. Partitioner: partition only by region so all records for the region reach one reducer.
3. Sort comparator: compare region first, then date, using a chronological representation such as ISO YYYY-MM-DD.
4. Grouping comparator: group by region only. Each reduce call receives one region's values ordered by the composite-key sort, provided the configuration is consistent.

6 marks: Trace a secondary sort and explain what it does not guarantee.

• Input: (East,2026-10-03,30), (West,2026-10-02,20), (East,2026-10-01,10).

Configuration/result	Meaning
Map key	(region,date); value includes (date,amount)
Partition	Same region → same reducer
Sort	(East,2026-10-01), (East,2026-10-03), (West,2026-10-02)
Group	East gets [(2026-10-01,10),(2026-10-03,30)]; West gets [(2026-10-02,20)]

1. The displayed combined ordering is the conceptual comparator order, not a claim about concatenating independently hashed reducer files.
2. Grouping only on region keeps that region in one reduce call; sorting on the complete key determines the order entering the group.
3. Partitioning on the whole composite key could split one region across reducers and break the desired grouping.
4. Hash partitioning several reducers does not guarantee global timestamp order. Region-first ordering is not date-first ordering either.
5. For one globally ordered output, use an appropriate total-order/range partition design with ordered output handling, or one reducer when the scale permits.
6. Reducer code must preserve the intended emission order. The framework does not sort arbitrary final reducer output again.

Back to units ↑

8. Job scheduling, pipelines and counters

1 mark: What is job scheduling?

• Job scheduling decides how available cluster resources are allocated among waiting and running applications, subject to policies such as order, fairness, queue capacity and priority.

4 marks: Compare FIFO, Fair and Capacity scheduling.

Swipe the table to see all columns →

Point	FIFO	Fair	Capacity
1. Goal	Submission-order service	Fair shares over time	Queue capacity entitlements
2. Useful for	Simple queue	Mixed workloads/pools	Multiple departments
3. Risk	Large job delays later jobs	Not equal finish times	Needs capacity and queue design
4. Preemption	Not its defining feature	Depends on configuration	Supported when configured

6 marks: Design a two-job web-log pipeline with monitoring and skew handling.

Raw logs → Job 1: validate and count per URL

↓

Job 1 successful output → Job 2: rank URLs

↓

Final report + malformed-record counters

1. Job 1 map: parse each event and emit (URL,1) for valid rows. Count malformed or missing URLs separately; document what counts as valid.
2. Job 1 reduce: sum by URL. A sum combiner is valid. Write a typed intermediate dataset with a stable schema.
3. Dependency: submit or release Job 2 only when Job 1 has succeeded and its committed output is complete. Do not read temporary partial files.
4. Job 2: compute ranking or top-k with a stated tie rule. A single global reducer may be a bottleneck; larger workloads can use local candidates followed by a final merge.
5. Monitoring: inspect counters, bytes and task-duration distributions. Counters summarize job events; they are not an instantaneous production alerting system.
6. Skew and recovery: if one URL dominates, a valid two-stage salted count can spread aggregation before merging by URL. Preserve original keys in the final merge and use distinct verified output paths so retries do not silently mix old and new results.

Common mistake: Chaining jobs usually materializes intermediate output between jobs; it is not the same as an always-running low-latency stream pipeline.

Back to units ↑

Quick 1-mark revision

1 mark: Define Mapper.

• A mapper transforms one input key-value record into zero or more intermediate key-value pairs.

1 mark: Define Reducer.

• A reducer processes the values grouped under an intermediate key and emits zero or more output pairs.

1 mark: What is a Partitioner?

• A Partitioner routes intermediate keys to reducer partitions, keeping keys that must be grouped together in the same partition.

1 mark: What is data locality?

• Data locality means running computation near its input data where possible, reducing network transfer.

1 mark: What is a RecordReader?

• A RecordReader converts split bytes into complete input key-value records for a mapper.

1 mark: What is a counter?

• A counter is a named numeric measure used to record events such as invalid rows or processed records during a job.

1 mark: What is a map-only job?

• A map-only job has zero reducers, so output goes to the filesystem without the reduce shuffle/sort phase.

1 mark: What is pipelining MapReduce jobs?

• It means coordinating dependent jobs so one job's committed output becomes another's input; ordinary job chaining uses saved intermediate data.

Technical references and scope

Original explanations and examples for revision. Technical behavior was checked against the references below. Hadoop 1 terms are labeled as legacy; Hadoop 2/3 use YARN. Configuration and installed versions can change defaults. Institutional names, personal information and supplied document headers are not reproduced.

• Apache MapReduce tutorial
• Reducer API and secondary sorting
• Map-side join prerequisites in Apache examples
• YARN architecture
• Capacity scheduler