Big Data Analytics Exam Prep

Unit 1Unit 2Unit 3

1. Big Data and the 5 Vs

1 mark: What is Big Data?

• Big Data is data whose scale, arrival rate or complexity requires more scalable handling than the available conventional tools can provide.

4 marks: Explain the five Vs with examples.

1. Volume: A retailer stores years of purchases. Partition storage and process parts in parallel.
2. Velocity: Payment events arrive continuously. Use a buffered ingestion path and bounded-latency processing.
3. Variety: Tables, JSON events, images and notes need different parsing and storage methods.
4. Veracity: Duplicate orders or missing sensor values require validation, cleaning and traceable sources. Value: A useful result, such as fewer failed deliveries, must justify the cost of collecting and processing data.

6 marks: A retailer has purchases, clickstreams and returns. Explain how the five Vs affect its system.

Volume + variety	Partition data; parse different inputs
Velocity + veracity	Buffer events; validate and deduplicate
Value	Measure a real outcome against cost

1. Volume: retain historical purchase and event partitions in scalable storage; size for growth and retention, not only today's files.
2. Velocity: buffer bursts so event producers are not tied directly to slow analytics. Fraud alerts may need seconds, while weekly reports can run in batches.
3. Variety: give purchases a defined schema, parse JSON clickstreams, and store large media separately with identifiers linking them to records.
4. Veracity: validate transaction IDs, deduplicate retries, standardize time zones and record missing fields. Preserve raw input for audit.
5. Value: measure an outcome, such as a reduction in stock-outs, rather than treating more stored bytes as success.
6. Trade-off: more replicas, fresher results and longer retention cost money. Choose each from business need and test result quality. Large volume alone does not prove Hadoop is necessary.

Common mistake: The five Vs describe challenges, not five software products. Do not define Big Data only as "very large data".

Back to units ↑

2. Data types and storage choices

1 mark: What is semi-structured data?

• Semi-structured data uses flexible keys or tags rather than a fixed table schema, as in JSON events.

4 marks: Compare structured, semi-structured and unstructured data.

Swipe the table to see all columns →

Point	Structured	Semi-structured	Unstructured
1. Organization	Fixed fields and types	Keys/tags; optional fields	No fixed table schema for the analysis
2. Example	Order table	JSON event	Photo, audio, free text
3. Processing	Validate columns; query rows	Parse keys; handle changes	Extract text/features as needed
4. Caution	Schema must be maintained	Still needs rules to interpret data	Not meaningless; can have metadata

6 marks: Should a growing retailer replace its RDBMS with Hadoop?

Live orders	RDBMS: updates and transactions
Historical logs	Distributed batch scan if justified
Decision	Combine tools when workloads differ

• Not automatically. Compare workload requirements before selecting a system.

1. Keep transactions safe: order entry and payments need constraints, updates and transaction guarantees. A relational database remains a strong fit.
2. Separate analytics: copy permitted historical and event data into an analytical store so heavy scans do not block live purchases.
3. Assess the bottleneck: measure data size, query latency, throughput and growth. Improve indexes, schemas or partitioning before assuming a distributed cluster is required.
4. Choose distributed work when justified: HDFS plus batch processing can spread large sequential scans across machines, but adds operations and coordination costs.
5. Handle variety deliberately: retain raw files with metadata, then create clean queryable datasets. "Schema-on-read" still requires a schema when interpreting data.
6. Conclude from cost and guarantees: a hybrid architecture is often reasonable. RDBMS products can also scale out; Hadoop is not inherently real-time, and HDFS is not a transactional database.

Back to units ↑

3. Analytics, lifecycle and decision-making

1 mark: What is Big Data Analytics?

• Big Data Analytics is the use of computational and statistical methods to find patterns, explain outcomes or support decisions from data that needs scalable handling.

4 marks: Compare descriptive, predictive and prescriptive analytics.

Swipe the table to see all columns →

Point	Descriptive	Predictive	Prescriptive
1. Question	What happened?	What may happen?	What should we do?
2. Input	Observed past events	Past data plus available current features	Predictions, costs and constraints
3. Example	Late-delivery count	Delay probability	Choose a route
4. Limit	Does not explain cause by itself	Prediction can be wrong	Action needs validation

6 marks: Explain the analytics lifecycle using delivery-delay prediction.

1. Define the decision and success measure

↓

2. Collect permitted order, route and GPS data

↓

3. Validate, clean and join records

↓

4. Explore and build time-aware features

↓

5. Train and test on separate time periods

↓

6. Serve results, act and monitor

1. Start with a measurable goal: reduce late deliveries without raising costs beyond a stated limit.
2. Collect only needed fields, with access controls and retention limits. Keep IDs and timestamps consistent across systems.
3. Check duplicate deliveries, impossible coordinates and missing values. Keep a record of transformations.
4. Construct features available at prediction time, such as distance and recent traffic. Do not use the actual arrival time as an input.
5. Test on later unseen deliveries and compare with a simple baseline. Data leakage can make an ineffective model look accurate.
6. Use predictions in a dispatch workflow and track errors, drift and actual delivery outcomes. A dashboard alone is not business value.

Back to units ↑

4. Typical analytical architecture and bottlenecks

1 mark: What is a data warehouse?

• A data warehouse is an integrated store organized for historical reporting and analysis, usually combining cleaned data from multiple operational sources.

4 marks: Explain a typical traditional analytical architecture.

1. Sources: operational systems record sales, finance and application events.
2. ETL: extract, clean and transform data before loading it into the analytical store.
3. Warehouse: combine historical data under consistent definitions; data marts serve a narrower subject.
4. Analysis: queries, OLAP and dashboards present results; freshness depends on the loading schedule.

6 marks: Analyze the limitations of a traditional warehouse-only design for Big Data.

Operational sources: sales, finance, applications

↓

ETL: extract, clean, transform and load

↓

Central warehouse → subject data marts

↓

Queries, OLAP analysis and dashboards

Nightly load	Old information for urgent decisions
Single bottleneck	Limited scan throughput
New data shape	Extra parsing and schema work

1. Ingestion delay: a nightly ETL job can leave a fraud report hours behind events. Match refresh intervals to the decision deadline.
2. Schema friction: new event fields and unstructured content may need preprocessing before fitting the warehouse schema.
3. Central bottlenecks: one storage or compute tier can limit concurrent scans. Scaling up has cost and hardware limits.
4. Data movement: repeatedly copying large datasets through ETL can consume bandwidth, space and time.
5. Operational contention: poorly separated analytics may compete with transaction processing. Isolate analytical workloads.
6. Balanced conclusion: these are limitations of a particular design, not a claim that every modern warehouse fails. Distributed or cloud warehouses, partitioning and incremental loading can address some limits; use a new architecture only where requirements justify it.

Common mistake: Do not write that all traditional databases cannot store Big Data. State the particular scale, latency or data-model constraint.

Back to units ↑

5. Modern analytical architecture and a streaming case

1 mark: Why is a new analytical architecture required?

• A new design is needed when the existing system cannot meet required scale, variety, freshness, resilience or cost.

4 marks: List four requirements of a modern Big Data architecture.

1. Scalable ingestion and storage: absorb growth and traffic bursts without losing traceability.
2. Appropriate processing latency: separate time-sensitive streams from heavy historical batch jobs.
3. Resilience and quality: recover from failures, validate records and handle duplicate or late events.
4. Governance and useful serving: restrict access, track lineage and expose results through a service, query engine or dashboard that supports the decision.

6 marks: Design a high-level architecture for real-time fraud detection.

Payment events + device signals

↓

Durable ingestion buffer

↓

Stream processing: validate, deduplicate, enrich

↓

Low-latency risk serving → alert or review

↓

Raw history → batch analysis → model evaluation

1. Ingest: assign event IDs and event timestamps. A durable buffer absorbs bursts and allows replay.
2. Validate: reject or quarantine malformed records; protect sensitive payment identifiers.
3. Enrich: combine the event with permitted recent-account features. Define handling of late or out-of-order data.
4. Score and act: calculate a risk score within the required latency and route high-risk cases to a controlled review or decision path.
5. Retain history: keep governed raw and cleaned datasets for audit and later batch analysis. HDFS-based MapReduce suits offline history, not a millisecond decision loop.
6. Measure and recover: monitor lag, failures, duplicate handling, false positives and detection rate. Replay and actions safe to repeat reduce duplicate effects, but exactly-once behavior must be designed end to end.

Back to units ↑

6. Challenges, distributed frameworks and trade-offs

1 mark: What is horizontal scalability?

• Horizontal scalability means increasing capacity by adding machines and distributing work or data across them.

4 marks: Explain four major challenges and one remedy for each.

1. Storage and processing: Partition data and process in parallel. Network transfer and skew can still limit speed.
2. Data quality: Validate, deduplicate and record provenance. Cleaning cannot recover facts never captured.
3. Security and privacy: Use least-privilege access, encryption and retention controls. Encryption alone does not make unauthorized use acceptable.
4. Scalability and failure: Add capacity, replicate data and retry failed work. Replication uses space and does not replace separate backups.

6 marks: Why are Big Data frameworks needed? Compare Hadoop MapReduce and Spark.

MapReduce	Large batch transformations
Spark	Dataflow and reusable cached work
Small simple workload	A database or simpler tool may be enough

• A distributed framework coordinates work that would otherwise require custom partitioning, scheduling, recovery and aggregation code.

1. Partitioning: divide a large dataset into manageable pieces. Correct partition keys help balance load.
2. Parallel execution: schedule tasks near data where possible. More nodes help only when work is parallelizable and coordination does not dominate.
3. Recovery: detect failed tasks and recompute work from recoverable input. This is fault tolerance, not immunity to every failure.
4. Hadoop MapReduce: uses map/shuffle/reduce stages and saved output between jobs; useful for large batch transformations, but repeated multi-job passes can add disk I/O and latency.
5. Spark: supports multi-stage dataflow and caching useful in iterative analysis. It can spill when data does not fit memory; "everything must fit in RAM" is wrong.
6. Selection: evaluate workload latency, iteration, cost, skills and data governance. Hadoop storage and Spark processing can coexist; a small dataset with simple queries may need neither.

Back to units ↑

Quick 1-mark revision

1 mark: What is veracity?

• Veracity is the trustworthiness and quality of data, including accuracy, consistency and completeness.

1 mark: What is value?

• Value is the useful, measurable benefit obtained from data, after considering collection, processing and decision costs.

1 mark: What is batch processing?

• Batch processing handles a collected set of records as a job, rather than processing every event as it arrives.

1 mark: What is stream processing?

• Stream processing handles events continuously as they arrive, with a defined latency and rules for state and event timing.

1 mark: Name two sources of Big Data.

• Application clickstream events and IoT sensor readings are two sources.

1 mark: What is fault tolerance?

• Fault tolerance is the ability to continue or recover useful operation after specified component failures.

1 mark: What is data provenance?

• Data provenance records where data came from and how it was transformed, helping audit and trust the result.

Technical references and scope

Original explanations and examples for revision. Technical behavior was checked against the references below. Hadoop 1 terms are labeled as legacy; Hadoop 2/3 use YARN. Configuration and installed versions can change defaults. Institutional names, personal information and supplied document headers are not reproduced.

• Apache Hadoop HDFS architecture
• Apache Hadoop MapReduce tutorial
• NIST Big Data definition
• Apache Spark dataflow and persistence guide
• IBM Big Data Analytics overview