BDA Theory Syllabus
BDA Theory Syllabus
Embedded Files
UNIT-I: (Wiley & Big Java 4th)
Data structures in Java: Linked List, Stacks, Queues, Sets, Maps; (Chapter 15, 16)
Generics: Generic classes and Type parameters, Implementing Generic Types, Generic Methods,
(Chapter 17)
Wrapper Classes (Chapter 7)
Concept of Serialization (Chapter 19)
UNIT-II: Working with Big Data: Google File System, Hadoop Distributed File System (HDFS) Building blocks of Hadoop (Name node, Data node, Secondary Name node, Job Tracker, Task Tracker), Introducing and Configuring Hadoop cluster (Local, Pseudo-distributed mode, Fully Distributed mode), Configuring XML files.
UNIT-III: Writing Map Reduce Programs: A Weather Dataset, Understanding Hadoop API for Map Reduce Framework (Old and New), Basic programs of Hadoop Map Reduce: Driver code, Mapper code, Reducer code, Record Reader, Combiner, Practitioner
UNIT-IV: Stream Memory and Spark: Introduction to Streams Concepts– Stream Data Model and Architecture, Stream computing, Sampling Data in a Stream, Filtering Streams, Counting Distinct Elements in a Stream, Introduction to Spark Concept, Spark Architecture and components, Spark installation, Spark RDD (Resilient Distributed Dataset) – Spark RDD operations.
UNIT-V: Pig: Hadoop Programming Made Easier Admiring the Pig Architecture, going with the Pig Latin Application Flow, working through the ABCs of Pig Latin, Evaluating Local and Distributed Modes of Running Pig Scripts, Checking out the Pig Script Interfaces, Scripting with Pig Latin.
Applying Structure to Hadoop Data with Hive: Saying Hello to Hive, Seeing How the Hive is Put Together, Getting Started with Apache Hive, Examining the Hive Clients, Working with Hive Data Types, Creating and Managing Databases and Tables, Seeing How the Hive Data Manipulation Language Works, Querying and Analyzing data
Textbooks: -
1. Wiley & Big Java 4th Edition, Cay Horstmann, Wiley John Sons, INC
2. Hadoop: The Definitive Guide by Tom White, 3 rd. Edition, O’reilly
Reference Books: -
1. Hadoop in Action by Chuck Lam, MANNING Publ.
2. Hadoop for Dummies by Dirk deRoos, Paul C.Zikopoulos, Roman B.Melnyk,Bruce Brown, Rafael Coss
3. Hadoop in Practice by Alex Holmes, MANNING Publ.
4. Big Data Analytics by Dr. A.Krishna Mohan and Dr.E.Laxmi Lydia
5. Hadoop Map Reduce Cookbook, SrinathPerera, ThilinaGunarathne
Software Links: -
1. Hadoop:http://hadoop.apache.org/
2. Hive: https://cwiki.apache.org/confluence/display/Hive/Home
3. Piglatin: http://pig.apache.org/docs/r0.7.0/tutorial.html
UNIT-1
UNIT-1
1. Linked list Data structure implementation in Java using built-in methods
1. Linked list Data structure implementation in Java using built-in methods
Program: - Which accepts any data type {Object as the type: This allows you to mix and match different types in one list.}
Program: - Which accepts any data type {Object as the type: This allows you to mix and match different types in one list.}
import java.util.LinkedList;
import java.util.Scanner;
public class UniversalLinkedList
{
public static void main(String[] args)
{
// 'Object' is the parent of all classes in Java.
// Using <Object> allows this list to store Integers, Strings, Doubles, etc.
LinkedList<Object> anyList = new LinkedList<>();
Scanner sc = new Scanner(System.in);
int choice;
do {
System.out.println("\n--- Universal LinkedList (Accepts Any Type) ---");
System.out.println("1. Add Integer\n2. Add Double\n3. Add String\n4. Add Boolean\n5. Display & Size\n6. Exit");
System.out.print("Enter choice: ");
choice = sc.nextInt();
switch (choice) {
case 1 -> {
System.out.print("Enter an integer: ");
// sc.nextInt() returns a primitive 'int', Java boxes it into an 'Integer' object
anyList.add(sc.nextInt());
}
case 2 -> {
System.out.print("Enter a double (e.g. 10.5): ");
// sc.nextDouble() boxes into a 'Double' object
anyList.add(sc.nextDouble());
}
case 3 -> {
System.out.print("Enter a string: ");
anyList.add(sc.next()); // Strings are already objects
}
case 4 -> {
System.out.print("Enter a boolean (true/false): ");
anyList.add(sc.nextBoolean()); // Boxes into a 'Boolean' object
}
case 5 -> {
System.out.println("List Contents: " + anyList);
System.out.println("Total Items: " + anyList.size());
}
}
} while (choice!= 6);
sc.close();
}
}
2. Stack Operations implementation in Java using built-in methods
2. Stack Operations implementation in Java using built-in methods
import java.util.Stack;
import java.util.Stack;
import java.util.Scanner;
import java.util.Scanner;
public class UniversalStack {
public class UniversalStack {
public static void main(String[] args) {
public static void main(String[] args) {
// <Object> allows the stack to hold any type of data
// <Object> allows the stack to hold any type of data
Stack<Object> stack = new Stack<>();
Stack<Object> stack = new Stack<>();
Scanner sc = new Scanner(System.in);
Scanner sc = new Scanner(System.in);
String choice;
String choice;
do {
do {
System.out.println("\n--- Universal Stack (Any Input) ---");
System.out.println("\n--- Universal Stack (Any Input) ---");
System.out.println("Type 'push' to add, 'pop' to remove, 'exit' to quit:");
System.out.println("Type 'push' to add, 'pop' to remove, 'exit' to quit:");
choice = sc.next();
choice = sc.next();
if (choice.equalsIgnoreCase("push")) {
if (choice.equalsIgnoreCase("push")) {
System.out.print("Enter anything: ");
System.out.print("Enter anything: ");
// sc.next() reads input as a String, which is a type of Object
// sc.next() reads input as a String, which is a type of Object
Object input = sc.next();
Object input = sc.next();
stack.push(input); // Built-in push method
stack.push(input); // Built-in push method
System.out.println("Pushed onto stack.");
System.out.println("Pushed onto stack.");
} else if (choice.equalsIgnoreCase("pop")) {
} else if (choice.equalsIgnoreCase("pop")) {
if (!stack.isEmpty()) {
if (!stack.isEmpty()) {
// pop() removes the last item entered
// pop() removes the last item entered
System.out.println("Removed: " + stack.pop());
System.out.println("Removed: " + stack.pop());
} else {
} else {
System.out.println("Stack is empty!");
System.out.println("Stack is empty!");
}
}
}
}
System.out.println("Current Stack: " + stack);
System.out.println("Current Stack: " + stack);
} while (!choice.equalsIgnoreCase("exit"));
} while (!choice.equalsIgnoreCase("exit"));
}
}
}
}
3. Queue Implementation in Java using built-in methods
3. Queue Implementation in Java using built-in methods
import java.util.LinkedList;
import java.util.LinkedList;
import java.util.Queue;
import java.util.Queue;
import java.util.Scanner;
import java.util.Scanner;
public class UniversalQueue {
public class UniversalQueue {
public static void main(String[] args) {
public static void main(String[] args) {
// Queue is an interface, LinkedList provides the implementation
// Queue is an interface, LinkedList provides the implementation
Queue<Object> queue = new LinkedList<>();
Queue<Object> queue = new LinkedList<>();
Scanner sc = new Scanner(System.in);
Scanner sc = new Scanner(System.in);
String command;
String command;
while (true) {
while (true) {
System.out.println("\nQueue: " + queue);
System.out.println("\nQueue: " + queue);
System.out.print("Enter data to add (or type 'remove' or 'exit'): ");
System.out.print("Enter data to add (or type 'remove' or 'exit'): ");
command = sc.next();
command = sc.next();
if (command.equalsIgnoreCase("exit")) break;
if (command.equalsIgnoreCase("exit")) break;
if (command.equalsIgnoreCase("remove")) {
if (command.equalsIgnoreCase("remove")) {
if (!queue.isEmpty()) {
if (!queue.isEmpty()) {
// poll() removes the front item
// poll() removes the front item
System.out.println("Removed from front: " + queue.poll());
System.out.println("Removed from front: " + queue.poll());
} else {
} else {
System.out.println("Queue is empty.");
System.out.println("Queue is empty.");
}
}
} else {
} else {
// Add the user input to the back of the queue
// Add the user input to the back of the queue
queue.add(command);
queue.add(command);
}
}
}
}
}
}
}
}
4. Set Operations Implementation in Java using built-in methods
4. Set Operations Implementation in Java using built-in methods
import java.util.HashSet;
import java.util.HashSet;
import java.util.Set;
import java.util.Set;
import java.util.Scanner;
import java.util.Scanner;
public class UniversalSet {
public class UniversalSet {
public static void main(String[] args) {
public static void main(String[] args) {
// Set prevents duplicate entries
// Set prevents duplicate entries
Set<Object> set = new HashSet<>();
Set<Object> set = new HashSet<>();
Scanner sc = new Scanner(System.in);
Scanner sc = new Scanner(System.in);
while (true) {
while (true) {
System.out.print("Enter a value for the Set (or 'exit'): ");
System.out.print("Enter a value for the Set (or 'exit'): ");
String input = sc.next();
String input = sc.next();
if (input.equalsIgnoreCase("exit")) break;
if (input.equalsIgnoreCase("exit")) break;
// add() returns false if the item already exists in the set
// add() returns false if the item already exists in the set
if (set.add(input)) {
if (set.add(input)) {
System.out.println("Added: " + input);
System.out.println("Added: " + input);
} else {
} else {
System.out.println("Duplicate detected! '" + input + "' not added.");
System.out.println("Duplicate detected! '" + input + "' not added.");
}
}
System.out.println("Current Set (Unordered): " + set);
System.out.println("Current Set (Unordered): " + set);
}
}
}
}
}
}
5. Map Operations Implementation in Java using built-in methods
5. Map Operations Implementation in Java using built-in methods
import java.util.HashMap;
import java.util.HashMap;
import java.util.Map;
import java.util.Map;
import java.util.Scanner;
import java.util.Scanner;
public class UniversalMap {
public class UniversalMap {
public static void main(String[] args) {
public static void main(String[] args) {
// Both Key and Value are stored as Objects
// Both Key and Value are stored as Objects
Map<Object, Object> map = new HashMap<>();
Map<Object, Object> map = new HashMap<>();
Scanner sc = new Scanner(System.in);
Scanner sc = new Scanner(System.in);
while (true) {
while (true) {
System.out.println("\n1. Add/Update Pair\n2. Search by Key\n3. Exit");
System.out.println("\n1. Add/Update Pair\n2. Search by Key\n3. Exit");
int choice = sc.nextInt();
int choice = sc.nextInt();
if (choice == 3) break;
if (choice == 3) break;
if (choice == 1) {
if (choice == 1) {
System.out.print("Enter Key: ");
System.out.print("Enter Key: ");
Object key = sc.next();
Object key = sc.next();
System.out.print("Enter Value: ");
System.out.print("Enter Value: ");
Object value = sc.next();
Object value = sc.next();
// put() maps the key to the value
// put() maps the key to the value
map.put(key, value);
map.put(key, value);
System.out.println("Stored successfully.");
System.out.println("Stored successfully.");
} else if (choice == 2) {
} else if (choice == 2) {
System.out.print("Enter key to search: ");
System.out.print("Enter key to search: ");
Object searchKey = sc.next();
Object searchKey = sc.next();
// get() retrieves the value associated with the key
// get() retrieves the value associated with the key
System.out.println("Value Found: " + map.getOrDefault(searchKey, "Not Found"));
System.out.println("Value Found: " + map.getOrDefault(searchKey, "Not Found"));
}
}
System.out.println("All Entries: " + map);
System.out.println("All Entries: " + map);
}
}
}
}
}
}
BDA LAB Syllabus
BDA LAB Syllabus
Software Requirements: -
Hadoop: https://hadoop.apache.org/release/2.7.6.html
Java :https://www.oracle.com/java/technologies/javase/javase8u211-later-archive- downloads.html
Eclipse : https://www.eclipse.org/downloads/
Experiment 1: Week 1, 2:
1. Implement the following Data structures in Java a) Linked Lists b) Stacks c) Queues d) Set e) Map
Experiment 2: Week 3:
2. (i)Perform setting up and Installing Hadoop in its three operating modes: Standalone, Pseudo distributed, Fully distributed (ii)Use web-based tools to monitor your Hadoop setup.
Experiment 3: Week 4:
3. Implement the following file management tasks in Hadoop: • Adding files and directories • Retrieving files • Deleting files Hint: A typical Hadoop workflow creates data files (such as log files) elsewhere and copies them into HDFS using one of the above command line utilities.
Experiment 4: Week 5:
4. Run a basic Word Count Map Reduce program to understand Map Reduce Paradigm.
Experiment 5: Week 6: 5. Write a map reduce program that mines weather data. Weather sensors collecting data every hour at many locations across the globe gather a large volume of log data, which is a good candidate for analysis with Map Reduce, since it is semi structured and record oriented.
Experiment 6: Week 7:
6. Use Map Reduce to find the shortest path between two people in a social graph. Hint: Use an adjacency list to model a graph, and for each node store the distance from the original node, as well as a back pointer to the original node. Use the mappers to propagate the distance to the original node, and the reducer to restore the state of the graph. Iterate until the target node has been reached.
Experiment 7: Week 8:
7. Implement Friends-of-friends algorithm in MapReduce. Hint: Two MapReduce jobs are required to calculate the FoFs for each user in a social network .The first job calculates the common friends for each user, and the second job sorts the common friends by the number of connections to your friends.
Experiment 8: Week 9:
8. Implement an iterative PageRank graph algorithm in MapReduce. Hint: PageRank can be implemented by iterating a MapReduce job until the graph has converged. The mappers are responsible for propagating node PageRank values to their adjacent nodes, and the reducers are responsible for calculating new PageRank values for each node, and for re-creating the original graph with the updated PageRank values.
Experiment 9: Week 10:
9. Perform an efficient semi-join in MapReduce. Hint: Perform a semi-join by having the mappers load a Bloom filter from the Distributed Cache and then filter results from the actual MapReduce data source by performing membership queries against the Bloom filter to determine which data source records should be emitted to the reducers.
Experiment 10: Week 11:
10. Install and Run Pig then write Pig Latin scripts to sort, group, join, project, and filter your data.
Experiment 12: Week 12:
11. Install and Run Hive then use Hive to create, alter, and drop databases, tables, views, functions, and indexes
Week 1 Solution
Week 1 Solution
1. Identify What is Installed
1. Identify What is Installed
First, list the installed Java packages to know exactly what to remove.
dpkg --list | grep -i jdk
dpkg --list | grep -i jdk
Note the package names (e.g., openjdk-11-jdk, oracle-java8-installer).
2. Remove the Java Packages
2. Remove the Java Packages
Run the command below corresponding to your installation type.
Option A: Standard Removal (OpenJDK)
This removes all OpenJDK versions (common for most Ubuntu users).
sudo apt-get purge "openjdk-*" "icedtea-*" -y
sudo apt-get purge "openjdk-*" "icedtea-*" -y
sudo apt-get autoremove -y
sudo apt-get autoremove -y
Option B: Oracle Java Removal
If you installed Oracle Java specifically:
sudo apt-get purge "oracle-java*-installer" -y
sudo apt-get purge "oracle-java*-installer" -y
sudo apt-get autoremove -y
sudo apt-get autoremove -y
3. Clean Up System Links
3. Clean Up System Links
To ensure no "dead" links remain in your system alternatives:
sudo update-alternatives --remove-all java
sudo update-alternatives --remove-all java
sudo update-alternatives --remove-all javac
sudo update-alternatives --remove-all javac
(If you see an error saying "no alternatives for java", that is good—it means the system is already clean.)
4. Remove Environment Variables
4. Remove Environment Variables
If you previously set JAVA_HOME manually, you must remove it to prevent errors with the new installation.
Open your bash config:
nano ~/.bashrc
nano ~/.bashrc
Find and delete any lines that look like:
export JAVA_HOME=...
export JAVA_HOME=...
export PATH=$JAVA_HOME/bin:$PATH
export PATH=$JAVA_HOME/bin:$PATH
Save and Exit: Press Ctrl+O, Enter, then Ctrl+X.
Refresh your session:
source ~/.bashrc
source ~/.bashrc
5. Verify Uninstallation
5. Verify Uninstallation
Run this command. It should now return "Command 'java' not found".
java -version
java -version
_________________________________________________________________________________________________________________
For Hadoop 2.10, the most suitable version is Java 8 (OpenJDK 8).
While newer versions of Hadoop (3.3+) support Java 11, the Hadoop 2.x series is most stable with Java 8. Java 7 is also supported but is end-of-life and not recommended.
Here is the step-by-step guide to installing and configuring it on Ubuntu.
1. Update Repository and Install OpenJDK 8
1. Update Repository and Install OpenJDK 8
Open your terminal (Ctrl+Alt+T) and run the following commands:
sudo apt update
sudo apt install openjdk-8-jdk -y
2. Verify the Installation
2. Verify the Installation
Once the installation is complete, verify the version to ensure it is active.
java -version
You should see output similar to: openjdk version "1.8.0_xxx"
3. Find the Java Installation Path
3. Find the Java Installation Path
You will need the exact path for your Hadoop configuration. Run this command:
update-alternatives --list java
Copy the path displayed (excluding /bin/java at the end).
Standard Ubuntu path: /usr/lib/jvm/java-8-openjdk-amd64
4. Configure Environment Variables
4. Configure Environment Variables
You need to set JAVA_HOME so Hadoop can find Java.1
A. Edit your bash configuration:
nano ~/.bashrc
B. Add these lines to the end of the file:
(Replace the path below if yours was different in Step 3)
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export PATH=$PATH:$JAVA_HOME/bin
C. Save and Exit:
Press CTRL + O, then Enter (to save).
Press CTRL + X (to exit).
D. Apply the changes:
source ~/.bashrc
5. (Crucial) Configure hadoop-env.sh
5. (Crucial) Configure hadoop-env.sh
Even with the system variable set, Hadoop 2.10 often requires JAVA_HOME to be explicitly defined in its own configuration file.
Navigate to your Hadoop configuration directory (usually $HADOOP_HOME/etc/hadoop/).
Edit hadoop-env.sh:
Bash
nano hadoop-env.sh
Find the line export JAVA_HOME=${JAVA_HOME}.
Change it to the specific path:
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
Verification
Verification
To confirm Java is ready for Hadoop, checking the JAVA_HOME variable should return your path:
echo $JAVA_HOME
1. Linked list Data structure implementation in Java using built-in methods
1. Linked list Data structure implementation in Java using built-in methods
Program: - Which accepts any data type {Object as the type: This allows you to mix and match different types in one list.}
Program: - Which accepts any data type {Object as the type: This allows you to mix and match different types in one list.}
import java.util.LinkedList;
import java.util.Scanner;
public class UniversalLinkedList
{
public static void main(String[] args)
{
// 'Object' is the parent of all classes in Java.
// Using <Object> allows this list to store Integers, Strings, Doubles, etc.
LinkedList<Object> anyList = new LinkedList<>();
Scanner sc = new Scanner(System.in);
int choice;
do {
System.out.println("\n--- Universal LinkedList (Accepts Any Type) ---");
System.out.println("1. Add Integer\n2. Add Double\n3. Add String\n4. Add Boolean\n5. Display & Size\n6. Exit");
System.out.print("Enter choice: ");
choice = sc.nextInt();
switch (choice) {
case 1 -> {
System.out.print("Enter an integer: ");
// sc.nextInt() returns a primitive 'int', Java boxes it into an 'Integer' object
anyList.add(sc.nextInt());
}
case 2 -> {
System.out.print("Enter a double (e.g. 10.5): ");
// sc.nextDouble() boxes into a 'Double' object
anyList.add(sc.nextDouble());
}
case 3 -> {
System.out.print("Enter a string: ");
anyList.add(sc.next()); // Strings are already objects
}
case 4 -> {
System.out.print("Enter a boolean (true/false): ");
anyList.add(sc.nextBoolean()); // Boxes into a 'Boolean' object
}
case 5 -> {
System.out.println("List Contents: " + anyList);
System.out.println("Total Items: " + anyList.size());
}
}
} while (choice!= 6);
sc.close();
}
}
2. Stack Operations implementation in Java using built-in methods
2. Stack Operations implementation in Java using built-in methods
import java.util.Stack;
import java.util.Stack;
import java.util.Scanner;
import java.util.Scanner;
public class UniversalStack {
public class UniversalStack {
public static void main(String[] args) {
public static void main(String[] args) {
// <Object> allows the stack to hold any type of data
// <Object> allows the stack to hold any type of data
Stack<Object> stack = new Stack<>();
Stack<Object> stack = new Stack<>();
Scanner sc = new Scanner(System.in);
Scanner sc = new Scanner(System.in);
String choice;
String choice;
do {
do {
System.out.println("\n--- Universal Stack (Any Input) ---");
System.out.println("\n--- Universal Stack (Any Input) ---");
System.out.println("Type 'push' to add, 'pop' to remove, 'exit' to quit:");
System.out.println("Type 'push' to add, 'pop' to remove, 'exit' to quit:");
choice = sc.next();
choice = sc.next();
if (choice.equalsIgnoreCase("push")) {
if (choice.equalsIgnoreCase("push")) {
System.out.print("Enter anything: ");
System.out.print("Enter anything: ");
// sc.next() reads input as a String, which is a type of Object
// sc.next() reads input as a String, which is a type of Object
Object input = sc.next();
Object input = sc.next();
stack.push(input); // Built-in push method
stack.push(input); // Built-in push method
System.out.println("Pushed onto stack.");
System.out.println("Pushed onto stack.");
} else if (choice.equalsIgnoreCase("pop")) {
} else if (choice.equalsIgnoreCase("pop")) {
if (!stack.isEmpty()) {
if (!stack.isEmpty()) {
// pop() removes the last item entered
// pop() removes the last item entered
System.out.println("Removed: " + stack.pop());
System.out.println("Removed: " + stack.pop());
} else {
} else {
System.out.println("Stack is empty!");
System.out.println("Stack is empty!");
}
}
}
}
System.out.println("Current Stack: " + stack);
System.out.println("Current Stack: " + stack);
} while (!choice.equalsIgnoreCase("exit"));
} while (!choice.equalsIgnoreCase("exit"));
}
}
}
}
3. Queue Implementation in Java using built-in methods
3. Queue Implementation in Java using built-in methods
import java.util.LinkedList;
import java.util.LinkedList;
import java.util.Queue;
import java.util.Queue;
import java.util.Scanner;
import java.util.Scanner;
public class UniversalQueue {
public class UniversalQueue {
public static void main(String[] args) {
public static void main(String[] args) {
// Queue is an interface, LinkedList provides the implementation
// Queue is an interface, LinkedList provides the implementation
Queue<Object> queue = new LinkedList<>();
Queue<Object> queue = new LinkedList<>();
Scanner sc = new Scanner(System.in);
Scanner sc = new Scanner(System.in);
String command;
String command;
while (true) {
while (true) {
System.out.println("\nQueue: " + queue);
System.out.println("\nQueue: " + queue);
System.out.print("Enter data to add (or type 'remove' or 'exit'): ");
System.out.print("Enter data to add (or type 'remove' or 'exit'): ");
command = sc.next();
command = sc.next();
if (command.equalsIgnoreCase("exit")) break;
if (command.equalsIgnoreCase("exit")) break;
if (command.equalsIgnoreCase("remove")) {
if (command.equalsIgnoreCase("remove")) {
if (!queue.isEmpty()) {
if (!queue.isEmpty()) {
// poll() removes the front item
// poll() removes the front item
System.out.println("Removed from front: " + queue.poll());
System.out.println("Removed from front: " + queue.poll());
} else {
} else {
System.out.println("Queue is empty.");
System.out.println("Queue is empty.");
}
}
} else {
} else {
// Add the user input to the back of the queue
// Add the user input to the back of the queue
queue.add(command);
queue.add(command);
}
}
}
}
}
}
}
}
4. Set Operations Implementation in Java using built-in methods
4. Set Operations Implementation in Java using built-in methods
import java.util.HashSet;
import java.util.HashSet;
import java.util.Set;
import java.util.Set;
import java.util.Scanner;
import java.util.Scanner;
public class UniversalSet {
public class UniversalSet {
public static void main(String[] args) {
public static void main(String[] args) {
// Set prevents duplicate entries
// Set prevents duplicate entries
Set<Object> set = new HashSet<>();
Set<Object> set = new HashSet<>();
Scanner sc = new Scanner(System.in);
Scanner sc = new Scanner(System.in);
while (true) {
while (true) {
System.out.print("Enter a value for the Set (or 'exit'): ");
System.out.print("Enter a value for the Set (or 'exit'): ");
String input = sc.next();
String input = sc.next();
if (input.equalsIgnoreCase("exit")) break;
if (input.equalsIgnoreCase("exit")) break;
// add() returns false if the item already exists in the set
// add() returns false if the item already exists in the set
if (set.add(input)) {
if (set.add(input)) {
System.out.println("Added: " + input);
System.out.println("Added: " + input);
} else {
} else {
System.out.println("Duplicate detected! '" + input + "' not added.");
System.out.println("Duplicate detected! '" + input + "' not added.");
}
}
System.out.println("Current Set (Unordered): " + set);
System.out.println("Current Set (Unordered): " + set);
}
}
}
}
}
}
5. Map Operations Implementation in Java using built-in methods
5. Map Operations Implementation in Java using built-in methods
import java.util.HashMap;
import java.util.HashMap;
import java.util.Map;
import java.util.Map;
import java.util.Scanner;
import java.util.Scanner;
public class UniversalMap {
public class UniversalMap {
public static void main(String[] args) {
public static void main(String[] args) {
// Both Key and Value are stored as Objects
// Both Key and Value are stored as Objects
Map<Object, Object> map = new HashMap<>();
Map<Object, Object> map = new HashMap<>();
Scanner sc = new Scanner(System.in);
Scanner sc = new Scanner(System.in);
while (true) {
while (true) {
System.out.println("\n1. Add/Update Pair\n2. Search by Key\n3. Exit");
System.out.println("\n1. Add/Update Pair\n2. Search by Key\n3. Exit");
int choice = sc.nextInt();
int choice = sc.nextInt();
if (choice == 3) break;
if (choice == 3) break;
if (choice == 1) {
if (choice == 1) {
System.out.print("Enter Key: ");
System.out.print("Enter Key: ");
Object key = sc.next();
Object key = sc.next();
System.out.print("Enter Value: ");
System.out.print("Enter Value: ");
Object value = sc.next();
Object value = sc.next();
// put() maps the key to the value
// put() maps the key to the value
map.put(key, value);
map.put(key, value);
System.out.println("Stored successfully.");
System.out.println("Stored successfully.");
} else if (choice == 2) {
} else if (choice == 2) {
System.out.print("Enter key to search: ");
System.out.print("Enter key to search: ");
Object searchKey = sc.next();
Object searchKey = sc.next();
// get() retrieves the value associated with the key
// get() retrieves the value associated with the key
System.out.println("Value Found: " + map.getOrDefault(searchKey, "Not Found"));
System.out.println("Value Found: " + map.getOrDefault(searchKey, "Not Found"));
}
}
System.out.println("All Entries: " + map);
System.out.println("All Entries: " + map);
}
}
}
}
}
}
Week 2 Solution
Week 2 Solution
A Comprehensive laboratory guide for installing and configuring Hadoop 2.10 on Ubuntu with JDK 8.
Prerequisites & Setup
Mode 1: Standalone Mode (Default, for debugging)
Mode 2: Pseudo-Distributed Mode (Simulated cluster, standard for learning)
Mode 3: Fully Distributed Mode (Real cluster setup)
Web Monitoring
Part 0: Prerequisites
Part 0: Prerequisites
Before starting the modes, ensure the base environment is ready.
1. Verify Java 8
Ensure you have the correct version (as installed in the previous step).
java -version
java -version
# Output should imply "1.8.0_xxx"
# Output should imply "1.8.0_xxx"
2. Install SSH (Secure Shell)
Hadoop requires SSH to manage its nodes, even on a single machine.
sudo apt-get install ssh -y
sudo apt-get install ssh -y
sudo apt-get install pdsh -y
sudo apt-get install pdsh -y
3. Configure Password-less SSH
Hadoop must log in to "localhost" without a password.
# Generate a new key (press Enter for all prompts)
# Generate a new key (press Enter for all prompts)
ssh-keygen -t rsa -P ""
ssh-keygen -t rsa -P ""
# Add the key to authorized keys
# Add the key to authorized keys
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
# Verify (It should NOT ask for a password now)
# Verify (It should NOT ask for a password now)
ssh localhost
ssh localhost
exit
exit
4. Download and Extract Hadoop 2.10
# Download
# Download
wget https://archive.apache.org/dist/hadoop/common/hadoop-2.10.1/hadoop-2.10.1.tar.gz
wget https://archive.apache.org/dist/hadoop/common/hadoop-2.10.1/hadoop-2.10.1.tar.gz
# Extract
# Extract
tar -xzvf hadoop-2.10.1.tar.gz
tar -xzvf hadoop-2.10.1.tar.gz
# Move to a cleaner path (Optional but recommended)
# Move to a cleaner path (Optional but recommended)
sudo mv hadoop-2.10.1 /usr/local/hadoop
sudo mv hadoop-2.10.1 /usr/local/hadoop
5. Configure Environment Variables
Edit your bashrc file:
nano ~/.bashrc
nano ~/.bashrc
Add the following to the end:
# Hadoop Variables
# Hadoop Variables
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
Save and exit (Ctrl+O, Enter, Ctrl+X), then run: source ~/.bashrc
6. Set JAVA_HOME in Hadoop Config
Edit hadoop-env.sh to hardcode the Java path (Hadoop often struggles to find it otherwise).
nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh
nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh
Find export JAVA_HOME=${JAVA_HOME} and change it to:
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
Part 1: Operating Mode I — Standalone (Local) Mode
Part 1: Operating Mode I — Standalone (Local) Mode
Concept: This is the default mode. Hadoop runs as a single Java process. It is used for debugging and does not use HDFS; it uses the local Linux file system.
Configuration:
No XML configuration is required (default files are empty).
Laboratory Task: Run a "Grep" example to verify Hadoop can calculate.
Create input data:
mkdir input
mkdir input
cp $HADOOP_HOME/etc/hadoop/*.xml input
cp $HADOOP_HOME/etc/hadoop/*.xml input
Run the Hadoop MapReduce job:
This searches for the string dfs in the input files.
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.10.1.jar grep input output 'dfs[a-z.]+'
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.10.1.jar grep input output 'dfs[a-z.]+'
View Output:
cat output/*
cat output/*
(If you see a list of text matches, Standalone mode is working.)
Part 2: Operating Mode II — Pseudo-Distributed Mode
Part 2: Operating Mode II — Pseudo-Distributed Mode
Concept: This simulates a full cluster on a single machine. All daemons (NameNode, DataNode, ResourceManager, NodeManager) run as separate Java processes on localhost. This is the standard mode for students.
Configuration Steps:
Navigate to the config folder: cd $HADOOP_HOME/etc/hadoop/
1. core-site.xml (Defines the NameNode address)
nano core-site.xml
nano core-site.xml
Add inside <configuration> tags:
XML
<configuration>
<configuration>
<property>
<property>
<name>fs.defaultFS</name>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
<value>hdfs://localhost:9000</value>
</property>
</property>
</configuration>
</configuration>
2. hdfs-site.xml (Defines replication factor)
Since we have only one node, replication must be 1.
nano hdfs-site.xml
nano hdfs-site.xml
Add inside <configuration> tags:
XML
<configuration>
<configuration>
<property>
<property>
<name>dfs.replication</name>
<name>dfs.replication</name>
<value>1</value>
<value>1</value>
</property>
</property>
</configuration>
</configuration>
3. mapred-site.xml (Defines MapReduce framework)
First, copy the template:
cp mapred-site.xml.template mapred-site.xml
cp mapred-site.xml.template mapred-site.xml
nano mapred-site.xml
nano mapred-site.xml
Add inside <configuration> tags:
XML
<configuration>
<configuration>
<property>
<property>
<name>mapreduce.framework.name</name>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<value>yarn</value>
</property>
</property>
</configuration>
</configuration>
4. yarn-site.xml (Defines ResourceManager)
nano yarn-site.xml
nano yarn-site.xml
Add inside <configuration> tags:
XML
<configuration>
<configuration>
<property>
<property>
<name>yarn.nodemanager.aux-services</name>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<value>mapreduce_shuffle</value>
</property>
</property>
</configuration>
</configuration>
5. Execution
Format the NameNode (Do this only once!):
hdfs namenode -format
hdfs namenode -format
Start HDFS (NameNode & DataNode):
start-dfs.sh
start-dfs.sh
Start YARN (ResourceManager & NodeManager):
start-yarn.sh
start-yarn.sh
Verify Daemons:
Run jps. You should see 5 key processes: NameNode, DataNode, SecondaryNameNode, ResourceManager, NodeManager.
Part 3: Operating Mode III — Fully Distributed Mode
Part 3: Operating Mode III — Fully Distributed Mode
Concept: This is a real-world cluster involving multiple machines (Nodes). One machine is the Master (NameNode), and others are Slaves (DataNodes).
Laboratory Note: To perform this in a lab, you generally need 2+ Virtual Machines (VMs) networked together (Bridged Adapter). Let's assume you have a Master (IP: 192.168.1.10) and a Slave (IP: 192.168.1.11).
Configuration Changes (Different from Pseudo-mode):
1. Networking (On all nodes)
Edit /etc/hosts on ALL machines so they can ping each other by name.
Plaintext
192.168.1.10 master
192.168.1.10 master
192.168.1.11 slave1
192.168.1.11 slave1
2. SSH Copy (Master to Slaves)
The Master must be able to SSH into all Slaves without a password.
ssh-copy-id slave1
ssh-copy-id slave1
3. core-site.xml (On Master)
Change localhost to the Master's hostname.
XML
<property>
<property>
<name>fs.defaultFS</name>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
<value>hdfs://master:9000</value>
</property>
</property>
4. Slaves file (On Master)
Edit $HADOOP_HOME/etc/hadoop/slaves.
Remove localhost and add your slave hostnames:
Plaintext
slave1
slave1
5. Execution
Format NameNode on Master only.
Run start-dfs.sh and start-yarn.sh on Master only.
The script will automatically use SSH to start DataNodes on the slave machines.
(ii) Web-Based Monitoring Tools
(ii) Web-Based Monitoring Tools
Once you have started Hadoop (in Pseudo or Fully Distributed mode), you can monitor the system using your browser.
1. HDFS NameNode UI
URL: http://localhost:50070
Purpose: View the file system, check how much storage is available, and see if your DataNodes are "Live" or "Dead".
Note for Lab: Go to Utilities -> Browse the file system to see your HDFS files graphically.
2. YARN ResourceManager UI
URL: http://localhost:8088
Purpose: Monitor running jobs (applications). If you run a MapReduce job, it will appear here. You can see its progress (0% to 100%) and logs.
3. JobHistory Server (Optional)
URL: http://localhost:19888
Purpose: View details of finished jobs. (Requires starting mr-jobhistory-daemon.sh start historyserver).
Next Step: Would you like a sample WordCount Java program code to run and visualize on this new cluster setup?
Page updated
Google Sites
Report abuse