Big Data Processing Platform Management

(Big data Engineering)

Course Description

Big data platform architecture and management. Various big data sources and types. Programming and management practices on distributed data system or cloud system. Mapreduce paradiam. Tools for working with structure and unstructure data on a large distributed file system. Batch and real-time streaming data processing. Working pipeline for big data processing from data source. Common open-sourced and cloud-based visual data analytics platform. Recommendation. Data. Lake. Open data. Case studies.

Class Overview

The cores of this class are three things : 1) big data platform management including Hadoop ecosystem 2) big data visualization and analytic tool 3) Some concepts of parallel and distributed computing.

The first priority will be on processing with big data with Hadoop ecosystems and tools where we spend about half of the time on it. Second will be distributed computing programming where we discuss along since the Hadoop FS relies on it . Third will be Elastic ecosystem where it includes processing pipeline and visualization which can be connected to the first part we learn.

We will learn by doing mostly. There will NOT be a lot of theories much. But we mention some and its design principle and some related concepts as we go on.

Students will learn to practice the skill set and get used to ecosystem so that in the future they can adapt themselves these similar things.

After finish ingthe course, students will get or earn more skill of problem-solving and ability to learn new things.

Note, there will be not many TA that can help you on setting up.

You will have to practice to operate your VM and Linux yourself. You are welcome to help your friends too.

Mostly, every week you will have hands-on lab or programming lab to finish up which may take around 1.5-2.0 hr depending on your skill. You may spend more time on exploring for solutions and studying/understanding them though.

The main principle is learn how to and understand why it works; as well as their architecture; not just copy and cut-paste, since we won't improve your knowledge.

Learning strategy:

- In-class meeting is for lecture and labwork/assignment discussions.

-Students are required to manipulate their VMs themselves.

-homework due every 2-3 weeks and quizzes using Quizziz every 3 weeks.

(programming assignment along with the VM setting up)