Distributed Systems & Data Engineering:

• Hadoop Ecosystem: HDFS, MapReduce, YARN

• Design and implementation of high-performance, fault-tolerant data pipelines

Scalable Analytics with Apache Spark:

• Core: RDDs, DataFrames, Apache Spark

• Programming in Scala and Python

• Deployment and optimization using Databricks for in‑memory iterative processing
  
  

Machine Learning Engineering:

• Algorithms: K‑means clustering, gradient descent, perceptron (binary and multiclass)

• Implementation of scalable machine learning models on Big Data platforms
  
  

Cloud Engineering & Cluster Management:

• Google Cloud Platform: Cluster creation, configuration, and resource manipulation

• Integrating cloud infrastructure with distributed processing frameworks

Perceptron and Gradient Descent in Scala & Spark

• Perceptron Implementation:

  • Developed a perceptron from scratch in Scala, implementing the step function, net input computation, and iterative weight updates with a bias adjustment.

  • Demonstrated binary classification on a synthetic dataset and extended the approach by processing a real-world cardio training dataset via Spark DataFrames.

• Gradient Descent for Linear Regression:

  • Implemented both batch and stochastic gradient descent on synthetic datasets and normalized data.

  • Built a suite of vector operations in Scala (including dot product, vector subtraction, and scalar multiplication) to support efficient gradient computations.

  • Validated and measured model convergence through custom accuracy functions.

Graph Analytics with Spark RDD

• Iterative Connected Components:

  • Created a Spark RDD–based algorithm that simulates iterative MapReduce cycles for connected components.

  • Developed custom mapper and reducer functions that propagate lexicographic minimum labels across nodes, iterating until convergence.

  • Extended the project to demonstrate dynamic graph operations such as counting arcs, identifying sinks, and detecting triangles, showing proficiency in grouping, union operations, and RDD transformations.

• Additional RDD Tests:

  • Conducted multiple lab exercises on fundamental RDD operations: word count, partitioning, flatMap, union, and complex joins.

  • Experimented with grouping and aggregating key-value pairs to optimize shuffles and understand the efficiency trade-offs in distributed processing.

K-means Clustering in Spark RDD (Scala)

• Clustering Algorithm Implementation:

  • Designed a first implementation of k‑means clustering using Spark RDDs, emphasizing the initialization, assignment, and update steps.

  • Analyzed the performance of the basic algorithm and introduced vector operations to support distance computation and centroid updates.

  • Worked with the Iris dataset by reading data via RDDs, inferring the number of clusters from distinct labels, and evaluating clustering efficiency.

Dynamic Data Analytics with Spark DataFrames & SQL

• Flat Data and Nested Data Processing:

  • Utilized Spark DataFrame operations to perform filtering, grouping, and aggregation on flat CSV data (e.g., Books, Users, and Ratings).

  • Enhanced skills in Spark SQL by registering DataFrames as temporary views and executing complex queries.

  • Managed and flattened nested JSON structures (using functions like explode, from_unixtime, and month) to extract and pivot information from social network data.

Cloud Cluster Management on Google Cloud

• Google Cloud Integration:

  • Designed and administered Google Cloud clusters to run Spark applications, focusing on resource configuration and deployment in a cloud environment.

  • Demonstrated practical experience in integrating cloud-based processing with scalable analytics via Databricks and Spark’s distributed capabilities.