Lorenzo Panebianco

Github Repositories:

In these public repositories, I share final projects from several Coursera courses in Data Science and Machine Learning that I’ve completed.

Machine Learning with Apache Spark
Final project: You are a data engineer at an aeronautics consulting company. Your company prides itself in being able to efficiently design airfoils for use in planes and sports cars. Data scientists in your office need to work with different algorithms and data in different formats. While they are good at Machine Learning, they count on you to be able to do ETL jobs and build ML pipelines. In this project you will use the modified version of the NASA Airfoil Self Noise dataset. You will clean this dataset, by dropping the duplicate rows, and removing the rows with null values. You will create an ML pipe line to create a model that will predict the SoundLevel based on all the other columns. You will evaluate the model and towards the end you will persist the model.
IBM Data Science Capstone Project
Final project: In this capstone, we predict if the Falcon 9 first stage will land successfully. SpaceX advertises Falcon 9 rocket launches on its website, with a cost of 62 million dollars; other providers cost upward of 165 million dollars each, much of the savings is because SpaceX can reuse the first stage. Therefore if we can determine if the first stage will land, we can determine the cost of a launch. This information can be used if an alternate company wants to bid against SpaceX for a rocket launch.

Sogei projects:

The following projects, of which I am the sole contributor, are being developed in a corporate environment, so direct access to the code is restricted. Here I provide an overview of my contributions and discuss the used technologies.

CREMA: Cluster REsources Monitoring and Analysis is a dashboard designed for monitoring resource usage in YARN applications on a Hadoop cluster. Data is periodically extracted from the YARN Resource Manager REST API, forming a Python dataframe that lists completed YARN applications with metrics such as elapsed time, RAM, and CPU usage. This data feeds an interactive dashboard accessible within the company’s LAN. The dashboard’s frontend and backend are built entirely in Python, complemented by an auxiliary HTML/CSS page. Future enhancements include integrating an external database for persistent data storage, developing the frontend in React, and creating a similar dashboard for a Redis cluster. Click here for a preview.

Checksum Tracciamento: This project centers on a Spark application designed to verify the successful completion of a MapReduce job. This job transfers a Hive database, called tracciamento, from HDFS to an external Network Attached Storage (NAS) accessible to all Hadoop cluster worker nodes via NFS. Following the data transfer, the Spark application computes the MD5 checksum for both the original files and their copies. The filenames, the calculated MD5 checksums, and the comparison results are saved in a Spark DataFrame, which is stored on HDFS in partitioned ORC files. The Spark application is written in Python.
Bureau Van Dijk: Bureau van Dijk (a Moody’s Analytics company) provides access to global databases such as Orbis, which contain detailed financial, ownership, and structural information on millions of companies worldwide. The Guardia di Finanza, Italy’s financial police force, leverages this data to uncover aggressive tax planning strategies, offshore tax schemes, and carousel fraud. As part of my work, I developed Spark applications that perform advanced queries and data transformations on datasets originating from these sources.

Page updated

Google Sites

Report abuse