10 Academy profile

A project to demonstrate some of the main concepts behind data engineering using open-source tools such as Airflow, DBT, great expectations, PostgreSQL, and Redash in order to perform an end-to-end ELT data pipeline.

The main objective of this project is to help the organization obtain critical intelligence based on public and private data they collect and organize.

This is going to be achieved by deploying an end-to-end ELT data pipeline that will extract the required data from several sources of data generation tools, then loading it into a data warehouse (single source of truth) in order to later transform the obtained data that can serve the needs of several people in the organization's staff like data scientists, machine learning engineers, business and data analysts as well as several reporting staff members.

An ETL data pipeline to collect and extract vocal data, transform and load it to an S3 bucket using Kafka clusters, Airflow, and spark for a text-t0-speech conversion project.

This project recognizes the value of large data sets for speech-to-text and sees the opportunity that there are many text corpora for Amharic and Swahili languages, want to design and build a robust, large-scale, fault-tolerant, highly available Kafka clusters that can be used to post a sentence and receive an audio file.

Producing a tool that can be deployed to process posting and receiving text and audio files from and into a Kafka topic, apply transformation in a distributed manner, and load it into an S3 bucket in a suitable format to train a speech-to-text model would do the required job.

Cryptocurrency and stock trading engineering: A scalable back-testing infrastructure and a reliable, large-scale trading data pipeline.

In this project, the main objective is to make it simple for everyone to enter the world of cryptocurrencies and the general stock market trade. It also wants to give investors a reliable source of investment while lowering the risk associated with trading cryptocurrencies.

Although the past performance of any financial market is never a reliable indicator of the future, it is important to run backtests that simulate current and past particular situations as well as their trend over time. Having a clear understanding of the financial system, and stock market trading, and recognizing the complex data engineering systems involved in the crypto and general stock market trading systems are essential.

Time series data set sales prediction.

The aim of this project is to predict the sales six weeks ahead across all the stores of the Rossman Pharmaceutical company using Machine and Deep Learning. The different factors affecting sales are promotions, competitions, school-state holidays, seasonality, and locality.

Building and serving an end-to-end product that delivers this prediction to analysts in the finance team.