Project Koios is a personal computational cluster which I am building to do small scale simulations and to assess the feasibility of using smaller computers to do cluster computing. The goal of this project is not to determine if we can build HPC (high performance computing) clusters similar to research universities and national labs using this hardware, but to determine if small personal clusters are useful as a (1) tool for education and professional development, (2) a development platform where computational concepts and workflow ideas can be developed without initially having to deal with contention and long queue lines common in HPC facilities.
The format of this project will primarily be a daily blog containing my ideas and impressions, an assortment of notes and directions which I will try to organize, and some type of ongoing document which documents the changes to the cluster over time. I am building the cluster from the bottom up and intentionally not using package manager which automate the setup. Personally, I think that a bottom up approach builds a more solid understanding of the underlying technology (particularly what can go wrong, so it becomes easier to troubleshoot problems).
Finance Projects
Market Data
Cluster installation notes can be found here. I do my best to update these notes as the best as I can.
Website
www.projectkoios.com, Notes
Cluster infrastructure
installation of distributed file systems
installation of MPI
installation of job submission managers
Basic Skills and Programming
python
numpy
scipy
matplotlib
scikit-learn
jupyter-notebook
Machine Learning
neural networks /deep learning (tensorflow/pytorch/caffee/mxnet
SVM
XGBoost
Random Forest
Generalized Linear Models
[20210630] Installation of operating systems
4x NUC7i5BNH/4-core i5 processor/32GB RAM/1TB NVMe SSD Drive/1TB SATA SSD Drive
4x Raspberry Pi 4B 8GB/64GB Flash Drive
4x NVIDIA Jetson Nano