Working Tools

Git is a version control system allowing for complex versioning with both remote and local repository support allowing for continued work even when remote repositories are unavailable. Considered the standard for repository management, GitLab and GitHub are hosting services which follow the git system. Especially useful as it allows multiple branches of the same repository to exist, each with their own changes, and the ability to merge these branches when the changes are ready.

Similar to PyCharm, IntelliJ is a fully fledged Java Integrated Development Environment (IDE) and is focused specifically on Java projects. This IDE has many of the same features as PyCharm, with additional support for Java specific tools such as Maven integration.

Visual Studio Code (VS Code) is a "jack of all trades" for coding. It is a general text editor with plugins available to download that support a vast amount of coding languages and features. With enough plugins, one can transform VS Code into an almost fully fledge IDE! 

For cases where a beefy, feature rich IDE is not needed, this is the perfect tool. Additionally, its support for almost any coding language makes it perfect for tackling projects in a language that might not have an easily accessible or usable IDE. With VS Code you can tailor the experience to your liking.

Docker is an environment virtualization tool that is especially useful for creating reproducible environments, i.e. making sure that the project will run with necessary software level dependencies fulfilled, no matter the actual physical machine it will be running on. 

The point of this tool is to allow easy set up of all project dependencies so that, once set up in a dockerfile, the project can be run by anyone with that dockerfile without any further downloading or tinkering necessary to get it running, regardless of that persons actual software environment. These specifications range from OS level specification all the way to specific version of code language and packages/libraries. This ensures a consistent runtime environment for all members of a project throughout a projects development life cycle, all the way from a projects inception to deployment. 

A machine learning library which which provides a robust framework for the whole ML pipeline from pre-processing to training via autograd to inference. A large part of the API is integrated from Keras which was originally a separate package and provides "pre-built" solutions such as layers and architectures with several elements that can be customized to specific use cases. Additionally, the TensorBoard API provides visualization tools which allow for easy debugging and information regarding the training process.

There are two execution modes, eager mode and graph mode. Eager mode is good for model development and does not construct graphs for the underlying operations. These graphs "can provide optimizations that make models run faster with better memory efficiency", and are good for productionalizing an already developed model.

Apache Spark is a distributed parallel compute engine, and PySpark is the Python interface. Many large data pipelines which require complex transformations are more easily handled in Python than native SQL. Additionally, loading in data to Python using pandas read_sql functions can take quite a while. This is where PySpark comes in, adding parallelization to querying and data transformations. The Spark session can be run locally, or harness a properly set up computing cluster for maximum performance gains. The local session might still garner performance improvements if the machine has multi-core capabilities, and what once would take several minutes on a single thread can now be parallelized and reduce execution time significantly.

While Spark is built mainly for large scale ETL pipelines and classical ML algorithms (through the native SparkML library), there have been community made libraries for integrating Spark with more modern ML API's such as TensorFlow and PyTorch. However, there are also other tools such as Ray, which is explored below, that have been directly built for distributed modern ML pipelines and take advantage of GPU compute power for parallelization. Spark is typically more focused on classical ETL which relies more on CPU compute power, although this looks to be changing with the release libraries such as Horovod, as well as Spark 3 which natively introduced GPU acceleration features.

Ray is similar to Spark in that it is meant for distributing work to multiple worker machines, however, Spark is more geared towards data transformations and traditional ML methods instead of the new complex deep learning methods that are becoming more common today. Ray is more geared towards GPU-based operations which greatly benefit DL applications and allows for the use of native Python code and libraries such as scikit-learn, PyTorch, and TensorFlow without much adjustments. This is in contrast to PySpark which requires significant set-up and modifications to get things working properly.

Enterprise data warehouse(EDW) used to store massive data sets and enable data table manipulations/retrieval via queries. This database has its own SQL dialect with some slightly different keywords such as TOP (instead of LIMIT). The database can be queried by the native "Teradata Studio" app and also can be connected to via the teradatasql or teradataml Python package which has some built in ML functionality. 

This data solution is considered somewhat legacy and has become far less prevalent than the more modern GCP/Azure/AWS based alternatives. Large-scale queries often times run into spool space errors if not properly set up, and scaling is lackluster at times, slowing down query speeds on especially high-traffic days. The overall experience feels clunky when compared to more modern alternatives, data debt being the main factor for users to stay on this platform.

The Vertex AI Platform is a large-scale, complex, fully managed service offered by Google. The platform provides purpose-built MLOps tools for data scientists and ML engineers to automate, standardize, and manage ML projects.

The platform offers many benefits, notably, Vertex AI Workbench which is a fully managed, scalable, enterprise-ready compute infrastructure for running Jupyter Notebooks. This service allows customers to run JupyterLab Notebooks on Google Cloud servers with the ability to specify hardware environments to match compute needs, from beefy Nvidia A100 GPU based environments for training deep neural networks to high RAM environments for easy in-memory EDA(Exploratory Data Analysis) with Pandas, all without having to set up any hardware or worry about availability (as long as you can afford it).

The platform also comes with a variety of other tools which are all connected to the overall Google Cloud Platform such as Model Endpoints, Datasets, and Pipelines. All of these tools work well with each other, and are usually designed to be used together, along with other aspects of GCP such as BigQuery or GCS.