Methodology

Pipeline Design

To create a streamlined pipeline, a proposed workflow making use of convolutional neural network models in a object detection algorithm is to be implemented. The convolutional networks will be made into separate modules to accomplish the respective objectives. In this implementation the models are to be stacked on each other, with the output of one being used as the input of the other model. 

At the head of the pipeline, an object detection implementation is to be applied in order to isolate the mosquito and non-mosquito objects from the initial group image inputs. Once objects are localized, the image data of said objects are to go through image classifiers to be further classified into mosquito and non mosquito objects, and then lastly into a certain mosquito species.

Image Dataset

To create a workflow that can discriminate, classify and count mosquito and non-mosquito objects, it has been determined that three datasets are to be prepared for each of the main processes. These three datasets are to be used for supervised learning.

1. Object Detection of Mosquito and Non-mosquitos Objects

• This dataset will be used to train a Region Proposal Network that will help in isolating and differentiating objects of interest (foreground) from the rest of the image (background).

• This image dataset is to contain images of mosquitos and non mosquito objects scattered in variations of assortments. 

• The dataset will also contain annotations that provide the coordinates of each object and the correct number of objects in the image.

2. Image Discrimination of Mosquito and Non-mosquitos Images

• This dataset will be used to train a convolutional neural network that will determine if am single object is between two classes: mosquito, and non-mosquito. 

• This image dataset is to contain images of mosquitos and non-mosquito objects. 

3. Image Identification of Dengue Carrying Mosquitos

• This dataset will be used to train a convolutional neural network that will determine if a mosquito object is of a dengue-carrying variety: Aedes  genus or Culex genus

• This image dataset is to contain images of separate images of  Aedes genus mosquitos , or Culex genus mosquitos

4. Image Identification of Aedes - Species

   • This dataset will be used to train a convolutional neural network that will determine if a mosquito object is either Aedes aegypti or Aedes albopictus,

   • This image dataset is to contain images of separate images of  Aedes aegypti and Aedes albopictus

For the Object Detection dataset, data is gathered via image acquisition of insect collection board setups before fogging and misting operations. The insect collection boards will be placed in specific areas indoors and outdoors. The setups are to be monitored and kept track before and after the fogging/misting operation. The images of setups before and after operations are to be collected as they are being monitored. The ideal image is to contain a collection of objects, including both mosquito and non-mosquitoes. 

Additional data will also be included to help train the model to recognize and differentiate foreground and background objects. This extra data is to include datasets where objects are manually placed into setups to help further train the model in its learning.

For the Discrimination dataset,  the image dataset is procured through a combination of manual image collection, and usage of existing image data available in the other datasets as well as online. Two classes are to be prepared, one for mosquitoes and the other for non-mosquito. For the non-mosquito image dataset,  image subjects are collected and are scrutinized to increase variety and help determine the characteristics that 

For the Identification Dataset,  the image dataset is procured through manually raising and photographing mosquitos.  The classes, Aedes aegypti, Aedes albopictus, and Culex pipiens are created from this. Additional images of Culex pipiens are added through additional datasets of mosquitos available online due to a smaller sample size collected pre-Pandemic. 

Mosquito & Non-Mosquito Collection

Collection Board Setup

Collection boards for data collection are made using illustration boards, adhesives, and insect bait. These boards are used in similar    Each board is to be placed within targeted mosquito breeding grounds. Once collected, each board is preserved and wrapped in plastic to be transported. Once safely transported, each board was documented and used for the image datasets. All of the boards and the respective contents were inspected and classified with the assistance of the entomology laboratory staff. 

Within Campus

Last September 03, 2019, there was a fogging operation held in the dormitories within the University of the Philippines Los Baños campus. A private company was in charge of the fogging and had sent multiple teams to dormitories in preparation of the fogging. The fogging operations started at 2:00 PM and ended by 6:00 PM.

With the approval of the Chief of the University Housing Office, collection board setups were allowed to be placed within the dormitories for collecting images of insect. Setups were placed in both indoor and outdoor locations of dorms with the assistance and guidance of dorm managers and staff.

Images of the collection boards were taken before and after fogging operations. Before the operation began, the dorms were vacated. After fogging the dormitories, no person was to return for a period of at least one hour to let the fog dissipate . Once the fog had cleared and staff allowed dormers to re-enter, the dorms were revisited again for the retrieval of the setups. 

Outside Campus

On September 20, 2019, a letter was sent to the Los Baños Municipal Health Office, requesting  details on scheduled fogging and misting operations within the Los Baños area. 

Revisited the Municipal Health Office on October 10, 2019 to coordinate with Sir Wilson Gascon, a Municipal Sanitation Officer. He provided dates for the next misting/fogging operation within the LB area. Two operations were to take place in two different barangays: Brgy. Tadlac and Brgy, Bayog, on October 18-19 and October 23-24 respectively.

Alongside a team of local government officers and coordinating with each barangay's respective barangay captains, the misting operations took place. With the permission of Sir Gascon and of residents of the barangay, multiple sticky boards were placed around the area. Proper safety procedures and protocols were followed during the operations.

After a 30 minute to one hour waiting period post-misting operation, the boards were collected and preserved.  A total of 34 insect collection boards were used for all the fogging/misting operations that were participated. 

Mosquito Rearing Procedure

The rearing procedure for the mosquitos included the entire life cycle of the mosquito, from egg to larvae to adult mosquito. The rearing took place within a laboratory setting. All procedures and operations done were performed under the guidance of an experienced staff member and handler of mosquitos. The mosquitos were checked on every other day or at the very least twice a week. 

The steps in rearing will depend on the current stage of the mosquitos. Steps may overlap due to some differences in the individual growth of each individual mosquitos. All containers used were labeled accordingly, based on type of species or genus and of date hatched.

Egg Stage

During the egg stage, a strip of filter paper, with the eggs of Aedes aegypti and Aedes albopictus mosquitoes, was placed into a container filled with water. The strip of paper was left there for 1-2 days to allow the eggs to hatch. 

The strip of filter paper holds up to a few thousand eggs of mosquitoes. The eggs last up to an estimate of two months of dry storage. In the case of other genus of mosquitoes such as Culex, eggs are manually collected from different sites and are placed into their respective containers.

Larval Stage

After the mosquito eggs hatch, the newly hatched larvae are to be further redistributed into different containers. The number of larvae for each container is estimated so each container would have roughly an equal number of mosquitos.

Once separated into different containers, mosquito larvae were given a tablet of brewer's yeast to provide them nutrition for growth. The tablet was left in the container to dissolve in the water. The amount of brewer's yeast was approximated based on the amount of time between feeding schedules. 

In this case, the mosquito larvae were given nutrition every other day as much as possible, with exception during holidays. Furthermore, every week, the water of the larvae was replaced with clean water to prevent over saturation of nutrients and to ensure proper and quick growth of the larvae. Lastly, if any of the mosquito larvae turn into pupa, they are to be transferred to another container filled with water, where all the other mosquito pupa per species is stored. 

Pupa Stage

By this stage, mosquitos are at a non-feeding stage and do not have to be given brewer's yeast for nutrition. The container was monitored for any changes with the pupae. Adult mosquitos are expected to emerge soon at this point, within a few days. 

Adult Stage

Adult mosquitos in the containers were transferred into a large glass box which housed all the adult mosquitos. There are separate boxes per species for the mosquitos raised. To feed the adult mosquitoes, a ball of cotton, dipped in a 9:1 solution of water and honey, is placed inside the glass box. The cotton ball was replaced every other day. 

Image Collection Setup

Adult mosquitoes of each species are collected manually through the usage of a suction device. After, the mosquitos are placed into separate containers for each respective classifications and are killed via starvation or suffocation. The carcasses of the mosquitos are used to create the appropriate image sets for the training data. Mosquito species were confirmed and checked upon collection with the assistance of the laboratory staff.

For the images that were taken of the mosquitos, an OPPO R5 Camera was used to ensure a clear image but with an affordable level of accessibility. The mosquito and non mosquito objects were placed on sticky paper backdrop to provide a strong level of contrast between foreground object and background which is beneficial to the supervised training of the neural network models. 

The images were then taken with the assistance of a fixed imaging stand setup as well as additional lighting to ensure the object of interest was well lit and not obscured. 

Dataset Preparation

To ensure a smooth and obstacle-free supervised training, certain processes were implemented before beginning training for each of the convolutional neural network models.  Datasets were prepared by being split into appropriate labeled folders for training, testing and validation. The data distribution between each followed a 8:1:1 ratio for the training, test, and validation sets respectively. Each classification for the dataset is split into labeled folders for ease.

Training image datasets have been resized to their respective sizes for each model. All image classification models have been set to 224x224 pixels, while the object detection dataset has been set to 300x300 pixels

Each of the dataset is properly labeled and kept separate with no overlap of image data with each dataset  holding 10,000 unique images in total. Annotations were done one by one for each image in the dataset using an annotation tool to keep bounding boxes accurate. The coordinates of each object is stored alongside filename, file path and image dimensions for consistency. Annotation files are kept in the same folders as the images and are saved in both xml and csv file types.

Dataset Augmentation

Data Augmentation was applied on the training image dataset before training begins to further minimize any additional computation needed. The image data was randomly adjusted per epoch of training. Changes included rotations, horizontal/vertical flips, brightness adjustments, hue adjustments, image zooms. These augmentations helped provide more variation as well as effectively increase the amount of training data without causing overfitting

Object Detection

The beginning of this proposed workflow begins with localizing and isolating objects of interest through object detection. The chosen method to achieve this is evaluated based on performance of determining and differentiating foreground from background via an object detection algorithm. After training the model, the appropriate evaluation metrics for object detection algorithms were tested and recorded to ensure accuracy.

Once an image is successfully classified, a csv output is produced. This output contains the bounding boxes of the objects of interest. This data will be used by the succeeding image classifier and will help discriminate and identify selected objects. These classifications will then be saved and kept track back on the file. 

Image Classification

The workflow will be composed of neural networks, specifically a convolutional neural network (CNN). The CNN is a type of Deep Learning algorithm that is capable of taking in image inputs and successfully learns and extract features from image inputs, starting from low-level features to more complex high-level features. The images are translated into matrices which are to be fed into the model. Through this process, it can train a model to recognize nuances in patterns based on values and weights.  CNNs are most commonly used in image recognition tasks and other image related implementations. 

{Figure}

In the preparation and formulation of the workflow, different approaches such as Binary Classification, Multiclass Classification, and Multi-label Classification were first considered and compared. 

In the case of binary classification, a three level setup of classification was proposed to ensure correct classification. Each level of classification tackled a separate point of comparison, each having an appropriate dataset collected and made to fit it. While this method allowed for more control and adjustment of parameters, it did have the drawback of requiring additional work for model training and dataset preparation.

For the case of multiclass classification, the output classes of the binary classification would be compiled into one and used in one model. Contrary to the previous method, this approach allows for faster and more efficient training if done correctly. However, due to the nature of the collected dataset, it should be noted that classes are not balanced and that dataset features being extracted may be too lopsided due to high difficulty in defining key features between certain classes.

Lastly, the third case, multi-label classification approach proposes the use of a labeling system to allow classes to have more than one classification. While it is applicable to the dataset, it does not fully make use of the flexibility and strengths of having multiple classes.

A binary classification approach was chosen to fit the needs of the proposed workflow. This approach allowed the workflow a safety net of ensuring a classification is confirmed and above a certain threshold before proceeding to the next level of classifying by a separate model. Additionally, each model gets to be trained on specific sets of data. Additional datasets were created to fit the goal of each level of classification: mosquito & non-mosquito discrimination, dengue-vector identification, and Aedes species identification.

The breakdown of the discrimination and identification to be implemented can be broken down into a basic binary tree. The initial binary classification is for mosquito and non-mosquito, then if the object is classified as mosquito, it proceeds to the next classifier and so forth. 

{Figure}

During supervised training of each convolutional neural network, the model will be trained to learn to detect and classify on each level. Each classifier takes in an input csv file containing the bounding boxes and previous classifications to build upon it.   The output of the preceding step will become the input for the other. The output is updated to include the resulting class based on the classification as well as the confidence score. This follows a standardized format to allow for simplicity for the workflow

To determine the success of the training and tuning of the convolution neural network, the appropriate evaluation metrics were used to evaluate the image classification models.

Mosquito Identification

The output of the previous object detection model and mosquito discriminator serve as the input for the image classification model implemented. The process of identifying the mosquito is broken down into two levels once an object is confirmed as a mosquito object. The first level of classification would be to identify between the Aedes (dengue carrying) and Culex (non-dengue carrying) genus. The second level of classification is based on the species of any confirmed dengue-carrying mosquito object, wherein it will be identified as either a  Aedes aegypti or as a Aedes albopictus.

For each level of classification, a separate dataset is prepared and sorted in advance.

Model Training

Model training will be based on most optimal adjustments of hyperparameters for most efficient training run time and execution. To determine the  most ideal setup for training, a number of combinations of setups will be tested. A selection of different neural network architectures was tested to see which produces the best performing model of group.

Based on the best performing model of the group, the model image classifiers were trained under it to see the best number of epochs for training. Training is monitored and the results of the training will be logged and evaluated. Checkpoints for training are also implemented to maximize the efficiency by giving options to retrain when necessary. No dropout was implemented into the models for training.

The amount of hyperparameter tuning will be minimized to within the scope of the study, so the hyperparameters for all the network training will make use of the same values. Learning rate was set to 0.00001 and an Adam optimizer was used for training,