Team:Patras/Artificial Intelligence
Artificial Intelligence
- Introduction
The advantages of computer science and machine learning are establishing a new era for various scientific fields. The developed algorithms are employed in pattern recognition, object detection, and different research areas. The availability of a significant amount of information is encouraging the substitution of the heuristic approaches that are widely used so far from machine learning techniques. By combining a given knowledge through a dataset with the computational efficiency of modern computers, it is possible to create a system able to learn and come to conclusions as a human does. Such systems are starting to embed in healthcare and the corresponding research. In most cases, they can make accurate predictions using diverse data sources, outrunning humans in terms of reliability and speed.
- Description
Our project is based on the collaboration of computer science and engineering with medicine and pharmacogenomics. At first, we applied machine learning and computer vision techniques and created a user-friendly system where a health professional can use it. Electrophoresis images from the BentoLab are used to train deep learning object detection models and test whether they correctly classify a patient's genotype to determine the exact administered dosage. Our models' validity is proved using a set of test images that haven't been used during the training process and images uploaded to our system by the three medical students we invited to test our BentoLab-AI system.
- Methodology
Our task was to locate and classify different objects of interest (OoI) in images of the electrophoresis gels. Each image consists of 6 sample columns, with each consecutive pair representing a sample of a person, 2 blank sample columns, and the ladder. To perform multiple object detection, a modification of the state-of-the-art algorithm Yolo is implemented. A dataset is provided by the wet lab that relates the DNA band's presence in each column to the corresponding sample's genotype. The dataset contains 20 images, and there are four (4) possible combinations of DNA bands for each column pair. A reference image can be seen in Figure 1 - left as captured by the BentoLab electrophoresis device.
Figure 1 - right shows the image as manually cropped to contain only the region of interest in different ratios and zoom levels.
Initially, each image's annotation takes place. A bounding box and a class describe each OoI. The bounding box is described by its center coordinates (X_center, Y_center), the height, and the width in terms of pixels. A confidence level in terms of the probability that is considered equal to the unit for the training data is also given. A class is defined as the column of the identified bands in each image. Consequently, there are eight (8) different classes describing the bands and one class representing the ladder. Classes are described in categorical form.
In Figure 2, the OoI, i.e., the ladder and the bands, are shown in red and blue, respectively.
Figure 2
The implemented network takes as input a resized image of fixed size 288 x 288 px divided in 324 (18x18) equal areas called cells, and it uses three (3) anchors. The anchors are the most frequently used widths and heights found in the dataset images, helpful in the training process. In Figure 3 is shown the form of network output. The network produces for every anchor of every cell those fourteen values.
Figure 3
Given the fact that the network will produce an output (prediction) for every anchor of the cells, the output size will be 18x18x3x14 px, and it is shown in Figure 4.
Figure 4
Grid_x, Grid_y is indexing the corresponding grid in the range [0,8]. The anchor is in the range [0,2].
The implemented deep learning network architecture consists of eleven (11) blocks of layers and the output layer. The blocks contain layers such as convolution, batch normalization, and Mish activation layer. We present parameters such as number filters, kernel size, and stride size for the convolutional layers. In Figure 5, the architecture of the network is shown.
Figure 5
As an example, in BLOCK_2, the ZP((1,0),(1,0)) is a zero-padding layer, C_2d (8, ker (3,3), str (2,2)) means 2d convolution layer with eight (8) filters, a kernel of size (3,3) and strides among height and width of 2. The B_N is the batch normalization layer. For the implementation, we used Keras, a laptop equipped with Nvidia Geforce GTX 1500 Ti with 4GB Ram. The images fed to the network in grayscale color and a Horizontal Sobel filter is applied to each of them.
The OUTPUT block in the above figure will produce the output of the network, which will be:
grid *grid * anchor * 14.
The Non-Maximum Suppression [1] technique is used to preserve the best bounding box for each detected object.
This work aims to classify a person' DNA sample based on the presence of bands in a pair of columns of an image to the following classes based on the genotype:
- Class_0 (C|T Intermediate myopathy risk) if in both columns DNA bands are present
- Class_1 (C|C Normal myopathy risk) if the first column has a DNA band and the second has not
- Class_2 (T|T High myopathy risk) if the first column has not a DNA band and the second has
- Class_3 (Error) if in both columns there are no DNA bands
In each column that both DNA bands are present, we need to determine if they are accepted because of the difference in brightness that they might have. To achieve that, for every DNA band, Yolo predicts if its pair exists, we calculate the background average pixel intensity, taking values from above and bottom bounding box area shifted by 5 10 15 20 pixels vertically from its original position (avg_back). We find the brightest of the two bands (bright_bb), and we calculate the threshold as follows.
If the predicted bounding boxes have an average pixel value below the threshold, they are removed from the prediction. The same procedure is applied for the remaining bounding boxes. We calculate the background average pixel intensity, taking values from above and bottom bounding box area shifted by 5 10 15 20 pixels vertically from its original position(b). The threshold is the following.
If a bounding box has an average pixel value smaller than the threshold, it is removed from the prediction.
- Data Set
Our original dataset produced by the wet lab team consists of a set of twenty (20) images. Five (5) of them will be used for testing and the rest for training. As we show in Figure 1, we decided to expand the dataset using cropped and zoomed images to have more available images and help our model perform good detection. An example of the dataset images is given in the next Figure 6 and Figure 7.
Figure 6
Figure 7
Our test dataset contains seventeen (17) images in different crops and ratios. An example of the images is shown in Figure 8.
Figure 8
Even though our project's main goal is to prove our idea by using AI techniques to solve health-related problems, our team tried to create Deep Learning models able to generalize their knowledge and perform accurately in images that they are not trained in. We created new images using a photo editing program, Photoshop, to generate more training data to improve our model's ability to correctly detect the OoI in new images. The procedure was the following; we cropped bands, and we moved them around the columns, so for every column, we can have a representation of bands. We can see the distribution of band classes in the dataset in Figure 9. The distribution after the creation of new images with Photoshop is given in Figure 10.
Figure 9
Figure 10
- Results
In this section, we are going to present the results of our work. The training parameters such as optimizer, epochs, and learning rate are shown in Figure 11.
Figure 11 - Training Parameters
Since we had 5817 images and a computer with 4GB ram on the GPU (Graphics Processor Unit), only 250 images were randomly selected from the dataset in every epoch. Each batch size consists of eight (8) also randomly selected images.
The training and test loss of the training procedure are shown in the next figure (Figure 12), where train and test loss have similar curves, meaning that the network can predict accurately with the test images.
Figure 12 - Train and test loss
Images with results from the training set are presented in the next figure (Figure 13).
Figure 13 - Results on train images
The AI system correctly recognized the genotypes of the four presented samples. In the first column, because of the presence of the band, the genotype is identified as T|T. The following sample was identified as T|C, due to the presence of the bands in both columns. Lastly, the presence of the band in column 6 stands for the C|C genotype with the recommendation of reducing the administrated dosage of statin.
Figure 13 - Results on train images
The AI system identified the bands on columns 1 and 2 that match the genotype T|C. On the following 3, 4 identified both bands again, but after comparing their intensity, classified the sample on class_2 (T|T genotype). The last two columns were classified the same as the first pair.
Figure 13 - Results on train images
The AI system correctly recognized the genotypes of the four presented samples. Both bands were identified in the first two columns but after comparing their intensity, classified the sample on class_2 (T|T genotype). Columns 3,4 had the same classification as the first case. In columns 5, 6, only the band in the sixth column was identified, resulting in the genotype of C|C. The last sample (columns 7, 8) identified with a genotype T|C due to the presence of both bands.
Figure 13 - Results on train images
The AI system correctly recognized the genotypes of the three presented samples. Only the first band was identified in the first two columns, giving a T|T genotype. Columns 3, 4 represent a sample with T|C genotype. And the last one, due to the identification only of the second band (column 6), is classified as C|C.
Figure 13 - Results on train images
The AI system correctly identified the genotypes of the three samples once again. On columns 1, 2 recognized as false the shadow on the 1st and accepted as valid only the band on the 2nd, classified the sample in Class 2 (C|C genotype).
Images with results from the test set are presented in the next figure (Figure 14).
Figure 14 - Results on test images
The AI system identified both the two bands (column 1, 2), comparing their intensity, classified the sample on class_2 (C|C genotype). In columns 3, 4, the Ai identified only one band in the fourth column that indicates a C|C genotype.
Figure 14 - Results on test images
The AI system identified both the first two bands (column 1, 2) matching the genotype T|C. In columns 3, 4, the program classified the sample as Class 2 C|C genotype after the comparison between the two bands.
Figure 14 - Results on test images
In image 3, the AI system identified two true negative columns (1, 3) and two true positive columns (2, 4) by comparing the bands' intensity. In this image, the two samples have a C|C genotype.
Figure 14 - Results on test images
In the image 4, the AI system correctly recognized the genotypes of the three presented samples. The first sample is classified as T|T, the second one as T|C, and the third as C|C genotype.
As we can see from Figure 13 and Figure 14, our system performs really well in locating the bands and assigning them to the corresponding class based on the column they are in. We can also observe that for each pair, most of the time, the comparison between the bands and the background removes the band correctly when it is needed.
We conducted a real-time test of our system with a medical student who followed our instructions and performed the experiment. The output picture of the experiment is shown below in Figure 15a.
Figure 15a - Image from Realtime experiment from medical student
In Figure 16, we present our system's results to these two pictures in different crops, zoom levels, etc.
Figure 16 - These images present the outputs of the Ai system with different crops and zoom levels from the original image. The identification and the classification of the sample are the same in all output images.
In Figure 15b, there is another image that the network isn't trained with which we test our network's predictions using samples of this image in different crops and zoom levels.
Even though the system may sometimes classify the wrong class for the DNA bands in cases where the image is not correctly cropped or taken from different height or bands appeared too small or too big cause of zoom level, in general, our network performs really well and most of the times accurately even though we possess a small number of images to train it. We suggest to crop the images & contain only the gel or contain the gel until the ladder's height while using our system to take the maximum out of it in terms of accuracy.
Sponsors
About
16 students from Patras blending Pharmacogenomics with Artificial Intelligence to redefine medicine
Follow Us
© iGEM Patras - All Rights Reserved