Lessons I’ve Learnt Doing Final Year Project in Deep Learning.

It’s time to do some recap from these past 3 years. This is gonna be a long long read so, buckle up!

Before doing serious Deep Learning implementation.

Lets us take some steps backward from when I discovered learning Deep Learning is a fascinating things. It all actually begun since 2015 when Google made their TensorFlow library free to used by everyone who wanna get insight into the world of Artificial Intelligence, without writing much codes. Google released the software library for free to be used by everyone so that users (besides the Google Brain’s team) can submit bugs report to them so that it can be improved for a better implementation.

The library is released alongside a tutorial of TensorFlow for Poets, a GitHub repository that contains the code pre-written by Google’s engineer and are ready to be used. The tutorial gives some insight on how to run Deep Learning training and doing inference on test data that can be downloaded over the internet. Essentially, the tutorial taught me about the implementation of Transfer Learning, which is a method of “transferring knowledge from previously learnt features onto new datasets”. Once the code (retrain.py) were executed, it will download the architecture models that I specified (say, Inception_V3) earlier to the working folder and run Transfer Learning to the flower dataset. This method will only trains the datasets with the final layers of Inception_V3 and compute the accuracy from it.

Paper on Identification of Different Stages of Diabetic Retinopathy.

Running that Python scripts repeatedly and sees the training progress really kinda inspiring, I mean, especially when you have your own dataset and you have an idea on what to do with it. By that time, I only have a laptop to run the Transfer Learning scripts, which is good enough and luckily my laptop is occupied with nVidia GPU, which is needed to run TensorFlow library with the help of GPU as a catalyst of the training. I actually tried both type of TensorFlow library, the one that run on CPU power and the one that I’m using run on GPU to leverage the parallelism computing.

During the period of exploring and understanding Transfer Learning, I also tried to do the training on much more bigger datasets such as Animal dataset that consists of almost 50-ish type of animals where each type of animals consists of thousands pieces of images. It feels so good to run training on different type of datasets and see how the algorithms react to the differences of image pixels, as well as playing around the Hyperparameters to bring difference results (crazy, ain’t it?). During this year, Transfer Learning is just merely run the Python script and Voila!, result be ready less than an hour.

Learning Artificial Neural Networks (ANN) and Convolutional Neural Networks (CNN).

Fast forward to 2018, is a year when I started to take an online courses of the implementation of CNN by using TensorFlow as its library (other such as CNTK, PyTorch, and Keras) offered by Udacity with the collaboration with Google. Since the requirements to take the course are just have knowing programming’s OOP, and some Python syntax, I jumped in. But prior of really jumping to the course I also took some Python classes to understand the basic syntax and its form and how to get started writing Python.

I’ve been thinking for several months that I plan to learn this CNN things and start writing a neural networks for specific kind of usage, which in my case is the study of diabetic retinopathy classification. There are a lot of free Machine Learning and Deep Learning courses on the net that also gives you professional certificates at the end of the course, I took some of them and use my spare times to learn from the basic traditional Artificial Intelligence up to Deep Learning models architecture such as CNN that is used for Image and Video-related classification and detection.

Depending on some people, I believe MOOC is a great place to learn and hone new skills as the subjects are not usually available anywhere without sign in into particularly courses in colleague or universities. Some people are not too familiar learning stuff just from sitting in front of their computer for couple of hours. Some prefers the face-to-face learning style, which is the best. Learning online courses over the internet is the fast track on gaining new knowledge at the same time applied it to the real-world problems.

There are several online courses that I took prior starting my Final Year Projects in Deep Learning-based project. Among of them are;

Coursera’s Specialization:

1. Machine Learning with TensorFlow on Google Cloud Platform
2. Advanced Machine Learning with TensorFlow on Google Cloud Platform
3. AI for Medicine

Udacity’s Courses:

1. Introduction to Artificial Intelligence
2. Intro to TensorFlow for Deep Learning by TensorFlow
3. Introduction to Python Programming

Hardware Investments

During the period of these 3 years, what I only knows about running Deep Learning on nVidia’s GPU is that I must have as many as CUDA Cores as possible for faster parallelism computing on the image data. But that’s not the only thing to consider when deciding which features Neural Networks need in a GPU. During the period of learning Transfer Learning with the TensorFlow for Poets tutorial, I only have my Acer’s 2013 laptop that were occupied with 96 CUDA Cores and 2GB of VRAM. The retraining process with the Python script provided ran well as the data were only trained on the final layer of the Inception and MobileNet (depends on the choice) model and transferred the “knowledge” to the dataset that I’m using.

Come June 2019 is when I finally decided to spend some money on building a proper computer for this research study, as well as to ease my other programming projects code compilation. At this period what I do know is that, any PC build even a budget one can already top up my laptop’s specification. Meaning any Desktop version of GPU with decent amount of RAM is “good enough” to catalyst my neural networks computation. Well, that was wrong. At least, for a beginner like me at the time I thought I already have enough until in the mid of Mac in the next year where I finally realized and got new knowledge as well where, the complex the neural networks is, the more VRAM (GPU’s RAM) it needs to store the temporary compute data.

Shifting from GPU to CPU compute

What I meant by “Complex neural networks” here is, by the number of the layers, how much neurons per layers, as well as other API that were implemented as the training test is executed. Unfortunately, my GPU is only occupied with 4GB of VRAM that according to my test can only trains 1 complex layer and minimum of 2 or 3 simple layers. Luckily at that moment, my CPU is already occupied with 6-Cores 12-Thread and 16 GB of RAM. Technically speaking, system’s RAM is 4 times bigger than my GPU’s VRAM, then why not run the training on CPU instead? Well, this is because matrix computational runs much much more faster and in a blazing speed (insert your acronym here) in GPU than on a CPU. But I have no other options at the time LOL for local training, CPU is a go-to even though it took some times to return the results. Matrix computation runs in parallel to each other and running it on the GPU is a good strategy as GPU is indeed built for that purpose, paralleling computing.

Re-train with Transfer Learning, never put this away.

After almost 3–4 months tinkering with training from the scratch, meaning I design my CNN and make it train with the datasets consists of hundreds of images of 1000 by 1000 width times height (cropped out from 4000 x 4000), I still haven’t got my desire Accuracy results. I Day to day I still thinking on how to boost up the results without overfitting the validation dataset. Time ticking second to another. All the results are only in the range of 30–50 percent of confidence, which is not well enough considering of what I believe with my dataset numbers are big enough to start with.

One thing I never put away is the lesson I’ve learnt in the past, when I already found a new methodology. It will be by my side because I know they will be times when I will need it again, especially during the short emergency time like this. Since I am no expert in this field, my background in programming is just from mobile development and some electronics project of Arduino, Deep Learning is a new thing for me. I mean, Deep Learning is not just merely a Train and Inference study after all. Training from the scratch will have you to manually look up to the results, evaluate it and see the Accuracy and Loss manually to decide whether your results are Overfit or Underfit to the Training or Validation dataset.

But that’s not the case for Transfer Learning, which is my stepping stone in this field. Transfer Learning repository provided by Google earlier already handled that case. I returned back to the repository, clone it into my computer and start putting my dataset in the respective directory. Trained it with the help of MobileNet and InceptionV3 model, I was able to get a much more higher Accuracy of around 70–80% if I still remember it correctly. But to boost the Accuracy is not always the hard part. I have to play around the Hyperparameters that are related to make sure the algorithms “learnt” and “understand” the image pattern.

Automated Machine Learning.

About 2 weeks before the presentation day I finally decided to focus on the implementation of AutoML. This moves eventually made me to study AutoML methodology in a very last minutes. What is it, how it is working, principles, and so others. Mostly from the paper presented by this team. Automated Machine Learning, or AutoML for short, is a new methodology aimed to design and train the data on a newly designed Neural Networks based on what kind of the data is. Assuming the data is an image type of data, the algorithms knows it’s an image data with some specifications, thus AutoML can create a better CNN model to train the image data thus also return a very high Accuracy of around 70–90% of Confidence.

What I felt so wrong with AutoML is that, I thought the word “AutoML” is just meaning the systems will automatically build the model and train whatever data ingested to the platform and do whatever tweaks to make sure it will returns a high Accuracy results that can be used later during Inference, but that was wrong. I’ve been using Google’s Cloud AutoML since its Beta inception (not to confused with Inception model) and thoughts that it was the same CNN in the background all over. Actually, AutoML is totall a different thing. Different methodology too.

I regretted for not studying the fundamental of AutoML at least the surface part until I feel it’s too late, about 1 week before my final presentation. That’s the moment when I have to go to all the related YouTube videos, as well as presentation papers to understand the “automated” concept and how it’s executed.

Other Lessons I’ve Learnt Along The Road.

Deep Learning field study commonly known as the field of study where only people that expert in programming software computers and mathematicians involvement. But that is no longer an option when Google decided to Open Source their library to the public for everyone to test out and return feedback. I am one of them.

In this timeline of pursuing this study either unofficial and officially started it, I learnt a couple of things that I believe every beginner SHOULD consider when stepping into this field of study. Among of them are;

1. Strong background in Mathematics (things like Probability, Analytics and more) and Programming (Python or R or any language of your choice). There a lot of people out there that are very ace in Mathematics, though programming with the help of libraries like TensorFlow is actually make the mathematical jargon being taken care of. But, in order to explains or to justify why the results became such and such, mathematics explanations and logic is where the results stand in your report later.
2. Learn from the beginning. Deep Learning is a subset of Machine Learning, which is also a subset of Artificial Intelligence. For me it’s not a wrong thing to jump straight away into Deep Learning and Computer Vision, but the fundamental still on the above layers of this field which is from Mathematics and Artificial Intelligence. Knowing the fundamental can make it easier to go through deeper stuff in the field. Essentially, during my early days pursuing my Bachelor, I never knew about Neural Networks as a concept to visualize human’s real Neuron Networks and how its interconnect to the other. I have this lecturer that teach Artificial Intelligence subject at my campus, used to have a chat with him couple of times and were asked if I knew SVM (usage and methodology), Random Forest and other more Machine Learning and Deep Learning methods. Later he suggested me to join his class once he starts Neural Networks topic which I attended for a few weeks until the end of the topic. Thanks to him I finally got the fundamental knowledge and how it relates to Matrix calculation as well as other computational depending on how complex the model design.
3. The ability to distinguish different type of data. Meaning you know this data can be used and the other one cannot be used. In image classification study, I put away blurry and noisy images from my datasets.
4. Computer architecture knowledge. You should know how computer works in general. Where the execution happens, what’s the memory used for, storage. You should also knows how the data being carried from Storage, to the CPU, to the RAM and etc. Having these basic knowledge enables you to know what are the specifications of computer that you will need when you already decided to build your own computer for Deep Learning purposes in the future. Don’t be cheap on some components such as your motherboard, my advise to straight up go for ATX form factor that comes with 4 slots of RAM that can support up to 128 GB of RAM, as well as chances to use double GPU cards that gives you double the VRAM capacity, if your budget is permissible.
5. Don’t cheap on GPU. This is my personal opinion because I really build a PC for Deep Learning purposes, beside for gaming (yep, I do sometimes to chill). At first, I believe any nVidia’s Desktop GPU would do as long as it has some hundreds of CUDA Cores, which is used when training images (not sure about other Deep Learning studies). So, I got myself a GTX 1650 that comes with 4GB of RAM and 896 of CUDA Cores. Just enough for gaming, as well as training one or 2 CNN layers. As I’m experimenting on few more layers and Neuron number, both Spyder IDE and command prompt (easier way to run Python script) starts to throw OOM errors of insufficient memory which make me realize that my GPU can’t hold more data inside the 4GB of its VRAM. What left for me at the moment is to initiate Transfer Learning as one of the last resort.
6. Reads a lot on the topic that you’re studying (my case is CNN implementation). You need to read a lot of materials in order to understand and implement others methodology. There are a lot of researchers out there that being doing study on this fields that are much more experienced than us, which make it is easier to try out their approaches to our case study. Platform like Medium often has many writers that write a lot of articles in Image Processing that can be used as a guidance. Deep Learning is a field of experimentation with its hyper-parameters. Some parameters can only works with A parameters and some might not. So, it takes a lots of experimentation to and time to test out each suggestions and methods. Even in CNN itself, there are a lot of CNN implementation and its architecture such as Inception, MobileNet and DenseNet.
7. Explore Cloud Services. Now, if you are in the budget constraint like me, I do use cloud services like Google Cloud Platform to have my dataset trained. But the thing is, I have less expertise in using it. I mean, I have difficulties in pipeline-ing my data to the VM properly. But I still suggest anyone to explore this things because if for Google, they’ll give you free credits about $300 for everyone test out all their products comprising from App Engine to Artificial Intelligence products. As of what I sees, Google’s platform kinda separated into 2 products for AI which technically both are actually under 1, that is Artificial Intelligence and AutoML product. I used AutoML a lot and that cost me to understand AutoML in a week before my presentation and report completion LOL.

Conclusion

Deep Learning is fascinating field of study because, on the surface one can do almost anything autonomous such as Object Detection where a computer can see objects in image or video through the camera and return the results of what is/are the objects that the system is looking. Applying Transfer Learning method on all Deep Learning field study could be a piece of cake jobs as long as you understand Python professionally. Although as fascinate as the name of the field is, a “deep” study on the field is also needed prior of jumping in because once inside, there are a lot more to think about besides the basic stuff.

I jumped into the field initially without any basic in Mathematic function, as well only the fun parts of building Neural Networks model. I found a lot of people been saying about “implementing AI in their project”. If you already have a model that is already “fit” for the project, then you can easily straight away implement it. If otherwise, then starting from the bottom is all you need to do; gather the data, clean it, separate it into 3 sub datasets, build the model, compile it and only then you’ll run the training. After training finish, observe the results for Overfit or Underfit case. If one of these exist, then the training need to be redo until you get the model that is “fit” to be used in Inference, or Classifications or whatever the case you’re studying.

I believe those above are the only things I remembers as a beginner in Deep Learning or some might say as Data Scientist field, doing it as a Final Year Project. It’s very challenging especially when I still have no idea what else to tweaks the code in order to make it do what I want or returns the results I wish to. Till next time fellas. Thanks for spending times till this point.