The app that helps with safe driving (CreateML test)

read in google docs: https://docs.google.com/document/d/1gKWVnRc9ZYRUDh8GEUwlEUMuqMTSiP3DzFvezw82YHI/edit?usp=sharing 

Introduction to the problem

Artificial intelligence and neural networks are really popular lately. I’m also interested in this topic and I work on many projects that use neural networks or machine learning. When I’ve first heard about CreateML I knew it will change things, and we will see more intelligent apps. It is so simple to have great results with this tool that I’m surprised that it’s not in more apps today. Then I thought about testing it. It’s great with simple tasks and it’s really easy to use, but the question is:

Will it work with more complex tasks?

To test it I need more complex tasks, so I need data that is harder to analyze. I’ve decided to work on photos of streets (I’m recording street while driving). Then there are steps that I will take:

1. Train model to recognize maximum speed signs with CreateML
2. Train model to recognize maximum speed signs with Turi create and object detection (to compare results)
3. Train model to decide if car should accelerate or brake (maybe turn left or right) with CreateML

Each model will give different results with different training and testing data, so I’ll differ that too. There will be artificial data, real photos and for testing, I’ll also test it while driving (some mistakes are more important than others, so it’s crucial to test it in real situations).

Results are in % of correct predictions (the more the better)

If not specified number of iterations used is around 1400

Data

speedSignsSimple1.1, speedSignsSimple1.2

The first data set contains simple signs, rotated and resized. Then there are photos on which these signs were put. Additionally, in speedSignsSimple1.1 there are 30 real photos and in speedSignsSimple1.2 60 real photos of signs (taken from a driver perspective)

speedSignsSimple2 - speedSignsSimple6

The same sets but with the increased number of real photos (additional 10 000 photos)

model blur

first artificial set with some photos blurry, with noise and with different brightness

speedSignsMy

New artificial set with blur noise and brightness change from the beginning.

speedSigns1

combined sets model blur + speedSignsSimple6 + speedSignsMy

speedSigns2

Almost identical set to speedSigns1 but with few more photos and to create model I increased number of iterations (8000)

first results

Data	Training	Validation	Evaluation	Date
speedSignsSimple1.1	100,00%	28,00%	31,00%	2018-11-01
speedSignsSimple1.2	100,00%	83,00%	40,00%	2018-11-02
speedSignsSimple2	100,00%	85,00%	19,00%	2018-11-03
speedSignsSimple3	100,00%	88,00%		2018-11-04
model blur + speedSignsSimple6	69,00%	60,00%	73,00%	2018-11-24
speedSignsMy	87,00%	49,00%	12,00%	2018-11-25
speedSigns1	64,00%	58,00%	77,00%	2018-11-27
speedSigns2	65,00%	57,00%	70,00%	2018-12-01
coming soon 😃
coming soon 😃

As we can see from the beginning results aren’t bad with 25% evaluation score and 100% of training score. Even though the training score is worse later on but the most important score — evaluation is significantly higher, so the speedSigns1 model is still the best. It’s score still doesn’t seem so impressive with only 77% but it's a great improvement and when I was driving it was very good at showing the right sign. The main mistake it did was not seeing the sign on some photos. Also, photos that are wrongly labelled has very low prediction score. The model isn’t sure where to put them and the app saves change only if the model prediction score is higher than 85%, so unwanted results will be ignored.

CONFUSION MATRIX (speedSigns2)

True\Pred	do30km	do40km	do50km	do70km	none
do30km	1	1	0	0	3
do40km	2	16	0	4	12
do50km	0	0	0	0	2
do70km	1	4	0	4	9
none	0	5	3	19	134

PRECISION RECALL (speedSigns2)

Class	Precision(%)	Recall(%)
do30km	25.00	20.00
do40km	61.54	47.06
do50km	0.00	0.00
do70km	14.8	22.22
none	83.75	83.23

To be continued...