Following Messi with TensorFlow and Object Detection

In previous publications we were using TensorFlow in combination with the Object Detection model, but always making use of the traditional pre-established datasets [example COCO database]. This time our challenge should take us another level and I will propose analyze a segment of a soccer game and identify its players [at least one of them].

When talking about soccer, the first thing that came to mind is look for the talented Argentinian star Lionel Messi. I will propose something:

• Let’s train a personalized model that allows us to locate Argentine star.
• Let’s analyze a real transmission [in this case I selected Uruguay-Argentina by Conmebol qualifying August 31, 2017] to see what results we can get.
• Let us think if it is possible to improve or contribute something to the tactics of this passionate sport, using this technology.

Note: We will use TensorFlow and the proposed model for Object Detection, all under the Apache License 2.0 license.

In this opportunity I will start by showing you the result obtained:

If you find them interesting, I invite you to dedicate a few more minutes to follow the step by step to understand the procedure of building this model.

We should complete three important moments:

• Build the custom dataset with the objects to be detected. [in this case Lionel Messi in full action].
• Train and evaluate the model using TensorFlow.
• Use the created model.

Step by Step process:

Build the custom dataset Moment

1. If we explore the official documentation, we will find an exclusive section to build our custom dataset. Here you can see it.
2. We need the images of the objects we want to detect and we need to label them.
1. For this case I used the RectLabel tool, a good option for MacOS. [there are other alternatives like LabelImg]
2. Then for the first tests I collected about 119 images [very little number, but is only for testing purposes] of a soccer game.
3. I have labeled about 100 units selecting the Argentine star.

Note: To put this example into production, it will definitely require many more images in this step.


3. The next step is to convert the photos and their corresponding tags into TFFormat files. Here you can see the script used. [I used a intermediate step to convert the xml annotation labels to a csv, here the script]
4. Remember that the ideal environment is to build a training set and the evaluation set.

Train/eval the custom model Moment

      5. With these complete files, we will begin to train and evaluate the model.

1. For this step I have used ML Engine since I did not have the local infrastructure to perform the process in 1 hour average. [here is the official description of the procedure]
2. 5 workers using standard gpu.
3. it is also highly recommended to associate TensorBoard to visualize the training and evaluation process.
4. Remember to use a direct interaction with Google Cloud Storage. In other words, all files must be in this type of repository in order to interact with the training engine.

*After a 21.77 units used, 1 hour 9 mins and about 20k iterations we have several checkpoints and an acceptable model.

       6. Using the most advanced checkpoint, we proceed to convert the trained model to inference. [Here you can see the official documentation]

Use custom model Moment

       7. Finished this procedure we will use the model from these options:

1. script that uses the model and generates a video with the result of the detections. Here is the implementation example.
2. script that can evaluate the video in realtime and decode the labels to calculate the appearances of the Argentine star. Here is the implementation example.

Note: same demo using slow-motion. Here

** more implementation information here [tensorflow-code-experiencesgithub repository]

Perhaps it is the beginning of an interesting line of work… how sports tactics can be improved, incorporating more players and validating associations. For now, we are using only the traditional TV transmission delivered, but what would happen if we had the option of cameras with more details or a complete view of the soccer field.

The positions of all the players could be calculated, in fact we validated that the players can be easily identified, then… could we find better tactics based on player’s position and their well-known skills?

I hope you had fun with this procedure, and I hope to see more and more TF implementations in sports disciplines such as this.

original publication: medium link

Following Messi with TensorFlow and Object Detection por Nicolas Bortolotti se distribuye bajo una Licencia Creative Commons Atribución-CompartirIgual 4.0 Internacional.