Gsoc 2016 Finish

For the last 3 months I was working with CloudCV for Google Summer Of Code, 2016. The aim of my project was to create a framework that can help people create a web based demo out of their machine learning code. We decided to break the project into two parts:

My commits to the repositories can be seen here and here. As this project was started from scratch, all of the commits to the repositories are mine.

The frontend webapp and backend lib talk to each other using a token provided by the webapp during registration of a new app by the user. The backend lib then exposes functions like cvfy.getTextArray and cvfy.sendTextArray to talk to the webapp for accepting user input and sending processed output to the demo page. All the documentation is available within the webapp itself.


I was not very confident that I would be able to complete this project in given time at first because there was some confusion about what was described in the project and what was supposed to be done. I would say that the project moulded itself considerably to become what it is today. I hope it will find audience among machine learning researchers and students. The UI and UX still remains to be polished a lot (no project is ever complete! :p) but it is in a prefectly usable state.

Difficulties I faced / Things I disliked:

  • Not having any design experience before, the most challenging part was finalising the UI. This project made me understand the importance of usability. What a developer may consider perfectly fine can be a nightmare to use for other people. It is really difficult to think out of the context of the application.
  • The project started off with being Github centric. There was a requirement to build the demo out of a user’s github repository. The problem here is, how do you streamline the pipeline and make it generalised, so that nearly all type of code can be used to make a demo? Well, upon working on the concept for some time, it turned out that it was a very difficult problem. I approached the issue using docker containers. Each repository adds a Dockerfile and a docker-compose.yml file to itself and CVFY reads those files to proceed with the deployment. I coded up everything and all things good, but it was very user unfriendly. You really cannot expect someone to learn using docker and docker-compose just to try out your project, and if they don’t try it out, why would they want to use it? This problem became more apparent when I added the option to run code outside of docker containers and just link the IP address and port of the service while configuring the application. This was a much neater solution and doesn’t require any beforehand knowhow of any tools. Considering this, the github part started looking like an unneccesary mess to me, and it brought down the development speed too, since I had to develop both workflows side-by-side. Eventually, I removed that code (commented out actually) from the main page and made it unreachable. Removing that dead code is still to be done.

Things I loved:

  • ReactJS and Redux. Period. I love it <3

Setup instructions for the project are given in cvfy-frontend README.

Note: Not all users have to do the setup themselves. A provider like CloudCV can run a hosted version of the service and others can use it, but self-hosting is also possible.

After the setup, the user is prompted with a first-run screen where he needs to put in the necessary details.

Mainscreen Mainscreen

First Run Setup First Run Setup

In-app documentation In-app documentation

After the first run setup, you can click on the “Create a demo” button on the homepage to go to signin with Github and go to your profile.

User profile page User profile page

Here you will see all the applications created by you and you can get a sharable link to your public profile to show all your demos in one place. Clicking on the “+” round button takes you to the app registration page where you fill in the details for your new app.

Registration page - Screen 1 Registration page - Screen 1

Registration page - Screen 2 Registration page - Screen 2

The token shown in the right side pane should be copied and this will be used in the backend lib (cvfy-lib) for cvfy.register function so that webapp and cvfy-lib can talk to each other.

Upon registering the app, the user is taken to the input component selection page where he gets to choose the kind (and number) of input his machine learning model accepts. Currently 3 input components are available:

  • Text Input
  • Text + Image Input
  • Image Input

Input Component selection page - Screen 1 Input Component selection page - Screen 1

Input Component selection page - Screen 2 Input Component selection page - Screen 2 The user can add any number of input fields.

Input Component selection page - Screen 3 Input Component selection page - Screen 3 Clicking on the “Preview” button on input component cards shows the preview of that component.

Clicking on “Save” button on the input component card takes you to the output component configuration page. Currently 6 output components are available:

  • Text Output
  • Image Output
  • Bar Graph Output
  • Scatter Graph Output
  • Pie Chart Output
  • Area Graph Output

Output Component selection page - Screen 1 Output Component selection page - Screen 1

Output Component selection page - Screen 2 Output Component selection page - Screen 2 The user can add any number of output fields.

Output Component selection page - Screen 3 Output Component selection page - Screen 3 Clicking on the “Preview” button on output component cards shows the preview of that component.

Output Component selection page - Screen 4 Output Component selection page - Screen 4 Clicking on the output component in the preview window expands the component for enhanced view.

Clicking on “Save” button on the output component card takes you to the Demo page.

Demo page - Screen 1 Demo page - Screen 1

Demo page - Screen 2 Demo page - Screen 2

Entering the input in input components and pressing the “Send” button sends the input imgae to the backend where cvfy-lib captures it with cvfy.getImageArray method and makes it available to the machine learning model. After processing the image, the model creates some metrics that are injected into the Area graph output component by using the cvfy.sendGraphArray method. The machine learning model also sends in additional data in the onscreen terminal attached to the demo page on the right side using cvfy.sendTextArray.

Backend code using cvfy-lib Backend code using cvfy-lib

Demo upon receiving output from the model - Screen 1 Demo upon receiving output from the model - Screen 1

Demo upon receiving output from the model - Screen 2 Demo upon receiving output from the model - Screen 2

Now you can go to your profile page and see your app available there.

Profile page Profile page

You can get the publicly sharable profile link by clicking the “Public Profile” button in the left sidebar.

Public Profile Page Public Profile Page

That’s it for this long-long post! I will continue contributing to this wonder project after the official GSoC period as well. Thanks to CloudCV for support during the program and Google for organising GSoC :)