Project 3 - WebAR - Be Your Own Hero


Sai Krishnan T N
Ramanathan Krishnamoorthy


For this final project we have decided to explore WebAR. WebAR is a new emerging area of web development that uses our web browser to provide an augmented reality experience. We all know AR and VR are the next big things in the world of technology and we also know how it has been evolving since the 90's. So integrating these technologies into the web browsers with the same immersive experience encourages the developers to explore more about the technology. WebAR is very easy to integrate with existing technologies like AJAX/Sockets and social media APIs for an interactive experience. Using other services like geolocation and social media we can create experiences tailored to the specific users age, interests or who they follow on social media. WebAR wil have a great opportunity to capture valuable detailed data (age, gender, geolocation, etc) on the users interacting with the application. But since its in early development phase, WebAR has its own disadvantages. Rendering detailed 3D objects in realtime at 60FPS is computationally too costly to execute and currently the GPU for hardware acceleration is not supported by the browser. Nevertheless, WebAR is a poweful technology whose future seems promising.

We have worked on various javascript libraries to implement Augmented Reality through the web browser. The project will run on web browsers which can access the user's webcam. We have used an open source machine learning library called tensorflow.js to track user body motion. There are also other libraries for other activities. Based on the user's motion, the application will augment graphics to the user's body parts. Following will be the features of the project. Following will be the features of the project.

1. The application will recognize the user standing in front of the camera. Then based on the costume selected it will augment a hat, mask, shirt, biceps and shorts on the user's body.
2. The user will be able to change the costume on the user by browsing through various designs ( There are 3 such theme based costumes ).
3. If the user is closer to the camera, the application will augment costumes related to the face and the user moves away from the camera other graphics related to body are augmented. When the user moves away from the camera, the accuracy of detected facial points reduces. So as a result, we are not augmenting the facial graphics when the user is far away from the camera.
4. The augmented graphics are also subject to scaling and rotation. That is, the augmented graphics rotate, scale and move accordingly based on the user's motion.
5. There are also three buttons augmented on the camera. The user can interact with these buttons through the left hand. One option is the play the music upon interating with the first button. The second button interaction will make the camera filled with snow flakes falling from top to bottom and similarly the third button with leaves.
6. User's gestures are also identified. Upon different postures by the user different activities take place. There are five such poses. Gesture 1 will clear all the augmented garaphics from the screen except the augmented body graphics. Gesture 2 is for changing the costume. Gesture 3 is for the removing the costume. Gesture 4 will make a girl walk from left to right in the screen. Gesture 5 will play some video on the screen.
7. All these screen activities can be recorded and made into a gif that can be shared on social media.

Please find the link to the source code and the youtube video below

Libraries Used:

Tensorflow JS
Jquery JS
Bootstrap JS and CSS
Gif JS
Gif Worker JS
Modernizr JS


Sublime Text Editor / Web browser

Hardware / Software Requirement:

No special Hardware or Software is required for this. Application can be run on any desktop browser having access camera. This application will run on smartphones(mobile devices) too. But since there are too many activities involved in this application, the navigation is not smooth when the application is run via mobile devices. The area is also limited as far as mobile devices are concerned. So it is always recommended to run the application via the desktop browser.


1. Start the Application using running index.html file in Chrome browser. Application is well tested and works perfectly in Chrome Browser. Its not supported in Safari because of Safari's security reasons.
2. Allow the Chrome to access the webcam. You are free to choose any other web cam attached with the computer. This option is available in Chrome's Setting Page.
2a) The Application is made to detect a single person standing in front of the camera. There cannot be any other person standing behind in the background of the camera. So make sure there are no persons or pcitures of humans in the background. Application works best in bright environment, so make sure there is ample of amount of light with a bright background for best working.
3. Once the screen has loaded you will see set of options on the left Pane. Select one of the costume. Its is mandatory to select a costume before trying other features of the Application.
4. Now the application will augment some kind of costume based over the user. As mentioned before, The level of Augmentation depends on the tracking of the human body. If the application has properly detected the body parts, then those body parts will be augmennted.
5. Standing close to the camera will help focus on the face and it will augment images on the face.
6. Standing at medium distance will help augment on upper body and face.
7. Standing at far, will reduce the tracking of the face and the points detected are not so accurate. Hence only the body will be tracked.
8. Any costumes can be tried for any number of times at any point of time.
9. There are 3 virtual buttons augmented in the reality in one side of the view.
10. Use your left wrist to move over each of the button to activate those action. First Button plays a audio. Next one augments snow fall on the user. The last one will bring fall of leaves in the screen.
11. There are few gestures that are incorporated in the application. It detects these gestures of the user and does some actions.
1. Gesture one : Change Costume
2. Gesture two : Clear all costume
3. Gesture three : Animate a cat walk of a woman
4. Gesture four : Play a video
5. Gesture five : Clear All. It clears the actions of three virtual buttons, gesture three and four.

Gesture Details:

Gesture one : Change Costume
Gesture two : Clear all costume
Gesture three : Animate a cat walk of a woman
Gesture four : Play a video
Gesture five : Clear All. It clears the actions of three virtual buttons, gesture three and four.

Link Sources:

Costume 1 :
Hat -
SunGlasses -
Shirt -
Shorts -
Arms -

Costume 2 :
Hat -
SunGlasses -
Shirt -
Shorts -
Arms -

Costume 3 :
Hat -
SunGlasses -
Shirt -
Shorts -
Arms -

Other Images/GIFs Sources:
Girl Walking -
Snow flakes -
Falling leaves -
Audio -
Video -
Red Button -
Blue Button -
Sound Image -