GSTR (Gesture) is a web-based application that helps bridge the gap in non-verbal communication. Using Clarifai’s Custom Training technology, GSTR recognizes sign language from visual inputs and then gives feedback and translation information.
As with the spoken language, there are many different types of sign language being used around the world today. Each sign language has its own set of gestures that can be just as nuanced as different tones used in spoken languages. The best way to learn new languages is, of course, practice. GSTR is an application that can help you learn American sign language by providing feedback and translation on your gestures.
GSTR uses Clarifai through a custom visual recognition model that the GSTR team trained on sign language to recognize signs and gestures. It then responds in two different modes – educational and translation. Educational mode provides feedback confirming correct implementation of the sign. Translation mode gives the text and audio definition corresponding to the sign.
WHY WE ❤ IT
We love it when developers take advantage of our Custom Training technology, which is (shameless plug ahead) the only visual recognition product out there that lets devs teach new concepts to machine learning algorithms with less than 10 data examples and in less than a few seconds! Read more about the project on Devpost.
HOW YOU DO IT
We caught up with Fulton Garcia of the GSTR team to talk about the inspiration for GSTR. Fulton is currently studying Computer Science and Information Technology at the University of Central Florida and has a strong love for anything pug and/or tech related!
Clarifai: What inspired your idea for GSTR?
Fulton: We were inspired to work with people with disabilities, Sydney has a niece that is developmentally disabled and she has always wanted the opportunity to integrate her love for the community and her love for programming so we came together as a team to make that dream a reality.
How did you build the app?
GSTR was built on Node.js using Express and MongoDB for the backend. Clarifai for the image recognition and WebcamJs for capturing the images. Clarifai made it easy to create a model with multiple concepts and became accurate with the small amount of images we we’re able to collect at the hackathon.
What was the best part about working with the Clarifai API?
Clarifai is a very powerful tool that is very simple to use. It allows developers to easily implement image recognition into their projects even if they have very little experience. You can make great things with Clarifai – it’s just up to you to come up with creative ways to implement it!
Thanks for sharing, GSTR team!
To learn more, check out our documentation and sign-up for a free Clarifai account to start using our API – all it takes is three lines of code to get up and running! We’re super excited to share all the cool things built by our developer community, so don’t forget to tweet @Clarifai to show us your apps.
And give Fulton & the GSTR team some props in the comments below. Until next time!
