Whether you ate too much this Thanksgiving holiday, or just want to be more careful about what you eat in general, I'm here to show you a Clarifai visual recognition app that anyone can use to inform eating habits! I’ll show how it works and provide instructions for how you can take this and extend it as your own project in a few minutes and even fewer steps.
Take a look at the app in action!
This Clarifai-Food-Nutrition-Demo app is a prototype food nutrition analysis app that runs in a browser, made using Clarifai’s food recognition technology and some clever hacking by our good friends over at Major League Hacking.
The “front-end” (what a user interacts with) is a single web page for input/output where a user can upload an image of food. The app responds by displaying an estimate of the calorie content of the food. The "backend” (a small javascript file that runs locally in the web browser) comes down to a few lines of JavaScript that tie together API calls to Clarifai and Wolfram Alpha to look up nutrition tables.
Lucky for you, almost all the code is already written! You only need to copy the repository and update a few parameters according to my instructions, which will link your local code to your user accounts so you can make whatever customizations you want and have your very own Clarifai starter app.
Step 0: Free starter code!
Copy all available files in this Github repo to your computer. You can use the `git clone [url]` command below at your command-line interface to copy it into a project folder directly. Alternatively, you can download the files through your browser.
git clone https://github.com/MLH/clarifai-food-nutrition-demo.gitStep 1: Get Keys to Clarifai and create your Clarifai app
Log in to your Clarifai user account. If you don’t have one yet, go here, and scroll to the part that says “Get Your Free API Key” to create your free account. From there, you can now create your new app! For this app, change the Base Workflow to Food (a Workflow is one or more models). This way Clarifai knows that the predictions you want to get should come from our specially-trained Food Image Recognition model.
Be sure to copy the API Key that is auto-generated for you. You will use it in Step 3 as an identifying parameter in your code.
Step 2: Get Keys to Wolfram Alpha
Wolfram Alpha is a computational knowledge engine. Basically, it’s a really cool fact-finding service that computes answers to factual queries by referencing externally sourced curated data, including nutrition tables.
If you don’t have a Wolfram Alpha Developer Portal account, you can create one for free. All you need for this project is an AppID string, so click through to get one automatically assigned to you. You’ll copy the AppID in the next step as a parameter for your code.
Step 3: Insert the Keys
Have your Clarifai app API key and a Wolfram Alpha AppID? Open the predict.js file and insert those tokens into the file where indicated. It will look like the image below. You should replace the string values in quotes with the respective API tokens. Save the file when done. That’s all the code you need to change!
Step 4: Start it up!Now, simply open the file index.html to start the app in a web browser of your choice. If it’s working, you will see a webpage with an interface to submit a photo.
The repo you downloaded comes with some sample images for you to try out. See if you can get a nutritional breakdown of the french fries!
Extra credit: Try it on a photo of your favorite Thanksgiving dish and share it with us on Twitter by tagging @Clarifai! (Keep in mind calorie accuracy is limited by whatever nutrition data Wolfram Alpha has.)
Wow! How does this work?The magic is happening in our little ‘predict.js’ file in the “doPredict” function, containing the call to Clarifai’s Predict API:
This doPredict function takes a value (e.g. an image of fries) and sends it to Clarifai (the "app" object) with a call to our Predict API.
app.models.predict(Clarifai.FOOD_MODEL, value).then(function(response) { …
Clarifai responds to this API call with all kinds of prediction data about what it sees in the image. For example, the “french fries” concept is the top prediction result for our image of french fries. The name of the top result is saved as the `tag` variable to be used as the search term for Wolfram Alpha, which will search with those words for nutrition info in its connected databases. The Clarifai tag and the Wolfram Alpha nutrition results are then combined and returned to the browser so the user can see the results. Et voilà! Bon Appétit.
Want to make the nutrition predictions even better? You can make your Clarifai app better at recognizing what you see by training it on your own photos. The feature is called Custom Training; we’ve written up a handy How-to Walkthrough and Overview for you here.
It often surprises people how easy it is to make a working custom model with Clarifai:
- Clarifai can learn to recognize almost anything if you give it at least 10 examples (more will further improve prediction accuracy).
- The model training step is super quick (almost instantaneous in many cases) since we run everything on servers which automatically scale up according to need. Training completes orders of magnitude faster than it would when training a model on your personal machine, which enables you to rapidly iterate on your model without waiting.
- You don’t need to use code to upload and label images (if code is not your preference), since we have a drag-and-drop Explorer UI to make it intuitive to train models by dropping in your own images via your browser. The Explorer UI tool comes with every Clarifai app, including the one you just made! Look for a symbol of an eye next to each app in your Clarifai apps. Fun fact: The Clarifai team built the Explorer tool as an interface on top of the same open source Clarifai JavaScript client that we just used for the food nutrition app in this post.
Regardless of your relationship with food, don’t let calorie counting get you too down this holiday season! With our new app, you can not only eat, drink and be merry, but be an informed consumer as well. Cheers!
