Spooky SMS & AI with Twilio and Clarifai

Introduction

Last week, a few of us here at Clarifai created a "Spooky or Not" app for Clarifai’s October hack day (more on how we trained the model next week!). To let people try the model from their own mobile device without installing anything, I coded up an app that lets anyone text a photo to a phone number and get back results that say whether the photo is spooky along with what about it makes it spooky!

It’s magical! But not magic. I used basic Python and two services to make this happen:

• Clarifai allows us to programmatically identify and analyze images using either a general model or a custom model. In this case, I was checking for "spooky" concepts returned by the general model, such as those seen in the screenshot above.
• Twilio is a very useful set of services that software developers use to programmatically send and receive text messages, and perform other communication functions using web service APIs.

In the rest of this post I will show how I tied these two services together with a Python Flask to make a functioning application.

Don't speak dev? You can still try out Clarifai via our website UI and the Explorer tool, which allows you to train custom recognition models with a drag-and-drop interface. Sign up for a free Clarifai account.

Ok, let's go!

Step 0: Set Up a Python Environment (Recommended)

The first step for any Python project is setting up a new virtual environment or container to hold your code. This keeps your installations separate from each other between projects and will save you headaches dealing with package updates. I used Pyenv for this project, but virtualenv, pipenv, or even a Docker container can fulfill the purpose. For detailed guidance, Twilio offers an excellent tutorial resource on Python Flask dev environment setup.

Step 1: Install Python Packages

Flask will be our lightweight web server, Twilio’s Python library allows us to interact with the Twilio API, and Clarifai’s Python library allows us to interact with the Clarifai API. 

Enter the following commands at your terminal in your new environment to get these installed:

pip install flask
pip install twilio
pip install clarifai

Step 2: Set up Clarifai

Clarifai is our AI to detect and predict concept tags for an image. Clarifai recognizes thousands of diverse concepts and can be trained to recognize almost anything. For the hack project, our team trained a model to learn an extra "spooky" concept from images we found for Halloween.

Clarifai's API is free to use for up to 5,000 operations per month per account, which is more than enough for a hack like this. All we need is a unique API key to identify the exact Clarifai application we want to call.

1. Log in to your Clarifai account. Sign up here if you don’t have an account yet!
2. Create a new application using the default options, then copy its API Key.
3. In your Python environment, enter the following: 

   export CLARIFAI_API_KEY=*your API key*

This line of code exports (saves) your key to an environment variable so that the value can be referenced without requiring you to hardcode it into your program.

Create a new Python file called clarifai_tagger.py and add the following code:

This function takes an image (URL) and returns a list of concept tags predicted by the Clarifai model.

Test out the function by adding this line of code to print results for a test image:

print('\n'.join(get_concept_tags('https://clarifai.com/cms-assets/20180320225538/general-006.jpg')))

You can try it out with any image URL. Just type "python clarifai_tagger" to run it. Go ahead! Try it!

Step 3: Sign up for Twilio

Twilio makes it seamless and simple for our humble Python program to interact with communication technologies like phone networks.

For this app, Twilio takes care of managing incoming and outgoing text messages for us and allows us to deal with the SMS message content using an API. 

For setup, you need to complete two steps to start:

1. Create a Twilio account
2. Reserve a phone number that can receive SMS

We'll come back to the settings for your new phone number in Step 5 when we deploy.

Step 4: Set Up Flask Web Server

Flask is a lightweight framework for web development in Python. It's simple, fun to use, and easy to extend.

In our app folder, make an app.py file to be your main Python Flask web server file. We only need to create one "route" with a function that can take an image and return results from Clarifai's API.

You can find a copy of the file I used for this project as a GitHub gist.

The code allows us to receive POST web requests from Twilio with Flask. It includes a function to query the Clarifai API and send Twilio back a reply with the results.

Step 5: Deploy (Tunnel with ngrok)

Your Flask app will need to be visible from the internet in order for Twilio to send requests to it. ngrok is a great tool for making local programs publicly accessible.

Run ngrok with the following line to tunnel port 5000:

./ngrok http 5000

This line of code creates a public URL access point (with the ngrok domain) and links it to your localhost at port 5000 (where your Python Flask server is running).

Copy the url that ngrok creates for you and paste it into your Twilio phone number settings, as shown in the following images:

Your local machine running ngrok:

Twilio Settings with an ngrok address for when "a message comes in":

Step 6: Try it out!

Run your app:

python app.py

Once your server is running, text an image to the phone number you set up in Twilio. In a short while, you should receive a text in reply!

Wait... so how does this work again?

1. Twilio sends a POST request to the supplied ngrok url, which points to your machine and triggers the /sms route of your Flask app.
2. The sms_reply function is called.
3. The URL to the image in the text message is passed to the clarifai_tagger module
4. A request to the Clarifai API is made, and a response with keywords associated with our image is returned.
5. The /sms route responds to Twilio’s request telling Twilio to send a message back with the tags we received from the Clarifai API.

Cool, huh?

CONCLUSION:

Thanks for reading! I showed you how I used Clarifai with Twilio and Python to make our Scarefai hack project usable over SMS. If you followed along, we created a Python program that accepts SMS text messages to a phone number and, if an image is attached, retrieves and returns Clarifai's predictions directly back to the sender. 

You can go beyond this and do something clever with the predictions that Clarifai returns. For example, if the image contains fire, send an alert to the nearest fire station.

If you code up with something interesting and want to share with us, we'd love to hear about it. Check out our Developer Guide and sign up to get yourself some API keys!

Acknowledgments

This project inspired by a similar project by our friends at Twilio.