Image Predictions Quick Start

This post is a detailed walkthrough of how to perform a simple image prediction using Clarifai's python API. Everything contained here is in the accompanying Jupyter notebook hosted on Github at the following links:

• Link to python notebook itself
• Link to python notebook in Google Colaboratory
• Link to page of all notebooks

Let's get started! If you want to jump straight into the notebook, the only change you need to make is to add your own PAT. Once that is in place you can run the whole thing to get your predictions.

Installing the Clarifai gRPC client and dependencies

First, we need to install Clarifai's gRPC software and dependencies. Clarifai's current version requires protobuf 3.20.3 as a minimum, but we can also upgrade protobuf to the latest version.

• os to access file system commands like listing the contends of a directory
• BytesIO to steam files as a stream of bytes to be processed
• skimage we'll run predict on the example images in skimage's data module
• matplotlib.pyplot to display the results as images
  PIL "pillow," an "image processing library" used to send images to Clarifai
• numpy to create a range of numbers

Initializing the Clarifai gRPC-based Client

In this step we just need to import all the relevant parts of the Clarifai packages and instantiate them to eventually create the connection with Clarifai's servers.

Setup the Authorization

Clarifai uses personal access tokens -PATs - to identify you as a user and let you access services.

We fill in the placeholder for PAT here

PAT = 'YOUR_PAT_HERE'

view raw 4-pat.py hosted with ❤ by GitHub

 

To create or find a PAT, in Clarifai Community, click on the circular icon for your username in the top right, select “Security”, and then create a PAT or copy an existing one if you’ve already created one. You can follow the below screenshots to see where to find and create PATs.

 

 

Now that we've setup the PAT, we also need to specify the application and account that owns the model we're going to use. Since we're choosing a model from Clarifai / Main, we use

 

	USER_ID = 'clarifai'
	APP_ID = 'main'
	metadata = (('authorization', 'Key ' + PAT),)
	userDataObject = resources_pb2.UserAppIDSet(user_id=USER_ID, app_id=APP_ID)

view raw 5-user_data.py hosted with ❤ by GitHub

Collecting Inputs (Images in this example)

Since we're running predict on multiple images in this example, we iterate through all the files in the skimage data module, then only keep the ones we have in the "descriptions" list. That's it! You can modify this notebook to pull in your own images and classes to try it yourself. Please send any questions our way; our community Slack channel can be found here. 

	# images in skimage to use
	descriptions = [
	"page",
	"chelsea",
	"astronaut",
	"rocket",
	"motorcycle_right",
	"camera",
	"horse",
	"coffee"
	]
	original_images = []
	images = []
	plt.figure(figsize=(16, 5))
	for filename in [filename for filename in os.listdir(skimage.data_dir) if filename.endswith(".png") or filename.endswith(".jpg")]:
	name = os.path.splitext(filename)[0]
	if name not in descriptions:
	continue
	image = Image.open(os.path.join(skimage.data_dir, filename)).convert("RGB")
	plt.subplot(2, 4, len(images) + 1)
	plt.imshow(image)
	plt.title(f"{filename}")
	plt.xticks([])
	plt.yticks([])
	original_images.append(image)
	images.append(image)
	plt.tight_layout()

view raw 6-inputs.py hosted with ❤ by GitHub

Preparing Inputs (Images in this example)

In this step we convert our images into a byte stream using BytesIO and the userDataObject called resources_pb2. 

	inputs = []
	for image in images:
	buffered = BytesIO()
	image.save(buffered, format="PNG")
	inputs.append(
	resources_pb2.Input(
	data=resources_pb2.Data(
	image=resources_pb2.Image(
	base64=buffered.getvalue()
	)
	)
	)
	)

view raw 7-prepare_inputs.py hosted with ❤ by GitHub

Make Predictions

Clarifai's General Model is a visual classifier for identifying a variety of concepts, common objects, etc. It is a great all-purpose solution for most visual recognition needs with industry-leading performance. It works on both images and videos.

	# Choose the general visual classifier
	MODEL_ID = 'general-image-recognition'
	post_model_outputs_response = stub.PostModelOutputs(
	service_pb2.PostModelOutputsRequest(
	user_app_id=userDataObject, # The userDataObject is created in the overview and is required when using a PAT
	model_id=MODEL_ID,
	inputs=inputs
	),
	metadata=metadata
	)
	if post_model_outputs_response.status.code != status_code_pb2.SUCCESS:
	print(post_model_outputs_response.status)
	raise Exception("Post model outputs failed, status: " + post_model_outputs_response.status.description)

view raw 8-predictions.py hosted with ❤ by GitHub

Generate Results

Here we plot diagrams that show the identified concepts compared to the probability of them appearing in the image.

	plt.figure(figsize=(16, 16))
	for i, (image, output) in enumerate(zip(original_images, post_model_outputs_response.outputs)):
	top_preds = {
	concept.name: concept.value for concept in output.data.concepts[:5]
	}
	plt.subplot(4, 4, 2 * i + 1)
	plt.imshow(image)
	plt.axis("off")
	plt.subplot(4, 4, 2 * i + 2)
	y = np.arange(len(top_preds))
	plt.grid()
	plt.barh(y, list(top_preds.values()))
	plt.gca().invert_yaxis()
	plt.gca().set_axisbelow(True)
	plt.yticks(y, list(top_preds.keys()))
	plt.xlabel("probability")
	plt.subplots_adjust(wspace=0.6)
	plt.show()

view raw 9-results.py hosted with ❤ by GitHub