Getting Started

The Clarifai API offers image and video recognition as a service. Whether you have one image or billions, you are only steps away from using artificial intelligence to recognize your visual content.

The API is built around a simple idea. You send inputs (an image or video) to the service and it returns predictions.

The type of prediction is based on what model you run the input through. For example, if you run your input through the 'food' model, the predictions it returns will contain concepts that the 'food' model knows about. If you run your input through the 'color' model, it will return predictions about the dominant colors in your image.

inputs outputs

Before you get started, if you haven't created an account and received your free API key, please do so before proceeding with this guide. You can begin making API calls for free, a credit card is not required.

Please note that your account will be limited to 100 API calls until you verify your email address. After verification, you will receive the full amount of API calls under your plan.

All API access is over HTTPS, and accessed via the https://api.clarifai.com domain. The relative path prefix /v2/ indicates that we are currently using version 2 of the API.

In the below examples, we use single brackets {variable} to indicate that this is a variable you should replace with a real value.

API Clients

We recommend using a client library to access the API. We offer official clients in various languages. View our API Clients.

For REST documentation please see the cURL examples.

Client Installation Instructions

// The JavaScript client works in both Node.js and the browser.


// Install the client from NPM

npm install clarifai

// Require the client

const Clarifai = require('clarifai');

// initialize with your api key. This will also work in your browser via http://browserify.org/

const app = new Clarifai.App({
 apiKey: 'YOUR_API_KEY'
});

// You can also use the SDK by adding this script to your HTML:

<script type="text/javascript" src="https://sdk.clarifai.com/js/clarifai-latest.js"></script>

# Pip install the client:
# pip install clarifai


#The package will be accessible by importing clarifai:

from clarifai import rest
from clarifai.rest import ClarifaiApp

# The client takes the `API_KEY` you created in your Clarifai
# account. You can set these variables in your environment as:

# - `CLARIFAI_API_KEY`

app = ClarifaiApp()

// Our API client is hosted on jCenter, Maven Central, and JitPack.

///////////////////////////////////////////////////////////////////////////////
// Installation - via Gradle (recommended)
///////////////////////////////////////////////////////////////////////////////

// Add the client to your dependencies:
dependencies {
    compile 'com.clarifai.clarifai-api2:core:2.3.0'
}

// Make sure you have the Maven Central Repository in your Gradle File
repositories {
    mavenCentral()
}

///////////////////////////////////////////////////////////////////////////////
// Installation - via Maven
///////////////////////////////////////////////////////////////////////////////

/*
<!-- Add the client to your dependencies: -->
<dependency>
  <groupId>com.clarifai.clarifai-api2</groupId>
  <artifactId>core</artifactId>
  <version>2.3.0</version>
</dependency>
*/

///////////////////////////////////////////////////////////////////////////////
// Initialize client
///////////////////////////////////////////////////////////////////////////////

new ClarifaiBuilder("YOUR_API_KEY")
    .client(new OkHttpClient()) // OPTIONAL. Allows customization of OkHttp by the user
    .buildSync(); // or use .build() to get a Future<ClarifaiClient>

// Installation via CocoaPods - https://cocoapods.org

// 1. Create a new XCode project, or use a current one.

// 2. Add the following to your Podfile:

//  pod 'Clarifai'

// 3. Install dependencies and generate workspace.

//  pod install

// 4. Open the workspace in Xcode

//  open YOUR_PROJECT_NAME.xcworkspace

// 5. You are now able to import ClarifaiApp.h and any other classes you need!

  #import ClarifaiApp.h

// Note: if you are using Swift in your project, make sure to include use_frameworks! in your Podfile. Then import Clarifai as a module.

  import Clarifai

// Install cURL: https://curl.haxx.se/download.html

Status Codes

All of our API endpoints return back a status code and description. To see a full list of our API status codes and their descriptions you can visit thefollowing page.

Explorer

In addition to the API, Clarifai now offers a web site that allows you to preview your Clarifai applications. You can view all the inputs you have added, perform searches and train new models.

inputs outputs

You can access the Explorer here

For a step-by-step guide on building and training a custom model using Explorer, checkout this walkthrough.

Authentication

Note: API Keys have replaced Access Tokens/Client ID & Secret

API Keys will now be used to authorize your Clarifai applications. You can go to API Keys page to create a new key. This change will not break existing applications - Access Tokens will be supported until late 2017. You can find more information on our blog.

Authentication to the API is handled through API Keys. You can limit the scope of an API Key, which enables the key to perform very specific operations on a given app, keeping your app secure.

To create an API Key, you can head over to the Keys Dashboard, and specify a) which app you would like to access, and b) what scopes you want the key to have.

api keys

Authorize Applications

Select the application that you want to authorize using this key. An API Key cannot be used across multiple apps.

Key Description

Use this field to create a descriptive name for your API Key. As you start to create a lot of keys, you would need this field to keep track of what key is for what purpose. Note: this field is not used for any identification purposes.

Scopes

You will see a list of scopes that you can add on to your API Key for the app that was selected. We recommend generating different keys for very specific purposes - this is the best approach to keep your app secure from malicious attacks. Do NOT share your API Key with other users. Here's a list of scopes that your key can have:

scope

Note: API Keys do not expire. In case your API Key gets compromised, you should delete that key, and create a new one with the same scopes.

Authorize API Calls

After creating your API Key, you are ready to make API calls. If you are using a client, authentication will be handled for you. If you are using the REST API, you will need to add the Authorization header as described in the cURL example.

const app = new Clarifai.App({apiKey: 'YOUR_API_KEY'});

from clarifai.rest import ClarifaiApp

app = ClarifaiApp(api_key='YOUR_API_KEY')

new ClarifaiBuilder("YOUR_API_KEY").buildSync();

ClarifaiApp *app = [[ClarifaiApp alloc] initWithApiKey:@"YOUR_API_KEY"];

curl -X POST \
  -H 'Authorization: Key YOUR_API_KEY' \
  -H "Content-Type: application/json" \
  -d '

If the API Key does not have the required scope(s), you will receive one of the following responses:

  1. Rejected Request: if a large portion of the response requires a scope that is missing
  2. Redacted Response: if majority of the response has the required scope(s), a small portion of the response will be redacted to not reveal unwarranted information

Predict

The Predict API analyzes your images or videos and tells you what's inside of them.

The API will return a list of concepts with corresponding probabilities of how likely it is these concepts are contained within the image or video.

When you make a prediction through the API, you tell it what model to use. A model contains a group of concepts. A model will only 'see' the concepts it contains.

Images

Via URL

To get predictions for an input, you need to supply an image and the model you'd like to get predictions from. You can supply an image either with a publicly accessible URL or by directly sending bytes. You can send up to 128 images in one API call. You specify the model you'd like to use with the {model-id} parameter.

Below is an example of how you would send image URLs and receive back predictions from the general model.

You can learn all about the different public models available later in the guide.


app.models.predict(Clarifai.GENERAL_MODEL, "https://samples.clarifai.com/metro-north.jpg").then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);
from clarifai.rest import Image as ClImage

model = app.models.get('general-v1.3')
image = ClImage(url='https://samples.clarifai.com/metro-north.jpg')
model.predict([image])

client.getDefaultModels().generalModel().predict()
    .withInputs(ClarifaiInput.forImage("https://samples.clarifai.com/metro-north.jpg"))
    .executeSync();

ClarifaiImage *image = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/metro-north.jpg"];
[_app getModelByName:@"general-v1.3" completion:^(ClarifaiModel *model, NSError *error) {
    [model predictOnImages:@[image]
                completion:^(NSArray<ClarifaiSearchResult *> *outputs, NSError *error) {
                    NSLog(@"outputs: %@", outputs);
                }];
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/metro-north.jpg"
          }
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/models/aaa03c23b3724a16a56b629203edc62c/outputs
Show All
Response
{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "outputs": [
    {
      "id": "ea68cac87c304b28a8046557062f34a0",
      "status": {
        "code": 10000,
        "description": "Ok"
      },
      "created_at": "2016-11-22T16:50:25Z",
      "model": {
        "name": "general-v1.3",
        "id": "aaa03c23b3724a16a56b629203edc62c",
        "created_at": "2016-03-09T17:11:39Z",
        "app_id": null,
        "output_info": {
          "message": "Show output_info with: GET /models/{model_id}/output_info",
          "type": "concept"
        },
        "model_version": {
          "id": "aa9ca48295b37401f8af92ad1af0d91d",
          "created_at": "2016-07-13T01:19:12Z",
          "status": {
            "code": 21100,
            "description": "Model trained successfully"
          }
        }
      },
      "input": {
        "id": "ea68cac87c304b28a8046557062f34a0",
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/metro-north.jpg"
          }
        }
      },
      "data": {
        "concepts": [
          {
            "id": "ai_HLmqFqBf",
            "name": "train",
            "app_id": null,
            "value": 0.9989112
          },
          {
            "id": "ai_fvlBqXZR",
            "name": "railway",
            "app_id": null,
            "value": 0.9975532
          },
          {
            "id": "ai_Xxjc3MhT",
            "name": "transportation system",
            "app_id": null,
            "value": 0.9959158
          },
          {
            "id": "ai_6kTjGfF6",
            "name": "station",
            "app_id": null,
            "value": 0.992573
          },
          {
            "id": "ai_RRXLczch",
            "name": "locomotive",
            "app_id": null,
            "value": 0.992556
          },
          {
            "id": "ai_VRmbGVWh",
            "name": "travel",
            "app_id": null,
            "value": 0.98789215
          },
          {
            "id": "ai_SHNDcmJ3",
            "name": "subway system",
            "app_id": null,
            "value": 0.9816359
          },
          {
            "id": "ai_jlb9q33b",
            "name": "commuter",
            "app_id": null,
            "value": 0.9712483
          },
          {
            "id": "ai_46lGZ4Gm",
            "name": "railroad track",
            "app_id": null,
            "value": 0.9690325
          },
          {
            "id": "ai_tr0MBp64",
            "name": "traffic",
            "app_id": null,
            "value": 0.9687052
          },
          {
            "id": "ai_l4WckcJN",
            "name": "blur",
            "app_id": null,
            "value": 0.9667078
          },
          {
            "id": "ai_2gkfMDsM",
            "name": "platform",
            "app_id": null,
            "value": 0.9624243
          },
          {
            "id": "ai_CpFBRWzD",
            "name": "urban",
            "app_id": null,
            "value": 0.960752
          },
          {
            "id": "ai_786Zr311",
            "name": "no person",
            "app_id": null,
            "value": 0.95864904
          },
          {
            "id": "ai_6lhccv44",
            "name": "business",
            "app_id": null,
            "value": 0.95720303
          },
          {
            "id": "ai_971KsJkn",
            "name": "track",
            "app_id": null,
            "value": 0.9494642
          },
          {
            "id": "ai_WBQfVV0p",
            "name": "city",
            "app_id": null,
            "value": 0.94089437
          },
          {
            "id": "ai_dSCKh8xv",
            "name": "fast",
            "app_id": null,
            "value": 0.9399334
          },
          {
            "id": "ai_TZ3C79C6",
            "name": "road",
            "app_id": null,
            "value": 0.93121606
          },
          {
            "id": "ai_VSVscs9k",
            "name": "terminal",
            "app_id": null,
            "value": 0.9230834
          }
        ]
      }
    }
  ]
}

Via Bytes

Below is an example of how you would send the bytes of an image and receive back predictions from the general model.


app.models.predict(Clarifai.GENERAL_MODEL, {base64: "G7p3m95uAl..."}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);
from clarifai.rest import Image as ClImage

model = app.models.get('general-v1.3')
image = ClImage(file_obj=open('/home/user/image.jpeg', 'rb'))
model.predict([image])

client.getDefaultModels().generalModel().predict()
    .withInputs(ClarifaiInput.forImage(new File("/home/user/image.jpeg")))
    .executeSync();

UIImage *image = [UIImage imageNamed:@"dress.jpg"];
ClarifaiImage *clarifaiImage = [[ClarifaiImage alloc] initWithImage:image];
[_app getModelByName:@"general-v1.3" completion:^(ClarifaiModel *model, NSError *error) {
    [model predictOnImages:@[clarifaiImage]
                completion:^(NSArray<ClarifaiSearchResult *> *outputs, NSError *error) {
                    NSLog(@"outputs: %@", outputs);
                }];
}];

// Smaller files (195 KB or less)

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "base64": "'"$(base64 /home/user/image.jpeg)"'"
          }
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/models/aaa03c23b3724a16a56b629203edc62c/outputs

// Larger Files (Greater than 195 KB)

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d @- https://api.clarifai.com/v2/models/aaa03c23b3724a16a56b629203edc62c/outputs << FILEIN
  {
    "inputs": [
      {
        "data": {
          "image": {
            "base64": "$(base64 /home/user/image.png)"
          }
        }
      }
    ]
  }
FILEIN
Show All
Response
{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "outputs": [
    {
      "id": "e1cf385843b94c6791bbd9f2654db5c0",
      "status": {
        "code": 10000,
        "description": "Ok"
      },
      "created_at": "2016-11-22T16:59:23Z",
      "model": {
        "name": "general-v1.3",
        "id": "aaa03c23b3724a16a56b629203edc62c",
        "created_at": "2016-03-09T17:11:39Z",
        "app_id": null,
        "output_info": {
          "message": "Show output_info with: GET /models/{model_id}/output_info",
          "type": "concept"
        },
        "model_version": {
          "id": "aa9ca48295b37401f8af92ad1af0d91d",
          "created_at": "2016-07-13T01:19:12Z",
          "status": {
            "code": 21100,
            "description": "Model trained successfully"
          }
        }
      },
      "input": {
        "id": "e1cf385843b94c6791bbd9f2654db5c0",
        "data": {
          "image": {
            "url": "https://s3.amazonaws.com/clarifai-api/img/prod/b749af061d564b829fb816215f6dc832/e11c81745d6d42a78ef712236023df1c.jpeg"
          }
        }
      },
      "data": {
        "concepts": [
          {
            "id": "ai_l4WckcJN",
            "name": "blur",
            "app_id": null,
            "value": 0.9973569
          },
          {
            "id": "ai_786Zr311",
            "name": "no person",
            "app_id": null,
            "value": 0.98865616
          },
          {
            "id": "ai_JBPqff8z",
            "name": "art",
            "app_id": null,
            "value": 0.986006
          },
          {
            "id": "ai_5rD7vW4j",
            "name": "wallpaper",
            "app_id": null,
            "value": 0.9722556
          },
          {
            "id": "ai_sTjX6dqC",
            "name": "abstract",
            "app_id": null,
            "value": 0.96476805
          },
          {
            "id": "ai_Dm5GLXnB",
            "name": "illustration",
            "app_id": null,
            "value": 0.922542
          },
          {
            "id": "ai_5xjvC0Tj",
            "name": "background",
            "app_id": null,
            "value": 0.8775655
          },
          {
            "id": "ai_tBcWlsCp",
            "name": "nature",
            "app_id": null,
            "value": 0.87474406
          },
          {
            "id": "ai_rJGvwlP0",
            "name": "insubstantial",
            "app_id": null,
            "value": 0.8196385
          },
          {
            "id": "ai_2Bh4VMrb",
            "name": "artistic",
            "app_id": null,
            "value": 0.8142488
          },
          {
            "id": "ai_mKzmkKDG",
            "name": "Christmas",
            "app_id": null,
            "value": 0.7996079
          },
          {
            "id": "ai_RQccV41p",
            "name": "woman",
            "app_id": null,
            "value": 0.7955615
          },
          {
            "id": "ai_20SCBBZ0",
            "name": "vector",
            "app_id": null,
            "value": 0.7775099
          },
          {
            "id": "ai_4sJLn6nX",
            "name": "dark",
            "app_id": null,
            "value": 0.7715479
          },
          {
            "id": "ai_5Kp5FMJw",
            "name": "still life",
            "app_id": null,
            "value": 0.7657637
          },
          {
            "id": "ai_LM64MDHs",
            "name": "shining",
            "app_id": null,
            "value": 0.7542407
          },
          {
            "id": "ai_swtdphX8",
            "name": "love",
            "app_id": null,
            "value": 0.74926054
          },
          {
            "id": "ai_h45ZTxZl",
            "name": "square",
            "app_id": null,
            "value": 0.7449074
          },
          {
            "id": "ai_cMfj16kJ",
            "name": "design",
            "app_id": null,
            "value": 0.73926914
          },
          {
            "id": "ai_LxrzLJmf",
            "name": "bright",
            "app_id": null,
            "value": 0.73790145
          }
        ]
      }
    }
  ]
}

Videos

With a video input, the Predict API response will return a list of predicted concepts for every frame of a video. Video is processed at 1 frame per second. This means you will receive a list of concepts for every second of your video.

You can run Predict on your video using a select number of public models. The models that are currently supported are: Apparel, Food, General, NSFW, Travel, and Wedding. You make an API call by providing the {model-id} parameter and your data parameter is video instead of image.

Video Limits

The Predict API has limits to the length and size it can support. A video, uploaded through URL, can be anywhere up to 80MB in size or 10mins in length. When a video is sent through by bytes, the Predict API can support 10MB in size.

If your video exceeds the limits, please follow our tutorial on how to break up a large video into smaller components, and send those into the Video API. Otherwise, the processing will time out and you will receive an error response.

Via URL

Below is an example of how you would send video URLs and receive back predictions from the general model.

clarifai.models.predict(Clarifai.GENERAL_MODEL, 'https://samples.clarifai.com/beer.mp4', {video: true})
  .then(log)
  .catch(log);
from clarifai.rest import Video as ClVideo

model = app.models.get('general-v1.3')
video = ClVideo(url='https://samples.clarifai.com/beer.mp4')
model.predict([video])
ClarifaiResponse<List<ClarifaiOutput<Frame>>> frames = client.getDefaultModels().generalVideoModel().predict()
    .withInputs(ClarifaiInput.forVideo("https://samples.clarifai.com/beer.mp4"))
    .executeSync();
Objective-C client details coming soon

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "video": {
            "url": "https://samples.clarifai.com/beer.mp4"
          }
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/models/aaa03c23b3724a16a56b629203edc62c/outputs
Show All
Response{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "outputs": [
    {
      "id": "d8234da5d1f04ca8a2e13e34d51f9b85",
      "status": {
        "code": 10000,
        "description": "Ok"
      },
      "created_at": "2017-06-28T14:58:41.835370141Z",
      "model": {
        "id": "aaa03c23b3724a16a56b629203edc62c",
        "name": "general-v1.3",
        "created_at": "2016-03-09T17:11:39.608845Z",
        "app_id": "main",
        "output_info": {
          "message": "Show output_info with: GET /models/{model_id}/output_info",
          "type": "concept",
          "type_ext": "concept"
        },
        "model_version": {
          "id": "aa9ca48295b37401f8af92ad1af0d91d",
          "created_at": "2016-07-13T01:19:12.147644Z",
          "status": {
            "code": 21100,
            "description": "Model trained successfully"
          }
        }
      },
      "input": {
        "id": "f0fc1a005f124d389da4d80823a3125b",
        "data": {
          "video": {
            "url": "https://samples.clarifai.com/beer.mp4"
          }
        }
      },
      "data": {
        "frames": [
          {
            "frame_info": {
              "index": 0,
              "time": 0
            },
            "data": {
              "concepts": [
                {
                  "id": "ai_zJx6RbxW",
                  "name": "drink",
                  "value": 0.98658466,
                  "app_id": "main"
                },
                {
                  "id": "ai_mCpQg89c",
                  "name": "glass",
                  "value": 0.97975093,
                  "app_id": "main"
                },
                {
                  "id": "ai_drK6ClJR",
                  "name": "alcohol",
                  "value": 0.9783862,
                  "app_id": "main"
                },
                {
                  "id": "ai_8XGJjH7R",
                  "name": "foam",
                  "value": 0.97157896,
                  "app_id": "main"
                },
                {
                  "id": "ai_TBlp0Pt3",
                  "name": "beer",
                  "value": 0.969543,
                  "app_id": "main"
                },
                {
                  "id": "ai_SsmKLB4z",
                  "name": "bar",
                  "value": 0.96628696,
                  "app_id": "main"
                },
                {
                  "id": "ai_5VHsZr8N",
                  "name": "liquid",
                  "value": 0.95581007,
                  "app_id": "main"
                },
                {
                  "id": "ai_Lq00FggW",
                  "name": "desktop",
                  "value": 0.92861253,
                  "app_id": "main"
                },
                {
                  "id": "ai_7vR9zv7l",
                  "name": "bubble",
                  "value": 0.9082134,
                  "app_id": "main"
                },
                {
                  "id": "ai_B3MXt5Ng",
                  "name": "refreshment",
                  "value": 0.9020835,
                  "app_id": "main"
                },
                {
                  "id": "ai_7Xg5SQRW",
                  "name": "luxury",
                  "value": 0.8990605,
                  "app_id": "main"
                },
                {
                  "id": "ai_786Zr311",
                  "name": "no person",
                  "value": 0.89708906,
                  "app_id": "main"
                },
                {
                  "id": "ai_3R5pJ6hB",
                  "name": "lager",
                  "value": 0.8938055,
                  "app_id": "main"
                },
                {
                  "id": "ai_7qwGxLch",
                  "name": "gold",
                  "value": 0.8892093,
                  "app_id": "main"
                },
                {
                  "id": "ai_wmbvr5TG",
                  "name": "celebration",
                  "value": 0.88606626,
                  "app_id": "main"
                },
                {
                  "id": "ai_4lvjn8qv",
                  "name": "closeup",
                  "value": 0.881963,
                  "app_id": "main"
                },
                {
                  "id": "ai_pkvDRSJ1",
                  "name": "mug",
                  "value": 0.8674431,
                  "app_id": "main"
                },
                {
                  "id": "ai_12dz73B9",
                  "name": "bottle",
                  "value": 0.86288416,
                  "app_id": "main"
                },
                {
                  "id": "ai_zFnPQdgB",
                  "name": "wood",
                  "value": 0.86252767,
                  "app_id": "main"
                },
                {
                  "id": "ai_8LWlDfFD",
                  "name": "table",
                  "value": 0.86069393,
                  "app_id": "main"
                }
              ]
            }
          },
          {
            "frame_info": {
              "index": 1,
              "time": 1000
            },
            "data": {
              "concepts": [
                {
                  "id": "ai_zJx6RbxW",
                  "name": "drink",
                  "value": 0.98658466,
                  "app_id": "main"
                },
                {
                  "id": "ai_mCpQg89c",
                  "name": "glass",
                  "value": 0.97975093,
                  "app_id": "main"
                },
                {
                  "id": "ai_drK6ClJR",
                  "name": "alcohol",
                  "value": 0.9783862,
                  "app_id": "main"
                },
                {
                  "id": "ai_8XGJjH7R",
                  "name": "foam",
                  "value": 0.97157896,
                  "app_id": "main"
                },
                {
                  "id": "ai_TBlp0Pt3",
                  "name": "beer",
                  "value": 0.969543,
                  "app_id": "main"
                },
                {
                  "id": "ai_SsmKLB4z",
                  "name": "bar",
                  "value": 0.96628696,
                  "app_id": "main"
                },
                {
                  "id": "ai_5VHsZr8N",
                  "name": "liquid",
                  "value": 0.95581007,
                  "app_id": "main"
                },
                {
                  "id": "ai_7vR9zv7l",
                  "name": "bubble",
                  "value": 0.9082134,
                  "app_id": "main"
                },
                {
                  "id": "ai_B3MXt5Ng",
                  "name": "refreshment",
                  "value": 0.9020835,
                  "app_id": "main"
                },
                {
                  "id": "ai_786Zr311",
                  "name": "no person",
                  "value": 0.89708906,
                  "app_id": "main"
                },
                {
                  "id": "ai_3R5pJ6hB",
                  "name": "lager",
                  "value": 0.8938055,
                  "app_id": "main"
                },
                {
                  "id": "ai_7qwGxLch",
                  "name": "gold",
                  "value": 0.8834515,
                  "app_id": "main"
                },
                {
                  "id": "ai_pkvDRSJ1",
                  "name": "mug",
                  "value": 0.8674431,
                  "app_id": "main"
                },
                {
                  "id": "ai_b01mhdxB",
                  "name": "party",
                  "value": 0.8603341,
                  "app_id": "main"
                },
                {
                  "id": "ai_XNmzgDnF",
                  "name": "pub",
                  "value": 0.85809004,
                  "app_id": "main"
                },
                {
                  "id": "ai_2gmKZLxp",
                  "name": "cold",
                  "value": 0.85319245,
                  "app_id": "main"
                },
                {
                  "id": "ai_Lq00FggW",
                  "name": "desktop",
                  "value": 0.8506696,
                  "app_id": "main"
                },
                {
                  "id": "ai_54zxXFGL",
                  "name": "full",
                  "value": 0.84634554,
                  "app_id": "main"
                },
                {
                  "id": "ai_zFnPQdgB",
                  "name": "wood",
                  "value": 0.8446485,
                  "app_id": "main"
                },
                {
                  "id": "ai_wmbvr5TG",
                  "name": "celebration",
                  "value": 0.8383831,
                  "app_id": "main"
                }
              ]
            }
          },
          {
            "frame_info": {
              "index": 2,
              "time": 2000
            },
            "data": {
              "concepts": [
                {
                  "id": "ai_zJx6RbxW",
                  "name": "drink",
                  "value": 0.9856042,
                  "app_id": "main"
                },
                {
                  "id": "ai_mCpQg89c",
                  "name": "glass",
                  "value": 0.97975093,
                  "app_id": "main"
                },
                {
                  "id": "ai_8XGJjH7R",
                  "name": "foam",
                  "value": 0.9755833,
                  "app_id": "main"
                },
                {
                  "id": "ai_786Zr311",
                  "name": "no person",
                  "value": 0.9733174,
                  "app_id": "main"
                },
                {
                  "id": "ai_TBlp0Pt3",
                  "name": "beer",
                  "value": 0.969543,
                  "app_id": "main"
                },
                {
                  "id": "ai_SsmKLB4z",
                  "name": "bar",
                  "value": 0.94170487,
                  "app_id": "main"
                },
                {
                  "id": "ai_5VHsZr8N",
                  "name": "liquid",
                  "value": 0.92778283,
                  "app_id": "main"
                },
                {
                  "id": "ai_2gmKZLxp",
                  "name": "cold",
                  "value": 0.9227257,
                  "app_id": "main"
                },
                {
                  "id": "ai_3PlgVmlN",
                  "name": "food",
                  "value": 0.9179274,
                  "app_id": "main"
                },
                {
                  "id": "ai_drK6ClJR",
                  "name": "alcohol",
                  "value": 0.90887475,
                  "app_id": "main"
                },
                {
                  "id": "ai_B3MXt5Ng",
                  "name": "refreshment",
                  "value": 0.9045203,
                  "app_id": "main"
                },
                {
                  "id": "ai_3R5pJ6hB",
                  "name": "lager",
                  "value": 0.8938055,
                  "app_id": "main"
                },
                {
                  "id": "ai_WbwL0pPL",
                  "name": "breakfast",
                  "value": 0.87420183,
                  "app_id": "main"
                },
                {
                  "id": "ai_54zxXFGL",
                  "name": "full",
                  "value": 0.8699659,
                  "app_id": "main"
                },
                {
                  "id": "ai_pkvDRSJ1",
                  "name": "mug",
                  "value": 0.8674431,
                  "app_id": "main"
                },
                {
                  "id": "ai_XNmzgDnF",
                  "name": "pub",
                  "value": 0.85809004,
                  "app_id": "main"
                },
                {
                  "id": "ai_Lq00FggW",
                  "name": "desktop",
                  "value": 0.8506696,
                  "app_id": "main"
                },
                {
                  "id": "ai_zFnPQdgB",
                  "name": "wood",
                  "value": 0.8446485,
                  "app_id": "main"
                },
                {
                  "id": "ai_qNxqNBWN",
                  "name": "cream",
                  "value": 0.844169,
                  "app_id": "main"
                },
                {
                  "id": "ai_7D0mdp1W",
                  "name": "delicious",
                  "value": 0.8397074,
                  "app_id": "main"
                }
              ]
            }
          },
          {
            "frame_info": {
              "index": 3,
              "time": 3000
            },
            "data": {
              "concepts": [
                {
                  "id": "ai_8XGJjH7R",
                  "name": "foam",
                  "value": 0.996614,
                  "app_id": "main"
                },
                {
                  "id": "ai_zJx6RbxW",
                  "name": "drink",
                  "value": 0.9794438,
                  "app_id": "main"
                },
                {
                  "id": "ai_786Zr311",
                  "name": "no person",
                  "value": 0.9733174,
                  "app_id": "main"
                },
                {
                  "id": "ai_TBlp0Pt3",
                  "name": "beer",
                  "value": 0.9645849,
                  "app_id": "main"
                },
                {
                  "id": "ai_mCpQg89c",
                  "name": "glass",
                  "value": 0.94761443,
                  "app_id": "main"
                },
                {
                  "id": "ai_pkvDRSJ1",
                  "name": "mug",
                  "value": 0.92864025,
                  "app_id": "main"
                },
                {
                  "id": "ai_2gmKZLxp",
                  "name": "cold",
                  "value": 0.9227257,
                  "app_id": "main"
                },
                {
                  "id": "ai_5VHsZr8N",
                  "name": "liquid",
                  "value": 0.91797745,
                  "app_id": "main"
                },
                {
                  "id": "ai_3PlgVmlN",
                  "name": "food",
                  "value": 0.9179274,
                  "app_id": "main"
                },
                {
                  "id": "ai_WbwL0pPL",
                  "name": "breakfast",
                  "value": 0.904904,
                  "app_id": "main"
                },
                {
                  "id": "ai_B3MXt5Ng",
                  "name": "refreshment",
                  "value": 0.9045203,
                  "app_id": "main"
                },
                {
                  "id": "ai_54zxXFGL",
                  "name": "full",
                  "value": 0.889248,
                  "app_id": "main"
                },
                {
                  "id": "ai_BrnHNkt0",
                  "name": "coffee",
                  "value": 0.8689867,
                  "app_id": "main"
                },
                {
                  "id": "ai_7D0mdp1W",
                  "name": "delicious",
                  "value": 0.86591685,
                  "app_id": "main"
                },
                {
                  "id": "ai_SsmKLB4z",
                  "name": "bar",
                  "value": 0.8546975,
                  "app_id": "main"
                },
                {
                  "id": "ai_mZ2tl6cW",
                  "name": "health",
                  "value": 0.8544879,
                  "app_id": "main"
                },
                {
                  "id": "ai_cHsR7RS8",
                  "name": "milk",
                  "value": 0.852397,
                  "app_id": "main"
                },
                {
                  "id": "ai_zFnPQdgB",
                  "name": "wood",
                  "value": 0.8446485,
                  "app_id": "main"
                },
                {
                  "id": "ai_qNxqNBWN",
                  "name": "cream",
                  "value": 0.844169,
                  "app_id": "main"
                },
                {
                  "id": "ai_8LWlDfFD",
                  "name": "table",
                  "value": 0.837438,
                  "app_id": "main"
                }
              ]
            }
          },
          {
            "frame_info": {
              "index": 4,
              "time": 4000
            },
            "data": {
              "concepts": [
                {
                  "id": "ai_8XGJjH7R",
                  "name": "foam",
                  "value": 0.996614,
                  "app_id": "main"
                },
                {
                  "id": "ai_786Zr311",
                  "name": "no person",
                  "value": 0.9748836,
                  "app_id": "main"
                },
                {
                  "id": "ai_TBlp0Pt3",
                  "name": "beer",
                  "value": 0.9645849,
                  "app_id": "main"
                },
                {
                  "id": "ai_zJx6RbxW",
                  "name": "drink",
                  "value": 0.96217895,
                  "app_id": "main"
                },
                {
                  "id": "ai_3PlgVmlN",
                  "name": "food",
                  "value": 0.9179274,
                  "app_id": "main"
                },
                {
                  "id": "ai_WbwL0pPL",
                  "name": "breakfast",
                  "value": 0.904904,
                  "app_id": "main"
                },
                {
                  "id": "ai_B3MXt5Ng",
                  "name": "refreshment",
                  "value": 0.9045203,
                  "app_id": "main"
                },
                {
                  "id": "ai_2gmKZLxp",
                  "name": "cold",
                  "value": 0.9030821,
                  "app_id": "main"
                },
                {
                  "id": "ai_mCpQg89c",
                  "name": "glass",
                  "value": 0.8983356,
                  "app_id": "main"
                },
                {
                  "id": "ai_pkvDRSJ1",
                  "name": "mug",
                  "value": 0.894987,
                  "app_id": "main"
                },
                {
                  "id": "ai_7D0mdp1W",
                  "name": "delicious",
                  "value": 0.894392,
                  "app_id": "main"
                },
                {
                  "id": "ai_54zxXFGL",
                  "name": "full",
                  "value": 0.889248,
                  "app_id": "main"
                },
                {
                  "id": "ai_mZ2tl6cW",
                  "name": "health",
                  "value": 0.8860091,
                  "app_id": "main"
                },
                {
                  "id": "ai_4sJLn6nX",
                  "name": "dark",
                  "value": 0.8837219,
                  "app_id": "main"
                },
                {
                  "id": "ai_zFnPQdgB",
                  "name": "wood",
                  "value": 0.8712319,
                  "app_id": "main"
                },
                {
                  "id": "ai_BrnHNkt0",
                  "name": "coffee",
                  "value": 0.8695712,
                  "app_id": "main"
                },
                {
                  "id": "ai_8LWlDfFD",
                  "name": "table",
                  "value": 0.8664293,
                  "app_id": "main"
                },
                {
                  "id": "ai_SsmKLB4z",
                  "name": "bar",
                  "value": 0.8546975,
                  "app_id": "main"
                },
                {
                  "id": "ai_cHsR7RS8",
                  "name": "milk",
                  "value": 0.852397,
                  "app_id": "main"
                },
                {
                  "id": "ai_qNxqNBWN",
                  "name": "cream",
                  "value": 0.844169,
                  "app_id": "main"
                }
              ]
            }
          },
          {
            "frame_info": {
              "index": 5,
              "time": 5000
            },
            "data": {
              "concepts": [
                {
                  "id": "ai_8XGJjH7R",
                  "name": "foam",
                  "value": 0.997158,
                  "app_id": "main"
                },
                {
                  "id": "ai_TBlp0Pt3",
                  "name": "beer",
                  "value": 0.97719705,
                  "app_id": "main"
                },
                {
                  "id": "ai_786Zr311",
                  "name": "no person",
                  "value": 0.9748836,
                  "app_id": "main"
                },
                {
                  "id": "ai_zJx6RbxW",
                  "name": "drink",
                  "value": 0.96217895,
                  "app_id": "main"
                },
                {
                  "id": "ai_pkvDRSJ1",
                  "name": "mug",
                  "value": 0.9210669,
                  "app_id": "main"
                },
                {
                  "id": "ai_3PlgVmlN",
                  "name": "food",
                  "value": 0.9124408,
                  "app_id": "main"
                },
                {
                  "id": "ai_54zxXFGL",
                  "name": "full",
                  "value": 0.90667903,
                  "app_id": "main"
                },
                {
                  "id": "ai_WbwL0pPL",
                  "name": "breakfast",
                  "value": 0.904904,
                  "app_id": "main"
                },
                {
                  "id": "ai_B3MXt5Ng",
                  "name": "refreshment",
                  "value": 0.9046865,
                  "app_id": "main"
                },
                {
                  "id": "ai_2gmKZLxp",
                  "name": "cold",
                  "value": 0.9030821,
                  "app_id": "main"
                },
                {
                  "id": "ai_BrnHNkt0",
                  "name": "coffee",
                  "value": 0.90262496,
                  "app_id": "main"
                },
                {
                  "id": "ai_mCpQg89c",
                  "name": "glass",
                  "value": 0.8983356,
                  "app_id": "main"
                },
                {
                  "id": "ai_4sJLn6nX",
                  "name": "dark",
                  "value": 0.8947689,
                  "app_id": "main"
                },
                {
                  "id": "ai_7D0mdp1W",
                  "name": "delicious",
                  "value": 0.894392,
                  "app_id": "main"
                },
                {
                  "id": "ai_SsmKLB4z",
                  "name": "bar",
                  "value": 0.8852426,
                  "app_id": "main"
                },
                {
                  "id": "ai_3R5pJ6hB",
                  "name": "lager",
                  "value": 0.8789175,
                  "app_id": "main"
                },
                {
                  "id": "ai_8LWlDfFD",
                  "name": "table",
                  "value": 0.8762902,
                  "app_id": "main"
                },
                {
                  "id": "ai_zFnPQdgB",
                  "name": "wood",
                  "value": 0.87279737,
                  "app_id": "main"
                },
                {
                  "id": "ai_mZ2tl6cW",
                  "name": "health",
                  "value": 0.8544879,
                  "app_id": "main"
                },
                {
                  "id": "ai_cHsR7RS8",
                  "name": "milk",
                  "value": 0.849733,
                  "app_id": "main"
                }
              ]
            }
          },
          {
            "frame_info": {
              "index": 6,
              "time": 6000
            },
            "data": {
              "concepts": [
                {
                  "id": "ai_8XGJjH7R",
                  "name": "foam",
                  "value": 0.99790645,
                  "app_id": "main"
                },
                {
                  "id": "ai_TBlp0Pt3",
                  "name": "beer",
                  "value": 0.97817445,
                  "app_id": "main"
                },
                {
                  "id": "ai_786Zr311",
                  "name": "no person",
                  "value": 0.97463626,
                  "app_id": "main"
                },
                {
                  "id": "ai_zJx6RbxW",
                  "name": "drink",
                  "value": 0.9659773,
                  "app_id": "main"
                },
                {
                  "id": "ai_pkvDRSJ1",
                  "name": "mug",
                  "value": 0.9273318,
                  "app_id": "main"
                },
                {
                  "id": "ai_4sJLn6nX",
                  "name": "dark",
                  "value": 0.9219268,
                  "app_id": "main"
                },
                {
                  "id": "ai_B3MXt5Ng",
                  "name": "refreshment",
                  "value": 0.9185593,
                  "app_id": "main"
                },
                {
                  "id": "ai_2gmKZLxp",
                  "name": "cold",
                  "value": 0.91295856,
                  "app_id": "main"
                },
                {
                  "id": "ai_3PlgVmlN",
                  "name": "food",
                  "value": 0.9119204,
                  "app_id": "main"
                },
                {
                  "id": "ai_54zxXFGL",
                  "name": "full",
                  "value": 0.91089505,
                  "app_id": "main"
                },
                {
                  "id": "ai_SsmKLB4z",
                  "name": "bar",
                  "value": 0.9056676,
                  "app_id": "main"
                },
                {
                  "id": "ai_BrnHNkt0",
                  "name": "coffee",
                  "value": 0.90262496,
                  "app_id": "main"
                },
                {
                  "id": "ai_mCpQg89c",
                  "name": "glass",
                  "value": 0.89882934,
                  "app_id": "main"
                },
                {
                  "id": "ai_WbwL0pPL",
                  "name": "breakfast",
                  "value": 0.8932399,
                  "app_id": "main"
                },
                {
                  "id": "ai_7D0mdp1W",
                  "name": "delicious",
                  "value": 0.892028,
                  "app_id": "main"
                },
                {
                  "id": "ai_zFnPQdgB",
                  "name": "wood",
                  "value": 0.88797945,
                  "app_id": "main"
                },
                {
                  "id": "ai_3R5pJ6hB",
                  "name": "lager",
                  "value": 0.88745904,
                  "app_id": "main"
                },
                {
                  "id": "ai_8LWlDfFD",
                  "name": "table",
                  "value": 0.87949455,
                  "app_id": "main"
                },
                {
                  "id": "ai_MmRdqDFp",
                  "name": "soap",
                  "value": 0.87376094,
                  "app_id": "main"
                },
                {
                  "id": "ai_5VHsZr8N",
                  "name": "liquid",
                  "value": 0.8715329,
                  "app_id": "main"
                }
              ]
            }
          },
          {
            "frame_info": {
              "index": 7,
              "time": 7000
            },
            "data": {
              "concepts": [
                {
                  "id": "ai_8XGJjH7R",
                  "name": "foam",
                  "value": 0.99790645,
                  "app_id": "main"
                },
                {
                  "id": "ai_TBlp0Pt3",
                  "name": "beer",
                  "value": 0.97817445,
                  "app_id": "main"
                },
                {
                  "id": "ai_786Zr311",
                  "name": "no person",
                  "value": 0.97463626,
                  "app_id": "main"
                },
                {
                  "id": "ai_zJx6RbxW",
                  "name": "drink",
                  "value": 0.9659773,
                  "app_id": "main"
                },
                {
                  "id": "ai_pkvDRSJ1",
                  "name": "mug",
                  "value": 0.9273318,
                  "app_id": "main"
                },
                {
                  "id": "ai_4sJLn6nX",
                  "name": "dark",
                  "value": 0.9219268,
                  "app_id": "main"
                },
                {
                  "id": "ai_B3MXt5Ng",
                  "name": "refreshment",
                  "value": 0.9185593,
                  "app_id": "main"
                },
                {
                  "id": "ai_2gmKZLxp",
                  "name": "cold",
                  "value": 0.91295856,
                  "app_id": "main"
                },
                {
                  "id": "ai_3PlgVmlN",
                  "name": "food",
                  "value": 0.9119204,
                  "app_id": "main"
                },
                {
                  "id": "ai_54zxXFGL",
                  "name": "full",
                  "value": 0.91089505,
                  "app_id": "main"
                },
                {
                  "id": "ai_SsmKLB4z",
                  "name": "bar",
                  "value": 0.9056676,
                  "app_id": "main"
                },
                {
                  "id": "ai_BrnHNkt0",
                  "name": "coffee",
                  "value": 0.90262496,
                  "app_id": "main"
                },
                {
                  "id": "ai_mCpQg89c",
                  "name": "glass",
                  "value": 0.89882934,
                  "app_id": "main"
                },
                {
                  "id": "ai_WbwL0pPL",
                  "name": "breakfast",
                  "value": 0.8932399,
                  "app_id": "main"
                },
                {
                  "id": "ai_7D0mdp1W",
                  "name": "delicious",
                  "value": 0.892028,
                  "app_id": "main"
                },
                {
                  "id": "ai_zFnPQdgB",
                  "name": "wood",
                  "value": 0.8913312,
                  "app_id": "main"
                },
                {
                  "id": "ai_3R5pJ6hB",
                  "name": "lager",
                  "value": 0.88745904,
                  "app_id": "main"
                },
                {
                  "id": "ai_8LWlDfFD",
                  "name": "table",
                  "value": 0.87949455,
                  "app_id": "main"
                },
                {
                  "id": "ai_MmRdqDFp",
                  "name": "soap",
                  "value": 0.87376094,
                  "app_id": "main"
                },
                {
                  "id": "ai_5VHsZr8N",
                  "name": "liquid",
                  "value": 0.8715329,
                  "app_id": "main"
                }
              ]
            }
          },
          {
            "frame_info": {
              "index": 8,
              "time": 8000
            },
            "data": {
              "concepts": [
                {
                  "id": "ai_8XGJjH7R",
                  "name": "foam",
                  "value": 0.99790645,
                  "app_id": "main"
                },
                {
                  "id": "ai_TBlp0Pt3",
                  "name": "beer",
                  "value": 0.97817445,
                  "app_id": "main"
                },
                {
                  "id": "ai_786Zr311",
                  "name": "no person",
                  "value": 0.97463626,
                  "app_id": "main"
                },
                {
                  "id": "ai_zJx6RbxW",
                  "name": "drink",
                  "value": 0.9659773,
                  "app_id": "main"
                },
                {
                  "id": "ai_4sJLn6nX",
                  "name": "dark",
                  "value": 0.9219268,
                  "app_id": "main"
                },
                {
                  "id": "ai_pkvDRSJ1",
                  "name": "mug",
                  "value": 0.9210669,
                  "app_id": "main"
                },
                {
                  "id": "ai_B3MXt5Ng",
                  "name": "refreshment",
                  "value": 0.9185593,
                  "app_id": "main"
                },
                {
                  "id": "ai_2gmKZLxp",
                  "name": "cold",
                  "value": 0.91295856,
                  "app_id": "main"
                },
                {
                  "id": "ai_3PlgVmlN",
                  "name": "food",
                  "value": 0.9119204,
                  "app_id": "main"
                },
                {
                  "id": "ai_54zxXFGL",
                  "name": "full",
                  "value": 0.91089505,
                  "app_id": "main"
                },
                {
                  "id": "ai_SsmKLB4z",
                  "name": "bar",
                  "value": 0.9056676,
                  "app_id": "main"
                },
                {
                  "id": "ai_BrnHNkt0",
                  "name": "coffee",
                  "value": 0.90262496,
                  "app_id": "main"
                },
                {
                  "id": "ai_mCpQg89c",
                  "name": "glass",
                  "value": 0.89882934,
                  "app_id": "main"
                },
                {
                  "id": "ai_7D0mdp1W",
                  "name": "delicious",
                  "value": 0.894392,
                  "app_id": "main"
                },
                {
                  "id": "ai_WbwL0pPL",
                  "name": "breakfast",
                  "value": 0.8932399,
                  "app_id": "main"
                },
                {
                  "id": "ai_zFnPQdgB",
                  "name": "wood",
                  "value": 0.88797945,
                  "app_id": "main"
                },
                {
                  "id": "ai_3R5pJ6hB",
                  "name": "lager",
                  "value": 0.88745904,
                  "app_id": "main"
                },
                {
                  "id": "ai_8LWlDfFD",
                  "name": "table",
                  "value": 0.87949455,
                  "app_id": "main"
                },
                {
                  "id": "ai_MmRdqDFp",
                  "name": "soap",
                  "value": 0.87376094,
                  "app_id": "main"
                },
                {
                  "id": "ai_5VHsZr8N",
                  "name": "liquid",
                  "value": 0.8715329,
                  "app_id": "main"
                }
              ]
            }
          }
        ]
      }
    }
  ]
}

Via Bytes

Below is an example of how you would send the bytes of a video and receive back predictions from the general model.

clarifai.models.predict(Clarifai.GENERAL_MODEL, {base64: 'AAAAIGZ...'}, {video: true})
  .then(log)
  .catch(log);
from clarifai.rest import Video as ClVideo

model = app.models.get('general-v1.3')
video = ClVideo(filename='/home/user/video.mp4')
model.predict([video])
ClarifaiResponse<List<ClarifaiOutput<Frame>>> frames = client.getDefaultModels().generalVideoModel().predict()
    .withInputs(ClarifaiInput.forVideo(new File("/home/user/video.mp4)")))
    .executeSync();
Objective-C client details coming soon
curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "video": {
            "base64": "'"$(base64 /home/user/video.mp4)"'"
          }
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/models/aaa03c23b3724a16a56b629203edc62c/outputs
Show All
Response{
    "status": {
        "code": 10000,
        "description": "Ok"
    },
    "outputs": [
        {
            "id": "f6f9e1b007d742fb9d777f35cf3bffd0",
            "status": {
                "code": 10000,
                "description": "Ok"
            },
            "created_at": "2017-06-28T16:00:51.258194418Z",
            "model": {
                "id": "aaa03c23b3724a16a56b629203edc62c",
                "name": "general-v1.3",
                "created_at": "2016-03-09T17:11:39.608845Z",
                "app_id": "main",
                "output_info": {
                    "message": "Show output_info with: GET /models/{model_id}/output_info",
                    "type": "concept",
                    "type_ext": "concept"
                },
                "model_version": {
                    "id": "aa9ca48295b37401f8af92ad1af0d91d",
                    "created_at": "2016-07-13T01:19:12.147644Z",
                    "status": {
                        "code": 21100,
                        "description": "Model trained successfully"
                    }
                }
            },
            "input": {
                "id": "b17d29ad1b714869a8c729e510ab22d0",
                "data": {
                    "video": {
                        "url": "https://s3.amazonaws.com/clarifai-api/vid/prod/ib81c84d5b2341858b86da18a2bd21d2/e86fbf516521425098081dd42e157a12",
                        "base64": "true"
                    }
                }
            },
            "data": {
                "frames": [
                    {
                        "frame_info": {
                            "index": 0,
                            "time": 0
                        },
                        "data": {
                            "concepts": [
                                {
                                    "id": "ai_VTlCx2f2",
                                    "name": "window",
                                    "value": 0.99909437,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_l8TKp2h5",
                                    "name": "people",
                                    "value": 0.99610686,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_VPmHr5bm",
                                    "name": "adult",
                                    "value": 0.9958472,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_SVshtN54",
                                    "name": "one",
                                    "value": 0.9937376,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_TJ9wFfK5",
                                    "name": "portrait",
                                    "value": 0.9899301,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_ZrPNDjxN",
                                    "name": "daylight",
                                    "value": 0.9885398,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_dxSG2s86",
                                    "name": "man",
                                    "value": 0.9833108,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_86sS08Pw",
                                    "name": "wear",
                                    "value": 0.9807093,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_GxSDhQ34",
                                    "name": "facial expression",
                                    "value": 0.9769263,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_Pf2b7clG",
                                    "name": "indoors",
                                    "value": 0.96838474,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_VRmbGVWh",
                                    "name": "travel",
                                    "value": 0.96641624,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_XNzGRk0F",
                                    "name": "side view",
                                    "value": 0.9603646,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_Zmhsv0Ch",
                                    "name": "outdoors",
                                    "value": 0.9434113,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_n9vjC1jB",
                                    "name": "light",
                                    "value": 0.94182396,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_WcnFrjw1",
                                    "name": "backlit",
                                    "value": 0.9347838,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_QKqjh1CM",
                                    "name": "vehicle window",
                                    "value": 0.92699903,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_mlrv94tv",
                                    "name": "reflection",
                                    "value": 0.90993655,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_41s912fX",
                                    "name": "fair weather",
                                    "value": 0.90100014,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_rsX6XWc2",
                                    "name": "building",
                                    "value": 0.88111985,
                                    "app_id": "main"
                                },
                                {
                                    "id": "ai_L83krFdq",
                                    "name": "veil",
                                    "value": 0.8785704,
                                    "app_id": "main"
                                }
                            ]
                        }
                    }
                ]
            }
        }
    ]
}

Note: The Search API currently is only available for images.

A common case for using Clarifai is to get the concepts predicted in an image and then use those concepts to power search.

The Search API allows you to send images (url or bytes) to the service and have them indexed by 'general' model concepts and their visual representations. Once indexed, you can search for images by concept or by image.

Image illustrating how to search by images using Clarifai's concepts

Add Images to Search Index

To get started with search, you must first add images to the search index. You can add one or more images to the index at a time. You can supply an image either with a publicly accessible URL or by directly sending image bytes. You can send up to 128 images in one API call.


app.inputs.create([
  {url: "https://samples.clarifai.com/metro-north.jpg"},
  {url: "https://samples.clarifai.com/wedding.jpg"},
  {base64: "G7p3m95uAl..."}
]).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# add image from image url
app.inputs.create_image_from_url("https://samples.clarifai.com/metro-north.jpg")

# add image from image filename
app.inputs.create_image_from_filename("local/file.jpg")

# add image from raw image bytes
raw_bytes = open("local/file.jpg", "rb").read()
app.inputs.create_image_from_bytes(raw_bytes)

# add image from base64 encoded image bytes
raw_bytes = open("local/file.jpg", "rb").read()
base64_bytes = base64.b64encode(raw_bytes)
app.inputs.create_image_from_base64(base64_bytes)

client.addInputs()
    .plus(
        ClarifaiInput.forImage("https://samples.clarifai.com/metro-north.jpg"),
        ClarifaiInput.forImage("https://samples.clarifai.com/wedding.jpg")
    )
    .executeSync();

ClarifaiImage *image1 = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/metro-north.jpg"];
ClarifaiImage *image2 = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/wedding.jpg"];

[app addInputs:@[image1, image2] completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
    NSLog(@"inputs: %@", inputs);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/metro-north.jpg"
          }
        }
      },
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/wedding.jpg"
          }
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/inputs
Show All
Response
{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "inputs": [
    {
      "id": "edc70c917475499abdc7151f41d6cf3e",
      "created_at": "2016-11-22T17:06:02Z",
      "data": {
        "image": {
          "url": "https://samples.clarifai.com/metro-north.jpg"
        }
      },
      "status": {
        "code": 30001,
        "description": "Download pending"
      }
    },
    {
      "id": "f96ca3bbf02041c59addcc13e3468b7d",
      "created_at": "2016-11-22T17:06:02Z",
      "data": {
        "image": {
          "url": "https://samples.clarifai.com/wedding.jpg"
        }
      },
      "status": {
        "code": 30001,
        "description": "Download pending"
      }
    }
  ]
}

Search By Concept

Once your images are indexed, you can search for them by concept.


app.inputs.search({ concept: {name: 'people'} }).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# search by public concept
app.inputs.search_by_predicted_concepts(concept='people')

# search by a list of concepts
app.inputs.search_by_predicted_concepts(concepts=['people'])

# search by concept id
app.inputs.search_by_predicted_concepts(concept_id='ai_dP13sXL4')

# search by a list of concept ids
app.inputs.search_by_predicted_concepts(concept_ids=['ai_dP13sXL4'])

client.searchInputs(SearchClause.matchConcept(Concept.forName("people")))
    .getPage(1)
    .executeSync();

// First create a search term with a concept you want to search.
ClarifaiConcept *conceptFromGeneralModel = [[ClarifaiConcept alloc] initWithConceptName:@"people"];
ClarifaiSearchTerm *searchTerm = [ClarifaiSearchTerm searchByPredictedConcept:conceptFromGeneralModel];

[app search:@[searchTerm] page:@1 perPage:@20 completion:^(NSArray<ClarifaiSearchResult *> *results, NSError *error) {
  // Print output of first search result.
  NSLog(@"inputID: %@", results[0].inputID);
  NSLog(@"URL: %@", results[0].mediaURL);
  NSLog(@"probability of public concept: %@", results[0].score);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "query": {
      "ands": [
        {
          "output": {
            "data": {
              "concepts": [
                {
                  "name": "people"
                }
              ]
            }
          }
        }
      ]
    }
  }'\
  https://api.clarifai.com/v2/searches
Show All
Response
{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "hits": [
    {
      "score": 0.98155165,
      "input": {
        "id": "f96ca3bbf02041c59addcc13e3468b7d",
        "created_at": "2016-11-22T17:06:02Z",
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/wedding.jpg"
          }
        },
        "status": {
          "code": 30000,
          "description": "Download complete"
        }
      }
    }
  ]
}

Search By Image

You can also search for images using another image. In this case, you provide an image (url or bytes) and the results will return all the images in your search index that are visually similar to the one provided.


app.inputs.search({ input: {url: 'https://samples.clarifai.com/puppy.jpeg'} }).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# search by image url
app.inputs.search_by_image(url="https://samples.clarifai.com/metro-north.jpg")

# search by existing input id
input_id = "some_existing_input_id"
app.inputs.search_by_image(image_id=input_id)

# search by raw bytes
data = "image_raw_bytes"
app.inputs.search_by_image(imgbytes=data)

# search by base64 bytes
base64_data = "image_bytes_encoded_in_base64"
app.inputs.search_by_image(base64bytes=base64_data)

# search by local filename
filename="filename_on_local_disk.jpg"
app.inputs.search_by_image(filename=filename)

# search from fileio
fio = open("filename_on_local_disk.jpg", 'rb')
app.inputs.search_by_image(fileobj=fio)

client.searchInputs(SearchClause.matchImageVisually(ClarifaiImage.of("https://samples.clarifai.com/metro-north.jpg")))
    .getPage(1)
    .executeSync();

ClarifaiSearchTerm *searchTerm = [ClarifaiSearchTerm searchVisuallyWithImageURL:@"https://samples.clarifai.com/metro-north.jpg"];

[app search:@[searchTerm] page:@1 perPage:@20 completion:^(NSArray<ClarifaiSearchResult *> *results, NSError *error) {
  // Print output of first search result.
  NSLog(@"inputID: %@", results[0].inputID);
  NSLog(@"URL: %@", results[0].mediaURL);
  NSLog(@"probability of visual similarity: %@", results[0].score);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "query": {
      "ands": [
        {
          "output": {
            "input": {
              "data": {
                "image": {
                  "url": "https://samples.clarifai.com/metro-north.jpg"
                }
              }
            }
          }
        }
      ]
    }
  }'\
  https://api.clarifai.com/v2/searches
Show All
Response
{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "hits": [
    {
      "score": 0.9999997,
      "input": {
        "id": "edc70c917475499abdc7151f41d6cf3e",
        "created_at": "2016-11-22T17:06:02Z",
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/metro-north.jpg"
          }
        },
        "status": {
          "code": 30000,
          "description": "Download complete"
        }
      }
    },
    {
      "score": 0.3915897,
      "input": {
        "id": "f96ca3bbf02041c59addcc13e3468b7d",
        "created_at": "2016-11-22T17:06:02Z",
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/wedding.jpg"
          }
        },
        "status": {
          "code": 30000,
          "description": "Download complete"
        }
      }
    }
  ]
}

Train

Clarifai provides many different models that 'see' the world differently. A model contains a group of concepts. A model will only see the concepts it contains.

There are times when you wish you had a model that sees the world the way you see it. The API allows you to do this. You can create your own model and train it with your own images and concepts. Once you train it to see how you would like it to see, you can then use that model to make predictions.

You do not need many images to get started. We recommend starting with 10 and adding more as needed. Before you train your first model you will have needed to create an application. From there you will be able to change your Base Workflow to optimize custom training using the knowledge base from select public models.

inputs outputs

Add Images With Concepts

To get started training your own model, you must first add images that already contain the concepts you want your model to see.


app.inputs.create({
  url: "https://samples.clarifai.com/puppy.jpeg",
  concepts: [
    {
      id: "boscoe",
      value: true
    }
  ]
});

app.inputs.create_image_from_url(url="https://samples.clarifai.com/puppy.jpeg", concepts=['boscoe'])

client.addInputs()
    .plus(
        ClarifaiInput.forImage("https://samples.clarifai.com/puppy.jpeg")
            .withConcepts(Concept.forID("boscoe"))
    )
    .executeSync();

ClarifaiImage *image = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/puppy.jpeg" andConcepts:@"cute puppy"];
[_app addInputs:@[image] completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
    NSLog(@"inputs: %@", inputs);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/puppy.jpeg"
          },
          "concepts":[
            {
              "id": "boscoe",
              "value": true
            }
          ]
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/inputs
Show All
Response
{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "inputs": [
    {
      "id": "e82fd13b11354d808cc48dc8f94ec3a9",
      "created_at": "2016-11-22T17:16:00Z",
      "data": {
        "image": {
          "url": "https://samples.clarifai.com/puppy.jpeg"
        },
        "concepts": [
          {
            "id": "boscoe",
            "name": "boscoe",
            "app_id": "f09abb8a57c041cbb94759ebb0cf1b0d",
            "value": 1
          }
        ]
      },
      "status": {
        "code": 30000,
        "description": "Download complete"
      }
    }
  ]
}

Create A Model

Once your images with concepts are added, you are now ready to create the model. You'll need a name for the model and you'll also need to provide it with the concepts you added above.

Take note of the model id that is returned in the response. You'll need that for the next two steps.


app.models.create(
  "pets",
  [
    { "id": "boscoe" }
  ]
).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

model = app.models.create('pets', concepts=['boscoe'])

client.createModel("pets")
    .withOutputInfo(ConceptOutputInfo.forConcepts(
        Concept.forID("boscoe")
    ))
    .executeSync();

[app createModel:@[concept] name:modelName conceptsMutuallyExclusive:NO closedEnvironment:NO
      completion:^(ClarifaiModel *model, NSError *error) {
        NSLog(@"model: %@", model);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "model": {
      "id": "pets",
      "output_info": {
        "data": {
          "concepts": [
            {
              "id": "boscoe"
            }
          ]
        },
        "output_config": {
          "concepts_mutually_exclusive": false,
          "closed_environment":false
        }
      }
    }
  }'\
  https://api.clarifai.com/v2/models
Show All
Response
{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "model": {
    "name": "pets",
    "id": "a10f0cf48cf3426cbb8c4805e246c214",
    "created_at": "2016-11-22T17:17:36Z",
    "app_id": "f09abb8a57c041cbb94759ebb0cf1b0d",
    "output_info": {
      "message": "Show output_info with: GET /models/{model_id}/output_info",
      "type": "concept",
      "output_config": {
        "concepts_mutually_exclusive": false,
        "closed_environment": false
      }
    },
    "model_version": {
      "id": "e7bcd534b61b4874a3ab69fba974c012",
      "created_at": "2016-11-22T17:17:36Z",
      "status": {
        "code": 21102,
        "description": "Model not yet trained"
      }
    }
  }
}

Train The Model

Now that you've added images with concepts, then created a model with those concepts, the next step is to train the model. When you train a model, you are telling the system to look at all the images with concepts you've provided and learn from them. This train operation is asynchronous. It may take a few seconds for your model to be fully trained and ready.


app.models.train("{model_id}").then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

// or if you have an instance of a model

model.train().then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

model = app.models.get('{model_id}')
model.train()

client.trainModel("{model_id}").executeSync();

ClarifaiImage *image = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/puppy.jpeg"]
[app getModel:@"{id}" completion:^(ClarifaiModel *model, NSError *error) {
    [model train:^(ClarifaiModel *model, NSError *error) {
        NSLog(@"model: %@", model);
    }];
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  https://api.clarifai.com/v2/models/{model_id}/versions
Show All
Response
{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "model": {
    "name": "pets",
    "id": "a10f0cf48cf3426cbb8c4805e246c214",
    "created_at": "2016-11-22T17:17:36Z",
    "app_id": "f09abb8a57c041cbb94759ebb0cf1b0d",
    "output_info": {
      "message": "Show output_info with: GET /models/{model_id}/output_info",
      "type": "concept",
      "output_config": {
        "concepts_mutually_exclusive": false,
        "closed_environment": false
      }
    },
    "model_version": {
      "id": "d1b38fd2251148d08675c5542ef00c7b",
      "created_at": "2016-11-22T17:21:13Z",
      "status": {
        "code": 21103,
        "description": "Custom model is currently in queue for training, waiting on inputs to process."
      }
    }
  }
}

Predict With The Model

Now the moment you've been waiting for. First you added images with concepts. Then you created a model with those concepts. Then you trained the model on those images. Now you are ready to use your new model to get predictions. The predictions returned will only contain the concepts that you told it to see.

Note: you can repeat the above steps as often as you like. By adding more images with concepts and training, you can get the model to predict exactly how you want it to.


app.models.predict("{model_id}", ["https://samples.clarifai.com/puppy.jpeg"]).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

// or if you have an instance of a model

model.predict("https://samples.clarifai.com/puppy.jpeg").then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);
from clarifai.rest import Image as ClImage

model = app.models.get('{model_id}')

image = ClImage(url='https://samples.clarifai.com/puppy.jpeg')
model.predict([image])

client.predict("{model_id}")
    .withInputs(
        ClarifaiInput.forImage("https://samples.clarifai.com/puppy.jpeg")
    )
    .executeSync();

ClarifaiImage *image = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/puppy.jpeg"]
[app getModel:@"{id}" completion:^(ClarifaiModel *model, NSError *error) {
    [model predictOnImages:@[image]
                completion:^(NSArray<ClarifaiSearchResult *> *outputs, NSError *error) {
                    NSLog(@"outputs: %@", outputs);
                }];
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/puppy.jpeg"
          }
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/models/{model_id}/outputs
Show All
Response
{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "outputs": [
    {
      "id": "e8b6eb27de764f3fa8d4f7752a3a2dfc",
      "status": {
        "code": 10000,
        "description": "Ok"
      },
      "created_at": "2016-11-22T17:22:23Z",
      "model": {
        "name": "pets",
        "id": "a10f0cf48cf3426cbb8c4805e246c214",
        "created_at": "2016-11-22T17:17:36Z",
        "app_id": "f09abb8a57c041cbb94759ebb0cf1b0d",
        "output_info": {
          "message": "Show output_info with: GET /models/{model_id}/output_info",
          "type": "concept",
          "output_config": {
            "concepts_mutually_exclusive": false,
            "closed_environment": false
          }
        },
        "model_version": {
          "id": "d1b38fd2251148d08675c5542ef00c7b",
          "created_at": "2016-11-22T17:21:13Z",
          "status": {
            "code": 21100,
            "description": "Model trained successfully"
          }
        }
      },
      "input": {
        "id": "e8b6eb27de764f3fa8d4f7752a3a2dfc",
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/puppy.jpeg"
          }
        }
      },
      "data": {
        "concepts": [
          {
            "id": "boscoe",
            "name": "boscoe",
            "app_id": "f09abb8a57c041cbb94759ebb0cf1b0d",
            "value": 0.98308545
          }
        ]
      }
    }
  ]
}

Workflow

Workflows are a new entity added to Clarifai, which encompass one or more Public or Custom model(s). Every workflow is attached to one of your applications. Under each workflow, you will see a list of the public models and all custom models in that application when selecting models to add to your workflow. With Workflow Predict, you will be able to reduce the latency and in turn make your product more efficient.

Note: this won't have any impact on the price you are charged per call. You will still be charged for the same operation if it were separate calls to the API.

Workflow Setup

To set up a workflow, you will need to head over to the Applications page through your account. From there, you will need to select which application you want to create the workflow under.

Image showing the top-level Applications page on
the Clarifai Developer website

Then under that application, you will see a section labeled "Workflows" and a button to "Create Workflow".

Image showing My First Application and the Create Workflow button underneath the
Create API Key

After that, the page will reveal a new workflow form to fill out. Fill out the Workflow ID field, this will be used to make the API call, so make sure to give it something URL friendly! Included there, you will also a list that consists of a model field and a version associated with it. For the public models, you will be mandated to use the latest version. For your custom models, you will be able to select the version of your model. To add another model, you will just click underneath your latest addition on the "Add Model". The max limit of models associated with any given workflow is 5 models. If you would like to remove a model, there is a large X that will allow you to remove a model. Once you have finished adding everything, press the "Save Workflow" button and that will save the state of your workflow. Now you are ready to predict using your brand new workflow. You can edit a given workflow at any time, in case you don't like it.

Image showing a list of models (Moderation and General)
under a workflow with the name my-workflow

Workflow Predict

The Workflow Predict API allows you to predict using 1 or more model(s), regardless of them being public or custom, within a single API call! The max number of inputs processed at once with any given workflow is 32.

Now that you have that all set up, you will be able to predict under a workflow using the POST /v2/workflow/{workflow_id}/results endpoint. Your {workflow-id} currently is whatever you set as your ID. Then as far as your request body, nothing has changed with how you would normally do a predict. In the response body, you will see a results object and each object will be the response from the models in the same ordering from the workflow you set up.

Image showing the Explorer's workflow prediction results

You can also use the Explorer to see the results of your workflow's predictions on a given input.

app.workflow.predict('{workflow-id}', "https://samples.clarifai.com/metro-north.jpg").then(
    function(response){
      // Do something with response
    },
    function(err){
      // There was an error
    }
);
from clarifai.rest import Image as ClImage

workflow = app.workflows.get('{workflow-id}')
image = ClImage(url='https://samples.clarifai.com/metro-north.jpg')
workflow.predict([image])
client.workflowPredict("{workflow-id}")
        .withInputs(ClarifaiInput.forImage("https://samples.clarifai.com/metro-north.jpg"))
        .executeSync();
// Coming Soon
curl -X POST \
  -H 'authorization: Key YOUR_API_KEY' \
  -H 'content-type: application/json' \
  -d '{
    "inputs": [
        {
          "data": {
            "image": {
              "url": "https://samples.clarifai.com/metro-north.jpg"
          }
        }
      }  
    ]
}'\
https://api.clarifai.com/v2/workflows/{workflow-id}/results
Show All
Response{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "workflow": {
    "id": "my-workflow",
    "app_id": "c54b7637df12407aa9c57dfd6d5c057f",
    "created_at": "2017-07-10T01:45:05.672880Z"
  },
  "results": [
    {
      "status": {
        "code": 10000,
        "description": "Ok"
      },
      "input": {
        "id": "c88aeed9d04c471cace6f8e4801f1a1c",
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/metro-north.jpg"
          }
        }
      },
      "outputs": [
        {
          "id": "feae971167a04d1bbebb7ea49d6ba0f7",
          "status": {
            "code": 10000,
            "description": "Ok"
          },
          "created_at": "2017-07-10T12:01:44.929928529Z",
          "model": {
            "id": "d16f390eb32cad478c7ae150069bd2c6",
            "name": "moderation",
            "created_at": "2017-05-12T21:28:00.471607Z",
            "app_id": "main",
            "output_info": {
              "message": "Show output_info with: GET /models/{model_id}/output_info",
              "type": "concept",
              "type_ext": "concept"
            },
            "model_version": {
              "id": "b42ac907ac93483484483a0040a386be",
              "created_at": "2017-05-12T21:28:00.471607Z",
              "status": {
                "code": 21100,
                "description": "Model trained successfully"
              }
            }
          },
          "data": {
            "concepts": [
              {
                "id": "ai_QD1zClSd",
                "name": "safe",
                "value": 0.99999714,
                "app_id": "main"
              },
              {
                "id": "ai_kBBGf7r8",
                "name": "gore",
                "value": 3.7771046e-05,
                "app_id": "main"
              },
              {
                "id": "ai_8QQwMjQR",
                "name": "drug",
                "value": 1.0449563e-05,
                "app_id": "main"
              },
              {
                "id": "ai_V76bvrtj",
                "name": "explicit",
                "value": 5.2887003e-06,
                "app_id": "main"
              },
              {
                "id": "ai_RtXh5qkR",
                "name": "suggestive",
                "value": 4.7939684e-06,
                "app_id": "main"
              }
            ]
          }
        },
        {
          "id": "f635b40cbeee47ddb7b348a981e14faf",
          "status": {
            "code": 10000,
            "description": "Ok"
          },
          "created_at": "2017-07-10T12:01:44.929941126Z",
          "model": {
            "id": "aaa03c23b3724a16a56b629203edc62c",
            "name": "general-v1.3",
            "created_at": "2016-02-26T23:38:40.086101Z",
            "app_id": "main",
            "output_info": {
              "message": "Show output_info with: GET /models/{model_id}/output_info",
              "type": "concept",
              "type_ext": "concept"
            },
            "model_version": {
              "id": "aa9ca48295b37401f8af92ad1af0d91d",
              "created_at": "2016-07-13T00:58:55.915745Z",
              "status": {
                "code": 21100,
                "description": "Model trained successfully"
              }
            }
          },
          "data": {
            "concepts": [
              {
                "id": "ai_HLmqFqBf",
                "name": "train",
                "value": 0.9989112,
                "app_id": "main"
              },
              {
                "id": "ai_fvlBqXZR",
                "name": "railway",
                "value": 0.9975532,
                "app_id": "main"
              },
              {
                "id": "ai_Xxjc3MhT",
                "name": "transportation system",
                "value": 0.9959158,
                "app_id": "main"
              },
              {
                "id": "ai_6kTjGfF6",
                "name": "station",
                "value": 0.992573,
                "app_id": "main"
              },
              {
                "id": "ai_RRXLczch",
                "name": "locomotive",
                "value": 0.992556,
                "app_id": "main"
              },
              {
                "id": "ai_VRmbGVWh",
                "name": "travel",
                "value": 0.98789215,
                "app_id": "main"
              },
              {
                "id": "ai_SHNDcmJ3",
                "name": "subway system",
                "value": 0.9816359,
                "app_id": "main"
              },
              {
                "id": "ai_jlb9q33b",
                "name": "commuter",
                "value": 0.9712483,
                "app_id": "main"
              },
              {
                "id": "ai_46lGZ4Gm",
                "name": "railroad track",
                "value": 0.9690325,
                "app_id": "main"
              },
              {
                "id": "ai_tr0MBp64",
                "name": "traffic",
                "value": 0.9687052,
                "app_id": "main"
              },
              {
                "id": "ai_l4WckcJN",
                "name": "blur",
                "value": 0.9667078,
                "app_id": "main"
              },
              {
                "id": "ai_2gkfMDsM",
                "name": "platform",
                "value": 0.9624243,
                "app_id": "main"
              },
              {
                "id": "ai_CpFBRWzD",
                "name": "urban",
                "value": 0.960752,
                "app_id": "main"
              },
              {
                "id": "ai_786Zr311",
                "name": "no person",
                "value": 0.95864904,
                "app_id": "main"
              },
              {
                "id": "ai_6lhccv44",
                "name": "business",
                "value": 0.95720303,
                "app_id": "main"
              },
              {
                "id": "ai_971KsJkn",
                "name": "track",
                "value": 0.9494642,
                "app_id": "main"
              },
              {
                "id": "ai_WBQfVV0p",
                "name": "city",
                "value": 0.94089437,
                "app_id": "main"
              },
              {
                "id": "ai_dSCKh8xv",
                "name": "fast",
                "value": 0.9399334,
                "app_id": "main"
              },
              {
                "id": "ai_TZ3C79C6",
                "name": "road",
                "value": 0.93121606,
                "app_id": "main"
              },
              {
                "id": "ai_VSVscs9k",
                "name": "terminal",
                "value": 0.9230834,
                "app_id": "main"
              }
            ]
          }
        }
      ]
    }
  ]
}

Evaluate

Now that you've successfully trained the model, you may want to test its performance before using it in the production environment. The Model Evaluation tool allows you to perform a cross validation on a specified model version. Once the evaluation is complete, you’ll be able to view various metrics that will inform the model’s performance.

This evaluate operation is asynchronous and is currently available on the Explorer only.

How It Works

Model Evaluation performs a 5-split cross validation on data you used to train your custom model. cross validation In the cross validation process, it will:

  1. Set aside a random 20% subset of the training data and designate as a test set,
  2. Train a new model with the remaining 80% of the training data,
  3. Pass the test set data through this new model to make predictions,
  4. Compare the predictions against the test set’s actual labels, and
  5. Repeat steps 1) through 4) across 5 splits to average out the evaluation results

For a refresher on custom training, please refer to our training documentation.

Requirements

To run the evaluation on your custom model, it will need the meet the following criteria:

  • A custom trained model model version with:
    1. At least 2 concepts
    2. At least 10 training inputs per concept (At least 50 inputs per concept is recommended)

Running Evaluation

You can run the evaluation on a specific model version of your custom model in the Explorer . Go to your Application, click on your model of interest, and select the Versions tab. Simply click on the Evaluate button for the specific model version.

model-eval-versions model eval

The evaluation may take up to 30 minutes. Once it is complete, the Evaluate button will become View button. Click on the View button to see the evaluation results.

model eval results

Note that the evaluation may result in an error if the model version doesn’t satisfy the requirements above.

For more information on how to interpret the evaluation results and to improve your model, check out the Evaluation corner under the “Advanced” section below.

Feedback

The feedback endpoint allows you to send feedback on your prediction and search responses. Currently, you can send feedback on 3 types of responses: 1. predictions without regions, 2. predictions with regions, and 3. searches. The request bodies are slightly different but all share the feedback_info component. Note that sending feedback will not result in immediate changes to your predictions or searches, but you may see accuracy improvements to them over time as we aggregate the feedback for evaluation and improvement of your models.

We highly encourage integrating feedback into your application as this dramatically improves accuracy of models over time. You or users of your application can correct errors, such as: if the system returned cat for an image of a dog, you can send feedback with add_tags="dog", remove_tags="cat". You can also use feedback to further enhance the models by suggesting a new concept or region, such as: if the model returned "dog", you can add a new concept called "laborador_retriever".

The feedback endpoint does not count towards your usage limits. Please use it as much as you can.

Feedback Info

The feedback info object allows for specific information to be recorded from your application and your end users. All fields are optional unless noted.

  • event_type [required] - Use 'annotation' for prediction feedback, and use 'search_click' for search feedback.
  • output_id/search_id [required] - the id associated with the output or search received from a given API call. With this, the API can traceback the response.
  • end_user_id - the id associated with your end user. If you want to be able to understand which user was producing this feedback, you can associate it with this field.
  • session_id - the id associated with your user's interface. This is allows you to understand what the user was doing during that time and allows you an avenue to investigate more.

Prediction Feedback

To send feedback on predictions, you can use POST /v2/models/{model_id}/feedback.

Please use this request to send feedback on the model prediction responses without any region data. This includes any feedback on concept models such as General, Food, and Travel models.

In each request body, you can send

  • • concept id: either the correct concept name that you defined or the id that Clarifai defined in the predict response
  • • concept value: true if the concept is present in the input, or false if it is not present in the input

This allows for our model to re-evaluate the concept and improve the accuracy of our predictions.

Note that the event_type should be set to annotation.

clarifai.models.feedback(Clarifai.GENERAL_MODEL, 'https://samples.clarifai.com/dog.tiff', {
  id: '{input_id}',
  data: {
    concepts: [
      {'id': 'ai_8S2Vq3cR', 'value': true },
      {'id': 'ai_mFqxrph2', 'value': false }
    ]
  },
  info: {
    'eventType':  'annotation',
    'outputId':   '{output_id}',
    'endUserId':  '{end_user_id}',
    'sessionId':  '{session_id}'
  }
})
  .then(log)
  .catch(log);
model = app.models.get('general-v1.3')

model.send_concept_feedback(input_id='{input_id}', url='https://samples.clarifai.com/dog.tiff',
                        concepts=['ai_8S2Vq3cR'], not_concepts=['ai_mFqxrph2'],
                        feedback_info=FeedbackInfo(event_type='annotation',
                                                   output_id='{output_id}',
                                                   end_user_id='{end_user_id}',
                                                   session_id='{session_id}'))
ModelFeedbackRequest request =
    client.modelFeedback(client.getDefaultModels().generalModel().id())
       .withInputId("{input_id}")
       .withImageUrl("https://samples.clarifai.com/dog.tiff")
       .withConcepts(
           ConceptFeedback.forIdAndValue("ai_8S2Vq3cR", true),
           ConceptFeedback.forIdAndValue("ai_mFqxrph2", false)
       )
       .withEventType("annotation")
       .withOutputId("{output_id}")
       .withEndUserId("{end_user_id}")
       .withSessionId("{session_id}");
// Coming Soon
curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "input": {
      "id": "{input_id}",
      "data": {
        "image": {
          "url": "https://samples.clarifai.com/dog.tiff"
        },
        "concepts":[
          {"id": "ai_8S2Vq3cR", "value": true },
          {"id": "ai_mFqxrph2", "value": false }
        ]
      },
      "feedback_info":{
        "event_type":   "annotation",
        "output_id":    "{output_id}",
        "end_user_id":  "{end_user_id}",
        "session_id":   "{session_id}"
      }
    }
  }'\
  https://api.clarifai.com/v2/models/aaa03c23b3724a16a56b629203edc62c/feedback
Show All
Response{
    "status": {
        "code": 10000,
        "description": "Ok"
    }
}

Prediction with Regions Feedback

To send feedback on predictions, you can use POST /v2/models/{model_id}/feedback.

Please use this request to send feedback on the model prediction responses with region data. This includes any feedback on detection models such as Face Detection, Demographic, Celebrity, and Logo models. These models return a list of region objects, each of which contains local coordinates and detected concepts.

In the request body for the feedback for predictions with regions, you can send each region object with:

  • • correct bounding_box coordinates of the region
  • feedback for the originally predicted region: accurate if it's correct, misplaced if the coordinates should be modified, not_detected if the region was not originally detected by the model, or false_positive if the region should not have been detected by the model
  • • concept id: either the correct concept name that you defined or the id that Clarifai defined in the predict response
  • • concept value: true if the concept is present in the input, or false if it is not present in the input

Note that the event_type should be set to annotation.

clarifai.models.feedback(Clarifai.LOGO_MODEL, 'https://clarifai.com/developer/static/images/model-samples/logo-002.jpg', {
  id: '{input_id}',
  data: {
    'regions': [
       {
         'region_info': {
           'bounding_box': {
             'top_row': 0.3,
             'left_col': 0.2,
             'bottom_row': 0.7,
             'right_col': 0.8
           },
           'feedback': 'accurate'
         },
         'data': {
           'concepts': [
             {"id": "ai_TG4GHlnf", "value": true }
           ]
         }
       }
     ]
   },
  info: {
    'eventType':  'annotation',
    'outputId':   '{output_id}',
    'endUserId':  '{end_user_id}',
    'sessionId':  '{session_id}'
   }
 })
  .then(log)
  .catch(log);
model = app.models.get('logo')

model.send_region_feedback(input_id='{input_id}',
url='https://clarifai.com/developer/static/images/model-samples/logo-002.jpg',
    regions=[Region(region_info=RegionInfo(bbox=BoundingBox(top_row=0.3, left_col=0.2, bottom_row=0.7, right_col=0.8), feedback_type='accurate'),
                    concepts=[ConceptInput(id='ai_TG4GHlnf', value=True)])],
                    feedback_info=FeedbackInfo(event_type='annotation',
                                               output_id='{output_id}',
                                               end_user_id='{end_user_id}',
                                               session_id='{session_id}'))
ModelFeedbackRequest request =
    client.modelFeedback(client.getDefaultModels().logoModel().id())
        .withInputId("{input_id}")
        .withImageUrl("https://clarifai.com/developer/static/images/model-samples/logo-002.jpg")
        .withRegions(
            RegionFeedback.make(
                Crop.create().top(0.3f).left(0.2f).bottom(0.7f).right(0.8f),
                Feedback.ACCURATE
            )
              .withConceptFeedbacks(ConceptFeedback.forIdAndValue("ai_TG4GHlnf", true)))
        .withEventType("annotation")
        .withOutputId("{output_id}")
        .withEndUserId("{end_user_id}")
        .withSessionId("{session_id}");
// Coming Soon
curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "input": {
      "id": "{input_id}",
      "data": {
        "image": {
          "url": "https://clarifai.com/developer/static/images/model-samples/logo-002.jpg"
        },
        "regions":
        [
          {   
            "region_info": {
              "bounding_box": {
                "top_row": 0.3,
                "left_col": 0.2,
                "bottom_row": 0.7,
                "right_col": 0.8
              },
              "feedback": "accurate"
            },
            "data": {
              "concepts":[
                {"id": "ai_TG4GHlnf", "value": true }
              ]
            }
          }
        ]
      },
      "feedback_info":{
        "event_type":   "annotation",
        "output_id":    "{output_id}",
        "end_user_id":  "{end_user_id}",
        "session_id":   "{session_id}",
      }
    }
  }'\
  https://api.clarifai.com/v2/models/c443119bf2ed4da98487520d01a0b1e3/feedback
Show All
Response{
    "status": {
        "code": 10000,
        "description": "Ok"
    }
}

Search Feedback

To send feedback on searches, you can use POST /v2/searches/feedback.

This request is meant to collect the correctly searched inputs, which is usually done by capturing your end user's clicks on the given search results. Your feedback will help us improve our search algorithm.

In the request body for search feedback, you can specify:

  • • input id of a correct image ('hit') from the search results
  • search_id from the search API response

Note that the event_type should be set to search_click.

// Coming Soon
# Coming Soon
SearchesFeedbackRequest request = client.searchesFeedback()
    .withId("{input_id}")
    .withEndUserId("{end_user_id}")
    .withSessionId("{session_id}")
    .withEventType("search_click")
    .withSearchId("{search_id}");
// Coming Soon
curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "input": {
      "id": "{input_id}",
      "feedback_info": {
        "event_type":   "search_click",
        "search_id":    "{output_id}",
        "end_user_id":  "{end_user_id}",
        "session_id":   "{session_id}"
      }
    }
  }'\
  https://api.clarifai.com/v2/searches/feedback
Show All
Response{
    "status": {
        "code": 10000,
        "description": "Ok"
    }
}

Public Models

Clarifai provides a variety of pre-trained models that you can use to make predictions. These models were developed in-house and have been thoroughly tested. Before you train your own model, we suggest trying these out to see if they fit your needs.

You can browse all of our models here.

Applications

API calls are tied to an account and application. Any model you create or search indexes you add images to, will be contained within an application.

You can create as many applications as you want and can edit or delete them as you see fit. Each application has a unique API Key. These are used for authentication. You can learn more about authentication above.

Create an Application

To create an application, head on over to the applications page and press the 'Create a New Application' button.

At a minimum, you'll need to provide an application name and a public model to choose as your Base Workflow. You may also set the default language. If you plan on using a language other than English, you must use the General model. You can learn more about models and languages in the public model guide above.

image showing the edit app button on the Manage Application page

Base Workflow

When you create an application, you will need to select a Base Workflow, along with App Name and Default Language. You can choose from a select list of public models as your Base Workflow.

The Base Workflow you choose will optimize custom trained models to use the knowledge base from the selected public model. For example, if you're training a custom model around food, you could choose the Food model to gain optimized results. We recommend choosing the General model if you're not sure which public model would best suit your inputs.

Check out our model gallery for a description of the select public models that can be used as a base workflow. Also note that you cannot change the Base Workflow of an app once it's created.

Edit an Application

If at any point you'd like to change the application name, add/remove API Keys, and modify Workflow management for Predict, you may do so by visiting the application page and changing the values.

Image showing the edit application screen where you can change the name, add api keys, or delete the application

Delete an Application

If you'd like to delete an application, you may do so at any time by visiting the application page and pressing the 'Delete application' button. You'll be asked to confirm your change. Please note that once you delete an application, we cannot recover it. You will also lose all images, concepts and models associated with that application. Proceed with caution.

Languages

The Clarifai API supports many languages in addition to English. When making a predict api request, you can pass in the language you would like the concepts returned in.

Supported Languages

LanguageCode
Arabic (ar)ar
Bengali (bn)bn
Danish (da)da
German (de)de
English (en)en
Spanish (es)es
Finnish (fi)fi
French (fr)fr
Hindi (hi)hi
Hungarian (hu)hu
Italian (it)it
Japanese (ja)ja
Korean (ko)ko
Dutch (nl)nl
Norwegian (no)no
Punjabi (pa)pa
Polish (pl)pl
Portuguese (pt)pt
Russian (ru)ru
Swedish (sv)sv
Turkish (tr)tr
Chinese Simplified (zh)zh
Chinese Traditional (zh-TW)zh-TW

Supported Models

The only public model which supports languages other than English is the General model. If you make a predict request using a language other than English on a public model other than General, you will receive an error.

Default Language

When you create a new Application, you must specify a default language. This will be the default language concepts are returned in when you do not explicitly set a language in an API request. You cannot change the default language. You can however change languages per request.

create new app

Example Predict API Request

You can predict concepts in a language other then the Application's default, by explicitly passing in the language. Here is how you predict concepts in Chinese:


app.models.predict(Clarifai.GENERAL_MODEL, "https://samples.clarifai.com/metro-north.jpg", {language: 'zh'}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

m = app.models.get('general-v1.3')

# predict labels in simplified Chinese
m.predict_by_url('https://samples.clarifai.com/metro-north.jpg', lang='zh')

# predict labels in Japanese
m.predict_by_url('https://samples.clarifai.com/metro-north.jpg', lang='ja')

client.predict(client.getDefaultModels().generalModel().id())
    .withInputs(ClarifaiInput.forImage("https://samples.clarifai.com/metro-north.jpg"))
    .withLanguage("zh")
    .executeSync();

// first get the general model.
[app getModelByName:@"general-v1.3" completion:^(ClarifaiModel *model, NSError *error) {
  // create input to predict on.
  ClarifaiImage *input = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/metro-north.jpg"];

  // predict with the general model in Chinese.
  [model predictOnImages:@[input] withLanguage:@"zh" completion:^(NSArray<ClarifaiOutput *> *outputs, NSError *error) {
    for (ClarifaiConcept *concept in outputs[0].concepts) {
      NSLog(@"tag: %@", concept.conceptName);
      NSLog(@"probability: %f", concept.score);
    }
  }];
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
  "inputs": [
    {
      "data": {
        "image": {
          "url": "https://samples.clarifai.com/metro-north.jpg"
        }
      }
    }
  ],
  "model":{
    "output_info":{
      "output_config":{
        "language":"zh"
      }
    }
  }
}'\
  https://api.clarifai.com/v2/models/aaa03c23b3724a16a56b629203edc62c/outputs
Show All
Response{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "outputs": [
    {
      "id": "b9f3c12f1534440fa984dc463e491780",
      "status": {
        "code": 10000,
        "description": "Ok"
      },
      "created_at": "2017-01-31T20:59:27Z",
      "model": {
        "name": "general-v1.3",
        "id": "aaa03c23b3724a16a56b629203edc62c",
        "created_at": "2016-03-09T17:11:39Z",
        "app_id": null,
        "output_info": {
          "message": "Show output_info with: GET /models/{model_id}/output_info",
          "type": "concept"
        },
        "model_version": {
          "id": "aa9ca48295b37401f8af92ad1af0d91d",
          "created_at": "2016-07-13T01:19:12Z",
          "status": {
            "code": 21100,
            "description": "Model trained successfully"
          }
        }
      },
      "input": {
        "id": "b9f3c12f1534440fa984dc463e491780",
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/metro-north.jpg"
          }
        }
      },
      "data": {
        "concepts": [
          {
            "id": "ai_HLmqFqBf",
            "name": "铁路列车",
            "app_id": null,
            "value": 0.9989112
          },
          {
            "id": "ai_fvlBqXZR",
            "name": "铁路",
            "app_id": null,
            "value": 0.9975532
          },
          {
            "id": "ai_Xxjc3MhT",
            "name": "运输系统",
            "app_id": null,
            "value": 0.9959158
          },
          {
            "id": "ai_6kTjGfF6",
            "name": "站",
            "app_id": null,
            "value": 0.992573
          },
          {
            "id": "ai_RRXLczch",
            "name": "火车",
            "app_id": null,
            "value": 0.992556
          },
          {
            "id": "ai_VRmbGVWh",
            "name": "旅游",
            "app_id": null,
            "value": 0.98789215
          },
          {
            "id": "ai_SHNDcmJ3",
            "name": "地铁",
            "app_id": null,
            "value": 0.9816359
          },
          {
            "id": "ai_jlb9q33b",
            "name": "通勤",
            "app_id": null,
            "value": 0.9712483
          },
          {
            "id": "ai_46lGZ4Gm",
            "name": "铁路",
            "app_id": null,
            "value": 0.9690325
          },
          {
            "id": "ai_tr0MBp64",
            "name": "交通",
            "app_id": null,
            "value": 0.9687052
          },
          {
            "id": "ai_l4WckcJN",
            "name": "模煳",
            "app_id": null,
            "value": 0.9667078
          },
          {
            "id": "ai_2gkfMDsM",
            "name": "平台",
            "app_id": null,
            "value": 0.9624243
          },
          {
            "id": "ai_CpFBRWzD",
            "name": "城市的",
            "app_id": null,
            "value": 0.960752
          },
          {
            "id": "ai_786Zr311",
            "name": "沒有人",
            "app_id": null,
            "value": 0.95864904
          },
          {
            "id": "ai_6lhccv44",
            "name": "商业",
            "app_id": null,
            "value": 0.95720303
          },
          {
            "id": "ai_971KsJkn",
            "name": "跑道",
            "app_id": null,
            "value": 0.9494642
          },
          {
            "id": "ai_WBQfVV0p",
            "name": "城市",
            "app_id": null,
            "value": 0.94089437
          },
          {
            "id": "ai_dSCKh8xv",
            "name": "快速的",
            "app_id": null,
            "value": 0.9399334
          },
          {
            "id": "ai_TZ3C79C6",
            "name": "马路",
            "app_id": null,
            "value": 0.93121606
          },
          {
            "id": "ai_VSVscs9k",
            "name": "终点站",
            "app_id": null,
            "value": 0.9230834
          }
        ]
      }
    }
  ]
}

Example Search By Tag API Request

You can search for concepts in other languages even if the default language of your application is English. When you add inputs to your application, concepts are predicted for every language. Here is an example of searching for '人' which is simplified Chinese for 'people'.


app.inputs.search({
  concept: {
    name: '人'
  },
  language: 'ja'
}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# search '人' in simplified Chinese
app.inputs.search_by_predicted_concepts(u'人', lang='zh')

client.searchInputs(
    SearchClause.matchImageURL(ClarifaiImage.of("https://samples.clarifai.com/metro-north.jpg")))
    .withLanguage("zh")
    .getPage(1)
    .executeSync();

// create search term with concept you want to search predicted inputs with.
ClarifaiConcept *concept1 = [[ClarifaiConcept alloc] initWithConceptName:@"人"];
ClarifaiSearchTerm *searchTerm = [[ClarifaiSearchTerm alloc] initWithSearchItem:concept1 isInput:NO];

// search will find inputs predicted to be associated with the given concept.
[_app search:@[searchTerm] page:@1 perPage:@20 language:@"zh" completion:^(NSArray<ClarifaiSearchResult *> *results, NSError *error) {
  for (ClarifaiSearchResult *result in results) {
    NSLog(@"image url: %@", result.mediaURL);
    NSLog(@"probability: %f", [result.score floatValue]);
  }
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "query": {
      "ands": [
        {
          "output": {
            "data": {
              "concepts": [
                {
                  "name": "人"
                }
              ]
            }
          }
        }
      ],
      "language": "zh"
    }
  }'\
  https://api.clarifai.com/v2/searches

Example Search Concepts API Request

You can also search for concepts in a different language:


app.concepts.search('人*', 'zh').then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

app.concepts.search(u'人*', lang='zh')

client.searchConcepts("人*")
    .withLanguage("zh")
    .getPage(1)
    .executeSync();

// Search for all concept names in chinese, beginning with "人".
[_app searchForConceptsByName:@"人*" andLanguage:@"zh" completion:^(NSArray<ClarifaiConcept *> *concepts, NSError *error) {
  for (ClarifaiConcept *concept in concepts) {
    NSLog(@"tag name: %@", concept.conceptName);
  }
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "concept_query": {
      "name":"人*",
      "language": "zh"
    }
  }'\
  https://api.clarifai.com/v2/concepts/searches
Show All
Response{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "concepts": [
    {
      "id": "ai_l8TKp2h5",
      "name": "人",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_ZKJ48TFz",
      "name": "人",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_GlPlRlTZ",
      "name": "人为破坏",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_8ZsdCrVZ",
      "name": "人体模型",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_K1KL0zgk",
      "name": "人力的",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_Tm9d2BZ2",
      "name": "人口",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_NLF8h1fJ",
      "name": "人口",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_8bHdFtsg",
      "name": "人口",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_vLnr3Mcj",
      "name": "人孔",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_HRt4nfvL",
      "name": "人工智能",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_Qc3mqxTJ",
      "name": "人才",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_VFKQ0qD6",
      "name": "人物",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_Wz8JXXMB",
      "name": "人类免疫缺陷病毒",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_bzp3Lg81",
      "name": "人类的",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_dJ15S9s6",
      "name": "人群",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_MNCVrmml",
      "name": "人行天桥",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_CChWH41S",
      "name": "人行横道",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_4lbXrFgT",
      "name": "人造",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_277LRf4d",
      "name": "人造卫星",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    },
    {
      "id": "ai_H3RDmvSn",
      "name": "人造奶油",
      "created_at": "2016-03-17T11:43:01Z",
      "updated_at": "2016-03-17T11:43:01Z",
      "app_id": null,
      "language": "zh"
    }
  ]
}

Mobile SDK

Clarifai’s Mobile SDK enables machine learning directly on your device, bypassing the traditional requirement of internet connectivity and extensive computing power. It makes it easy to use image and video recognition on device and in real-time.

Mobile SDK

The Mobile SDK is currently available in Limited Preview. You can get more information about the Limited Preview program here.

Getting Started On Mobile

Since the Mobile SDK is in a Limited Preview program, you need to apply to get access. In order to apply you would require a Clarifai account. Please create one here before proceeding with this guide.

Once you are accepted into the program, you will receive email instructions on how to install the SDK and how to download your first public model on device.

Currently the mobile SDK supports both Objective-C and Swift.

Inputs On Device

The SDK is built around a simple idea. You give inputs (images) to the library and it returns predictions (concepts). You need to add inputs to make predictions on it, and/or to train custom models using the inputs as training data.

The sections below will showcase how to add Inputs on your device and tag them with concepts in order to train a custom model.

Add Device Inputs

All inputs are created from a DataAsset object in the Mobile SDK. A Data Asset is a container for the asset in question, plus metadata related to it. You can create a DataAsset initialized with an Image on Device or from a URL as shown in the example below.

// Only for the Mobile SDK
// Only for the Mobile SDK
// Only for the Mobile SDK
// Initialize CAIImage object from an image URL
NSURL *imageURL = [NSURL urlWithString:@"<your image url>"];
CAIImage *image = [[CAIImage alloc] initWithURL:imageURL];

// Initialize CAIImage object with an image on device
UIImage *deviceImage = [UIImage ...];
CAIImage *localImage = [[CAIImage alloc] initWithImage:deviceImage];

// A Data Asset is a container for the asset in question, plus metadata
// related to it
CAIDataAsset *dataAsset = [[CAIDataAsset alloc] initWithImage:image];

// An input object contains the data asset, temporal information, and
// is a fundamental component to be used by models to train on or
// predict
CAIInput *input = [[CAIInput alloc] initWithDataAsset:dataAsset];
// Only for the Mobile SDK

Add Concepts to an Input

You can also add concepts to your inputs if you want to use them for custom training instead of just predictions.

// Only for the Mobile SDK
# Only for the Mobile SDK
// Only for the Mobile SDK
// A concept is a descriptive attribute regarding a data asset. If used
// for training you may want to provide a score.
CAIConcept *concept = [[CAIConcept alloc] initWithId:@"<unique id>"
                                                name:@"<label describing the concept>"];

UIImage *deviceImage = [UIImage ...];
CAIImage *localImage = [[CAIImage alloc] initWithImage:deviceImage];

CAIDataAsset *dataAsset = [[CAIDataAsset alloc] initWithImage:localImage];
[dataAsset addConcepts:@[concept]];
// Only for the Mobile SDK

Prediction On Device

Just as with our API, you can use the predict functionality on device with any of our Public Models or any Custom Models that you have on your device.

Predictions generate outputs. An output has a similar structure to an input. It contains a data asset and concepts. The concepts associated with an output contain the predictions and their respective score (degree of confidence.)

// Only for the Mobile SDK
// Only for the Mobile SDK
// Only for the Mobile SDK
// See how to create an input from the examples above
CAIInput *input = [[CAIInput alloc] initWithDataAsset:dataAsset];

// Use the model you want to predict on. This can be a custom trained or one of our public models.
// The model in the sample code below is our General Model.
CAIModel *generalModel = [Clarifai sharedInstance].generalModel;
[generalModel predict:@[input] completionHandler:^(NSArray<CAIOutput *> * _Nullable outputs, NSError * _Nullable error) {
    // Iterate through outputs to learn about what has been predicted
    for (CAIOutput *output in outputs) {
    }
}];
// Only for the Mobile SDK

Models On Device

In order for predictions to happen on device, you must have models loaded on the device. For the Mobile SDK, we have Public Models and Custom Models available on device, similar to our web API.

On Device Public Models

Our team has developed a variety of pre-trained public models which our users can make predictions against. As of now, we have enabled our most comprehensive model, the General Model, to be loaded on device. As the product evolves and we start to understand how you're using the SDK, we will be making more of our Public Models available. The sample code below shows how to load the General Model and make a prediction against it.

// Only for the Mobile SDK
// Only for the Mobile SDK
// Only for the Mobile SDK
//Get General Model
CAIModel *generalModel = [Clarifai sharedInstance].generalModel;
// Only for the Mobile SDK

Custom Models

There are times when you wish you had a model that sees the world the way you see it. The SDK allows you to do this on device. You can create your own model and train it with your own images and concepts. Once you train it to see how you would like it to see, you can then use that model to make predictions.

Just to quickly refresh, a model contains a group of concepts and it will only see the concepts it contains; you use inputs as the training data and label them with concepts. You do not need many inputs to get started. We recommend starting with 10 and adding more as needed.

For the SDK, we have enabled two types of Custom Models: Local Models and Cloud Models.

Local Models

Local Models are created and trained on the mobile device itself, which can then be used for prediction. All of the data used to train a Local Model stays on device - nothing is sent to the cloud. Local models are great for privacy, but have limits around sharing, and scaling these models to other devices.

Create a Local Model

// Only for the Mobile SDK
// Only for the Mobile SDK
// Only for the Mobile SDK
// Create a custom trained model on device
CAIModel *customModel = [[CAIModel alloc] initWithId:@"<unique id>" name:@"<model name>"];
// Only for the Mobile SDK

Train a Local Model

// Only for the Mobile SDK
// Only for the Mobile SDK
// Only for the Mobile SDK
// Images you want to train on
UIImage *localImage = [UIImage ...];
CAIImage *image1 = [[CAIImage alloc] initWithImage:localImage];

UIImage *anotherImage = [UIImage ...];
CAIImage *image2 = [[CAIImage alloc] initWithImage:anotherImage];

// Create concept(s) with scores. 1 for THING, 0 for NOT-THING
CAIConcept *trainingConcept = [[CAIConcept alloc] initWithId:@"<unique id>"
                                                        name:@"<Nice name>"
                                                       score:1.0];
NSArray<CAIConcept *> *trainingConcepts = @[trainingConcept];

// Create Data Assets and Tag them with Concept(s)
CAIDataAsset *dataAsset = [[CAIDataAsset alloc] initWithImage:image1];
[dataAsset addConcepts:trainingConcepts];

CAIDataAsset *dataAsset2 = [[CAIDataAsset alloc] initWithImage:image2];
[dataAsset2 addConcepts:trainingConcepts];

// Create Inputs
CAIInput *input = [[CAIInput alloc] initWithDataAsset:dataAsset];
CAIInput *input2 = [[CAIInput alloc] initWithDataAsset:dataAsset2];
NSArray<CAIInput *> *trainingInputs = @[input, input2];
[[Clarifai sharedInstance] saveEntities:trainingInputs];

// Create custom trained model on device
CAIModel *customModel = [[CAIModel alloc] initWithId:@"<unique id>" name:@"<model name>"];

// Train the custom model
[customModel trainWithConcepts:trainingConcepts inputs:trainingInputs completionHandler:^(NSError * _Nullable error) {
    if (error) {
         NSLog(@"Custom model not trained");
         return;
     }

     // The model is ready to be used to make predictions
     NSLog(@"Custom model trained");
}];
// Only for the Mobile SDK

Cloud Models

Cloud Models are models that are created and trained in Clarifai’s cloud environment and can be viewed in Preview UI. You are able to use models that you have created in the past with Clarifai, or can create a brand new one for the SDK. Cloud models are great for deploying the same model on many devices where you would like prediction consistency.

Loading Models from the Cloud

// Only for the Mobile SDK
// Only for the Mobile SDK
// Only for the Mobile SDK

// loads all models created locally and from the cloud.
[[Clarifai sharedInstance] loadEntityType:CAIEntityTypeModel range:NSMakeRange(1, 20) completionHandler:^(NSArray<CAIModel *> * _Nullable models, NSError * _Nullable error) {
    // If you have trained models on the cloud, they will show up in the returned list of models.
    __block CAIModel *remoteModel = nil;
    [models enumerateObjectsUsingBlock:^(CAIModel * _Nonnull model, NSUInteger idx, BOOL * _Nonnull stop) {
        if ([model.modelId isEqualToString:@"your-remote-model's-id"]) {
            remoteModel = model;
            *stop = YES;
        } 
    }];

    // Models loaded from the cloud can be used for predicting offline, but re-training will occur on the cloud.
    [remoteModel predict:@[input] completionHandler:^(NSArray * _Nullable outputs, NSError * _Nullable error) {
        // Iterate through outputs to learn about what has been predicted
        for (CAIOutput *output in outputs) {
        }
    }];
}];
// Only for the Mobile SDK

Inputs

The API is built around a simple idea. You send inputs (images) to the service and it returns predictions. In addition to receiving predictions on inputs, you can also 'save' inputs and their predictions to later search against. You can also 'save' inputs with concepts to later train your own model.

Add Inputs

You can add inputs one by one or in bulk. If you do send bulk, you are limited to sending 128 inputs at a time.

Images can either be publicly accessible URLs or file bytes. If you are sending file bytes, you must use base64 encoding.

You are encouraged to send inputs with your own id. This will help you later match the input to your own database. If you do not send an id, one will be created for you.

Add an input using a publicly accessible URL

app.inputs.create({
  url: "https://samples.clarifai.com/metro-north.jpg"
}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

image = app.inputs.create_image_from_url("https://samples.clarifai.com/metro-north.jpg")

client.addInputs()
    .plus(ClarifaiInput.forImage("https://samples.clarifai.com/metro-north.jpg"))
    .executeSync();

ClarifaiImage *image = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/metro-north.jpg"];
[app addInputs:@[image] completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
    NSLog(@"inputs: %@", inputs);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/metro-north.jpg"
          }
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/inputs
Add an input using bytes

The data must be base64 encoded. When you add a base64 image to our servers, a copy will be stored and hosted on our servers. If you already have an image hosting service we recommend using it and adding images via the url parameter.


app.inputs.create({
  base64: "Zvfauhti4D..."
}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# add from filename
app.inputs.create_image_from_filename(filename)

# add from base64 bytes
app.inputs.create_image_from_base64(base64_bytes)

client.addInputs()
    .plus(ClarifaiInput.forImage(new File("image.png")))
    .executeSync();

ClarifaiImage *imageFromImage = [[ClarifaiImage alloc] initWithImage:@"dress.jpg"];
[app addInputs:@[imageFromImage] completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
    NSLog(@"inputs: %@", inputs);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "base64": '"`base64 /home/user/image.jpeg`"'"
          }
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/inputs
Add multiple inputs with ids

app.inputs.create([
  {
    url: "https://samples.clarifai.com/metro-north.jpg",
    id: 'train1'
  },
  {
    url: "https://samples.clarifai.com/puppy.jpeg",
    id: 'puppy1'
  }
]).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);
from clarifai.rest import Image as ClImage

img1 = ClImage(url="https://samples.clarifai.com/metro-north.jpg", image_id="train1")
img2 = ClImage(url="https://samples.clarifai.com/puppy.jpeg", image_id="puppy1")

app.inputs.bulk_create_images([img1, img2])

client.addInputs()
    .plus(
        ClarifaiInput.forImage("https://samples.clarifai.com/metro-north.jpg")
            .withConcepts(Concept.forID("id1")),
        ClarifaiInput.forImage("https://samples.clarifai.com/wedding.jpg")
            .withConcepts(Concept.forID("id2"))
    )
    .executeSync();

ClarifaiImage *train = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/metro-north.jpg"];
train.inputID = @"train";

ClarifaiImage *puppy = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/puppy.jpeg"];
puppy.inputID = @"puppy";

[app addInputs:@[train, puppy] completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
    NSLog(@"inputs: %@", inputs);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/metro-north.jpg"
          }
        },
        "id": "{id1}"
      },
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/puppy.jpeg"
          }
        },
        "id": "{id2}"
      }
    ]
  }'\
  https://api.clarifai.com/v2/inputs

Add Inputs With Concepts

If you would like to add an input with concepts, you can do so like this. Concepts play an important role in creating your own models using your own concepts. You can learn more about creating your own models above. Concepts also help you search for inputs. You can learn more about search here.

When you add a concept to an input, you need to indicate whether the concept is present in the image or if it is not present.

You can add inputs with concepts as either a URL or bytes.


app.inputs.create({
  url: "https://samples.clarifai.com/puppy.jpeg",
  concepts: [
    {
      id: "boscoe",
      value: true
    }
  ]
}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);
from clarifai.rest import Image as ClImage

# add by url
app.inputs.create_image_from_url("https://samples.clarifai.com/puppy.jpeg", concepts=['boscoe'])

# add by base64 bytes
app.inputs.create_image_from_base64(base64_bytes, concepts=['boscoe'])

# add by raw bytes
app.inputs.create_image_from_bytes(raw_bytes, concepts=['boscoe'])

# add by local file
app.inputs.create_image_from_filename(local_filename, concepts=['boscoe'])

# add multiple with concepts
img1 = ClImage(url="https://samples.clarifai.com/puppy.jpeg", concepts=['boscoe'], not_concepts=['our_wedding'])
img2 = ClImage(url="https://samples.clarifai.com/wedding.jpg", concepts=['our_wedding'], not_concepts=['cat','boscoe'])

app.inputs.bulk_create_images([img1, img2])

client.addInputs()
    .plus(ClarifaiInput.forImage("https://samples.clarifai.com/puppy.jpeg")
        .withConcepts(
            // To mark a concept as being absent, chain `.withValue(false)`
            Concept.forID("boscoe")
        )
    )
    .executeSync();

ClarifaiImage *puppy = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/puppy.jpeg"
                                              andConcepts:@[@"cute puppy"]];

[app addInputs:@[puppy] completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
    NSLog(@"inputs: %@", inputs);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/puppy.jpeg"
          },
          "concepts":[
            {
              "id": "boscoe",
              "value": true
            }
          ]
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/inputs

Add Inputs With Custom Metadata

In addition to adding an input with concepts, you can also add an input with custom metadata. This metadata will then be searchable. Metadata can be any arbitrary JSON.


app.inputs.create({
  url: "https://samples.clarifai.com/puppy.jpeg",
  metadata: {id: 'id001', type: 'plants', size: 100}
}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# metadata must be defined as JSON object
metadata = {'id':'id001', 'type':'plants', 'size':100}

# adding metadata along with url, filename, etc
app.inputs.create_image_from_url(url="https://samples.clarifai.com/puppy.jpeg", metadata=metadata)
app.inputs.create_image_from_filename(filename="aa.jpg", metadata=metadata)

# define an image with metadata for bulk import
img = Image(url="", metadata=metadata)

app.inputs.bulk_create_images([img])

final JsonObject metadata = new JsonObject();
metadata.addProperty("isPuppy", true);
client.addInputs()
    .plus(
        ClarifaiInput.forImage("https://samples.clarifai.com/puppy.jpeg")
            .withMetadata(metadata)
    ).executeSync();

ClarifaiImage *puppy = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/puppy.jpeg"
                                              andConcepts:@[@"cute puppy"]];
puppy.metadata = @{@"my_key": @[@"my",@"values"], @"cuteness": @"extra-cute"};
[app addInputs:@[puppy] completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
  NSLog(@"inputs: %@", inputs);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/puppy.jpeg",
            "allow_duplicate_url": true
          },
          "metadata": {
            "key": "value",
            "list":[1,2,3]
          }
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/inputs

Add Input With A Crop

When adding an input, you can specify crop points. The API will crop the image and use the resulting image. Crop points are given as percentages from the top left point in the order of top, left, bottom and right.

As an example, if you provide a crop as 0.2, 0.4, 0.3, 0.6 that means the cropped image will have a top edge that starts 20% down from the original top edge, a left edge that starts 40% from the original left edge, a bottom edge that starts 30% from the original top edge and a right edge that starts 60% from the original left edge.


app.inputs.create(
  {
    "url": "https://samples.clarifai.com/metro-north.jpg",
    "crop": [0.2, 0.4, 0.3, 0.6]
  }
).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# add url with crop
app.inputs.create_image_from_url(url="https://samples.clarifai.com/metro-north.jpg", crop=[0.2, 0.4, 0.3, 0.6])

client.addInputs()
    .plus(
        ClarifaiInput.forInputValue(
            ClarifaiURLImage.of("https://samples.clarifai.com/metro-north.jpg")
                .withCrop(Crop.create()
                    .top(0.2F)
                    .left(0.4F)
                    .bottom(0.3F)
                    .right(0.6F)
                )
        )
    )
    .executeSync();

ClarifaiCrop *crop = [[ClarifaiCrop alloc] initWithTop:0.2 left:0.3 bottom:0.7 right:0.8];
ClarifaiImage *puppy = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/puppy.jpeg" andCrop:crop];

[app addInputs:@[puppy] completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
    NSLog(@"inputs: %@", inputs);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/metro-north.jpg",
            "crop": [0.2, 0.4, 0.3, 0.6]
          }
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/inputs

Get Inputs

You can list all the inputs (images) you have previously added either for search or train.

If you added inputs with concepts, they will be returned in the response as well.

This request is paginated.


app.inputs.list({page: 1, perPage: 20}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# this is a generator
app.inputs.get_all()

# get a page of inputs
app.inputs.get_by_page(page=1, per_page=20)

client.getInputs() // optionally takes a perPage parameter
    .getPage(1)
    .executeSync();

[app getInputsOnPage:1 pageSize:20 completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
    NSLog(@"inputs: %@", inputs);
}];

curl -X GET \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/inputs

Get Input By Id

If you'd like to get a specific input by id, you can do that as well.


app.inputs.get({id}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

image = app.inputs.get(input_id)

client.getInputByID("{id}").executeSync();

[_app getInput:input_id completion:^(ClarifaiInput *input, NSError *error) {
    NSLog(@"input": %@, input);
}];

curl -X GET \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/inputs/{id}

Get Inputs Status

If you add inputs in bulk, they will process in the background. You can get the status of all your inputs (processed, to_process and errors) like this:


app.inputs.getStatus().then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

app.inputs.check_status()

client.getInputsStatus().executeSync();

[app getInputsStatus:^(int numProcessed, int numToProcess, int errors, NSError *error) {
    NSLog(@"number of inputs processed: %d", numProcessed);
}];

curl -X GET \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/inputs/status

Update Input With Concepts

To update an input with a new concept, or to change a concept value from true/false, you can do that:


app.inputs.mergeConcepts([
  {
    id: "{id}",
    concepts: [
      {
        id: "tree"
      },
      {
        id: "water",
        value: false
      }
    ]
  },
])


// or if you have an input instance
app.inputs.get({id}).then(
  function(input) {
    input.mergeConcepts([
      {
        id: "tree",
        value: true
      },
      {
        id: "water",
        value: false
      }
    ])
  },
  function(err) {
    // there was an error
  }
);

app.inputs.merge_concepts('{id}', concepts=['tree'], not_concepts=['water'])

client.mergeConceptsForInput("{input_id}")
    .plus(
        Concept.forID("tree"),
        Concept.forID("water").withValue(false)
    )
    .executeSync();

ClarifaiConcept *concept = [[ClarifaiConcept alloc] initWithConceptName:@"cute cat"];
[_app addConcepts:@[concept] forInputWithID:@"{id}" completion:^(ClarifaiInput *input, NSError *error) {
    NSLog(@"input: %@", input);
}];

curl -X PATCH \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "id": "{id}",
        "data": {
          "concepts": [
            {
              "id": "tree",
              "value": true
            },
            {
              "id": "water",
              "value": false
            }
          ]
        }
      }
    ],
    "action":"merge"
}'\
  https://api.clarifai.com/v2/inputs

Delete Concepts From An Input

To remove concepts that were already added to an input, you can do this:


app.inputs.deleteConcepts([
  {
    id: "{id}",
    concepts: [
      {
        id: "tree"
      },
      {
        id: "water",
        value: false
      }
    ]
  },
])

// or if you have an input instance
app.inputs.get({id}).then(
  function(input) {
    input.deleteConcepts([
      {
        id: "tree",
        value: true
      },
      {
        id: "water",
        value: false
      }
    ])
  },
  function(err) {
    // there was an error
  }
);

app.inputs.delete_concepts({id}, concepts=['tree', 'water'])

client.removeConceptsForInput("{input_id}")
    .plus(
        Concept.forID("tree"),
        Concept.forID("water")
    )
    .executeSync();

ClarifaiConcept *concept = [[ClarifaiConcept alloc] initWithConceptName:@"cute cat"];
[app deleteConcepts:@[concept] forInputWithID:{id} completion:^(ClarifaiInput *input, NSError *error) {
    NSLog(@"input: %@", input);
}];

curl -X PATCH \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "id":"",
        "data": {
            "concepts":[
                {"id":"mattid2", "value":true},
                {"id":"ferrari", "value":false}
            ]
        }
      }
    ],
    "action":"remove"
  }'\
  https://api.clarifai.com/v2/inputs/

Bulk Update Inputs With Concepts

You can update an existing input using its Id. This is useful if you'd like to add concepts to an input after its already been added.


app.inputs.mergeConcepts([
  {
    id: "{id1}",
    concepts: [
      {
        id: "tree",
        value: true
      },
      {
        id: "water",
        value: false
      }
    ]
  },
  {
    id: "{id2}",
    concepts: [
      {
        id: "animal",
        value: true
      },
      {
        id: "fruit",
        value: false
      }
    ]
  }
]).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# bulk merge concepts
input_ids = ["{id1}", "{id2}"]
concept_pairs = [
                 [('tree', True), ('water', False)],
                 [('animal', True), ('fruit', False)],
                ]
app.inputs.bulk_merge_concepts(input_ids, concept_pairs)

client.mergeConceptsForInput("{input_id}")
    .plus(
        Concept.forID("tree"),
        Concept.forID("water").withValue(false)
    )
    .executeSync();
ClarifaiConcept *newConcept = [[ClarifaiConcept alloc] initWithConceptID:@"tree"];
[_app getInput:@"{input_id}" completion:^(ClarifaiInput *input, NSError *error) {
  // Add tree concept to each current input's concept list.
  NSMutableArray *newConceptList = [NSMutableArray arrayWithArray:input.concepts];
  [newConceptList addObject:newConcept];
  input.concepts = newConceptList;

  // Merge the new list for one or more inputs.
  [_app mergeConceptsForInputs:@[input] completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error)   {
    NSLog(@"updated inputs: %@", inputs);
  }];
}];

curl -X PATCH \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "id": "{id1}",
        "data": {
          "concepts": [
            {
              "id": "tree",
              "value": true
            },
            {
              "id": "water",
              "value": false
            }
          ]
        }
      },
      {
        "id": "{id2}",
        "data": {
          "concepts": [
            {
              "id": "tree",
              "value": true
            },
            {
              "id": "water",
              "value": false
            }
          ]
        }
      }
    ],
    "action":"merge"
}'\
  https://api.clarifai.com/v2/inputs

Bulk Delete Concepts From A List Of Inputs

You can bulk delete multiple concepts from a list of inputs:


app.inputs.deleteConcepts([
  {
    id: "{id1}",
    concepts: [
      { id: "tree" },
      { id: "water" }
    ]
  },
  {
    id: "{id2}",
    concepts: [
      { id: "animal" },
      { id: "fruit" }
    ]
  }
]).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

input_ids = ["{id1}", "{id2}"]
concept_pairs = [
                 ['tree', 'water'],
                 ['animal', 'fruit']
                ]
app.inputs.bulk_delete_concepts(input_ids, concept_pairs)

// Coming soon

ClarifaiConcept *concept = [[ClarifaiConcept alloc] initWithConceptName:@"cute cat"];
[app deleteConcepts:@[concept] forInputWithID:input_id completion:^(ClarifaiInput *input, NSError *error) {
    NSLog(@"input: %@", input);
}];

curl -X PATCH \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "id": "{id1}",
        "data": {
          "concepts":[
            {
              "id": "mattid2"
            },
            {
              "id": "ferrari"
            }
          ]
        }
      },
      {
        "id": "{id2}",
        "data": {
          "concepts":[
            {
              "id": "mattid2"
            },
            {
              "id": "ferrari"
            }
          ]
        }
      }
    ],
    "action":"remove"
  }'\
  https://api.clarifai.com/v2/inputs

Delete Input By Id

You can delete a single input by id:


app.inputs.delete({id}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

app.inputs.delete("{id}")

client.deleteInput("{id}")
    .executeSync();

[_app deleteInputsByIDList:@[{id1}] completion:^(NSError *error) {
    NSLog(@"input has been deleted");
}];

curl -X DELETE \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/inputs/{id}

Delete A List Of Inputs

You can also delete multiple inputs in one API call. This will happen asynchronously.


app.inputs.delete([{id1}, {id2}]).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

app.delete(["{id1}", "{id2}"])

client.deleteInputsBatch()
    .plus("{id1}", "{id2}");
    .executeSync();

[_app deleteInputsByIDList:@[{id1}, {id2}] completion:^(NSError *error) {
    NSLog(@"inputs have been deleted");
}];

curl -X DELETE \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "ids":["{id1}","{id2}"]
  }'\
  https://api.clarifai.com/v2/inputs

Delete All Inputs

If you would like to delete all inputs from an application, you can do that as well. This will happen asynchronously.


app.inputs.delete().then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

app.inputs.delete_all()

client.deleteAllInputs().executeSync();

[app deleteAllInputs:^(ClarifaiInput *input, NSError *error) {
  NSLog(@"all inputs have been deleted");
}];

curl -X DELETE \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "delete_all":true
  }'\
  https://api.clarifai.com/v2/inputs

Models

There are many methods to work with models.

Create Model

You can create your own model and train it with your own images and concepts. Once you train it to see how you would like it to see, you can then use that model to make predictions.

When you create a model you give it a name and an id. If you don't supply an id, one will be created for you.


app.models.create("petsID").then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

app.models.create('petsID')

client.createModel("petsID").executeSync();

[_app createModel:nil name:@"petsModel" modelID:@"petsID" conceptsMutuallyExclusive:NO closedEnvironment:NO completion:^(ClarifaiModel *model, NSError *error) {
    NSLog(@"model: %@", model);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "model": {
      "id": "petsID"
    }
  }'\
  https://api.clarifai.com/v2/models

Create Model With Concepts

You can also create a model and initialize it with the concepts it will contain. You can always add and remove concepts later.


app.models.create(
  "petsID",
  [
    { "id": "boscoe" }
  ]
).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

model = app.models.create('petsID', concepts=['boscoe'])

client.createModel("petsID")
    .withOutputInfo(ConceptOutputInfo.forConcepts(
        Concept.forID("boscoe")
    ))
    .executeSync();

[_app createModel:@[@"cat", @"dog"] name:@"petsModel" modelID:@"petsID" conceptsMutuallyExclusive:NO closedEnvironment:NO completion:^(ClarifaiModel *model, NSError *error) {
    NSLog(@"model: %@", model);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "model": {
      "id": "petsID",
      "output_info": {
        "data": {
          "concepts": [
            {
              "id": "boscoe"
            }
          ]
        },
        "output_config": {
          "concepts_mutually_exclusive": false,
          "closed_environment":false
        }
      }
    }
  }'\
  https://api.clarifai.com/v2/models

Add Concepts To A Model

You can add concepts to a model at any point. As you add concepts to inputs, you may want to add them to your model.


app.models.initModel({model_id}).then(function(model) {
  updateModel,
  function(err) {
    // there was an error
  }
});

function updateModel(model) {
  model.mergeConcepts({"id": "boscoe"}).then(
    function(response) {
      // do something with response
    },
    function(err) {
      // there was an error
    }
  );
}

model = app.models.get('{model_id}')
model.add_concepts(['boscoe'])

client.modifyModel("")
    .withConcepts(Action.MERGE, Concept.forID("dogs"))
    .executeSync();

// Or, if you have a ConceptModel object, you can do it in an OO fashion
final ConceptModel model = client.getModelByID("{model_id}").executeSync().get().asConceptModel();
model.modify()
    .withConcepts(Action.MERGE, Concept.forID("dogs"))
    .executeSync();

ClarifaiConcept *concept = [[ClarifaiConcept alloc] initWithConceptName:@"dress"];
[app addConcepts:@[concept] toModelWithID:@"{model_id}" completion:^(ClarifaiModel *model, NSError *error) {
    NSLog(@"model: %@", model);
}];

curl -X PATCH \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "models": [
      {
        "id": "{model_id}",
        "output_info": {
          "data": {
            "concepts": [
              {
                "id": "dogs"
              }
            ]
          }
        }
      }
    ],
    "action": "merge"
  }'\
  https://api.clarifai.com/v2/models/

Remove Concepts From A Model

Conversely, if you'd like to remove concepts from a model, you can also do that.


app.models.initModel({model_id}).then(function(model) {
  updateModel,
  function(err) {
    // there was an error
  }
});

function updateModel(model) {
  model.deleteConcepts({"id": "boscoe"}).then(
    function(response) {
      // do something with response
    },
    function(err) {
      // there was an error
    }
  );
}

model = app.models.get('{model_id}')
model.delete_concepts(['boscoe'])

client.modifyModel("")
    .withConcepts(Action.REMOVE, Concept.forID("dogs"))
    .executeSync();

// Or, if you have a ConceptModel object, you can do it in an OO fashion
final ConceptModel model = client.getModelByID("").executeSync().get().asConceptModel();
model.modify()
    .withConcepts(Action.REMOVE, Concept.forID("dogs"))
    .executeSync();

ClarifaiConcept *concept = [[ClarifaiConcept alloc] initWithConceptName:@"dress"];
[app deleteConcepts:@[concept] fromModelWithID:@"{model_id}" completion:^(ClarifaiModel *model, NSError *error) {
    NSLog(@"model: %@", model);
}];

curl -X PATCH \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "models": [
      {
        "id": "{model_id}",
        "output_info": {
          "data": {
            "concepts": [
              {
                "id": "dogs"
              }
            ]
          }
        }
      }
    ],
    "action": "remove"
  }'\
  https://api.clarifai.com/v2/models/

Update Concept Name

The code below showcases how to update a concept's name given its id.

** Coming Soon
app.concepts.update(concept_id='concept_id', concept_name='new_concept_name')
** Coming Soon
** Coming Soon

curl -X PATCH \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "concepts": [
      {
        "id": "{concept_id}",
        "name": "{new_concept_name}"
      }
      ],
    "action": "overwrite"
  }'\
  https://api.clarifai.com/v2/concepts

Update Model Name and Configuration

Here we will change the model name to 'newname' and the model's configuration to have concepts_mutually_exclusive=true and closed_environment=true.


app.models.initModel({model_id}).then(
  updateModel,
  function(err) {
    // there was an error
  }
);

function updateModel(model) {
  model.update({
    name: 'newname',
    conceptsMutuallyExclusive: true,
    closedEnvironment: true,
    concepts: ['birds', 'hurd']
  }).then(
}

model = app.models.get('{model_id}')

# only update the name
model.update(model_name="newname")

# update the model attributes
model.update(concepts_mutually_exclusive=True, closed_environment=True)

# update more together
model.update(model_name="newname",
             concepts_mutually_exclusive=True, closed_environment=True)

# update attributes together with concepts
model.update(model_name="newname",
             concepts_mutually_exclusive=True,
             concepts=["birds", "hurd"])

client.modifyModel("")
    .withName("newname")
    .withConceptsMutuallyExclusive(true)
    .withClosedEnvironment(true)
    .executeSync();

[_app updateModel:@"{model_id}" name:@"newName" conceptsMutuallyExclusive:NO closedEnvironment:NO completion:^(ClarifaiModel *model, NSError *error) {
    NSLog(@"model: %@", model);
}];

curl -X PATCH \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "models": [
      {
        "id": "",
        "name": "newname",
        "output_info": {
          "output_config": {
            "concepts_mutually_exclusive": true,
            "closed_environment": true
          }
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/models/

Get Models

To get a list of all models including models you've created as well as public models:


app.models.list().then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# this is a generator
app.models.get_all()

client.getModels().getPage(1).executeSync();

[_app getModels:1 resultsPerPage:30 completion:^(NSArray<ClarifaiModel *> *models, NSError *error) {
    NSLog(@"models: %@", models);
}];

curl -X GET \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/models

Get Model By Id

All models have unique Ids. You can get a specific model by its id:


app.models.get({model_id}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# get model by id
model = app.models.get(model_id')

# get model by name
model = app.models.get('my_model1')

client.getModelByID("{model_id}").executeSync();

[_app getModel:@"model_id" completion:^(ClarifaiModel *model, NSError *error) {
    NSLog(@"model: %@", model);
}];

curl -X GET \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/models/{model_id}

Get Model Output Info By Id

The output info of a model lists what concepts it contains.


app.models.initModel({model_id}).then(
  getModelOutputInfo,
  handleError
);

function getModelOutputInfo(model) {
  model.getOutputInfo().then(
    function(response) {
      // do something with response
    },
    function(err) {
      // there was an error
    }
  );
}

model = app.models.get('my_model1')
model.get_info(verbose=True)

client.getModelByID("{model_id}").executeSync();

[_app getModelByID:@"{model_id}" completion:^(ClarifaiModel *model, NSError *error) {
    NSLog(@"model: %@", model);
}];

curl -X GET \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/models/{model_id}/output_info

List Model Versions

Every time you train a model, it creates a new version. You can list all the versions created.


app.models.initModel('{id}').then(
  function(model) {
    model.getVersions().then(
      function(response) {
        // do something with response
      },
      function(err) {
        // there was an error
      }
    );
  },
  function(err) {
    // there was an error
  }
);

model = app.models.get('{id}')
model.list_versions()

client.getModelVersions("{model_id}").getPage(1).executeSync();

[app listVersionsForModel:@"{model_id}" page:1 resultsPerPage:30 completion:^(NSArray<ClarifaiModelVersion *> *versions, NSError *error) {
    NSLog(@"versions: %@", versions);
}];

curl -X GET \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/models/{model_id}/versions

Get Model Version By Id

To get a specific model version, you must provide the model_id as well as the version_id. You can inspect the model version status to determine if your model is trained or still training.


app.models.initModel('{id}').then(
  function(model) {
    model.getVersion('{version_id}').then(
      function(response) {
        // do something with response
      },
      function(err) {
        // there was an error
      }
    );
  },
  function(err) {
    // there was an error
  }
);

model = app.models.get('{id}')
model.get_version('{version_id}')

client.getModelVersionByID("{model_id}", "{version_id}").executeSync();

// Or in a more object-oriented manner:
client.getModelByID("{model_id}")
    .executeSync().get() // Returns Model object
    .getVersionByID("{version_id}").executeSync();

[app getVersionForModel:@"{model_id}" versionID:{version_id} completion:^(ClarifaiModelVersion *version, NSError *error) {
    NSLog(@"version: %@", version);
}];

curl -X GET \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/models/{model_id}/versions/{version_id}

Get Model Training Inputs

You can list all the inputs that were used to train the model.


app.models.initModel('{id}').then(
  function(model) {
    model.getInputs().then(
      function(response) {
        // do something with response
      },
      function(err) {
        // there was an error
      }
    );
  },
  function(err) {
    // there was an error
  }
);

model = app.models.get('{id}')
model.get_inputs()

client.getModelInputs("{model_id}").getPage(1).executeSync();

[app listTrainingInputsForModel:@"{model_id}" page:1 resultsPerPage:30 completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
    NSLog(@"inputs: %@", inputs);
}];

curl -X GET \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/models/{model_id}/inputs

Get Model Training Inputs By Version

You can also list all the inputs that were used to train a specific model version.


app.models.initModel({id: '{model_id}', version: '{version_id}'}).then(
  function(model) {
    model.getInputs().then(
      function(response) {
        // do something with response
      },
      function(err) {
        // there was an error
      }
    );
  },
  function(err) {
    // there was an error
  }
);

model = app.models.get('{id}')
model.get_inputs('{version_id}')

client.getModelInputs("{model_id}")
    .fromSpecificModelVersion("{version_id}")
    .getPage(1)
    .executeSync();

[_app listTrainingInputsForModel:@"{model_id}" page:1 resultsPerPage:30 completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
    NSLog(@"inputs: %@", inputs);
}];

curl -X GET \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/models/{model_id}/versions/{version_id}/inputs

Delete A Model

You can delete a model using the model_id.


app.models.delete('{id}').then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

app.models.delete('{id}')

client.deleteModel("{model_id}").executeSync();

[app deleteModel:@"{model_id}" completion:^(NSError *error) {
    NSLog(@"model is deleted");
}];

curl -X DELETE \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/models/{model_id}

Delete A Model Version

You can also delete a specific version of a model with the model_id and version_id.


app.models.delete('{model_id}', '{version_id}').then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

app.models.delete('{id}', '{version_id}')

# or

model = app.models.get('{id}')
model.delete_version('{version_id}')

client.deleteModelVersion("{model_id}", "{version_id}").executeSync();

// Or in a more object-oriented manner:
client.getModelByID("{model_id}")
    .executeSync().get() // Returns Model object
    .deleteVersion("{version_id}")
    .executeSync();

[app deleteVersionForModel:{model_id} versionID:{version_id} completion:^(NSError *error) {
    NSLog(@"model version deleted");
}];

curl -X DELETE \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/models/{model_id}/versions/{version_id}

Delete All Models

If you would like to delete all models associated with an application, you can also do that. Please proceed with caution as these cannot be recovered.


app.models.delete().then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

app.models.delete_all()

client.deleteAllModels().executeSync();

[_app deleteAllModels:^(NSError *error) {
    NSLog(@"delete all models");
}];

curl -X DELETE \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/models/

Train A Model

When you train a model, you are telling the system to look at all the images with concepts you've provided and learn from them. This train operation is asynchronous. It may take a few seconds for your model to be fully trained and ready.

Note: you can repeat this operation as often as you like. By adding more images with concepts and training, you can get the model to predict exactly how you want it to.


app.models.train("{model_id}").then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

// or if you have an instance of a model

model.train().then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

model = app.models.get('{model_id}')
model.train()

client.trainModel("{model_id}").executeSync();

ClarifaiImage *image = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/puppy.jpeg"]
[app getModel:@"{id}" completion:^(ClarifaiModel *model, NSError *error) {
    [model train:^(ClarifaiModel *model, NSError *error) {
        NSLog(@"model: %@", model);
    }];
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  https://api.clarifai.com/v2/models/{model_id}/versions

Predict With A Model

Once you have trained a model you are ready to use your new model to get predictions. The predictions returned will only contain the concepts that you told it to see.


app.models.predict("{model_id}", ["https://samples.clarifai.com/puppy.jpeg"]).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

// or if you have an instance of a model

model.predict("https://samples.clarifai.com/puppy.jpeg").then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

from clarifai.rest import Image as ClImage

model = app.models.get('{model_id}')

image = ClImage(url='https://samples.clarifai.com/puppy.jpeg')
model.predict([image])

client.predict("{model_id}")
    .withInputs(
        ClarifaiInput.forImage("https://samples.clarifai.com/puppy.jpeg")
    )
    .executeSync();

ClarifaiImage *image = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/puppy.jpeg"]
[app getModel:@"{model_id}" completion:^(ClarifaiModel *model, NSError *error) {
    [model predictOnImages:@[image]
                completion:^(NSArray<ClarifaiSearchResult *> *outputs, NSError *error) {
                    NSLog(@"outputs: %@", outputs);
                }];
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/puppy.jpeg"
          }
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/models/{model_id}/outputs

Search Models By Name And Type

You can search all your models by name and type of model.


app.models.search('general-v1.3', 'concept').then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# search model name
app.models.search('general-v1.3')

# search model name and type
app.models.search(model_name='general-v1.3', model_type='concept')

client.findModel()
    .withModelType(ModelType.CONCEPT)
    .withName("general-v1.3")
    .getPage(1)
    .executeSync();

[app searchForModelByName:@"general-v1.3" modelType:ClarifaiModelTypeConcept completion:^(NSArray<ClarifaiModel *> *models, NSError *error) {
    NSLog(@"models: %@", models);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "model_query": {
      "name": "general-v1.3",
      "type": "concept"
    }
  }'\
  https://api.clarifai.com/v2/models/searches

Evaluation

Interpreting Results

model eval results Once the Model Evaluation is complete, you’ll be able to view the evaluation results in the Explorer here

We suggest that you start by looking at the Concept by Concept Probabilities Results and the Summary Table to get a sense of the overall model prediction performance and identify the high-performing and low-performing concepts. Afterwards, hone in on the Selection Details of the False Positives and False Negatives to identify any biases, if any, in how the model is predicting, and to correct any inputs that are mislabeled

Generally, you’ll be looking at results that represent either a) the average across 5 splits, or b) the test set of a single split, which is about 20% of your original training set. Note that a single split will be capped at 1,000 inputs.

Model Analysis

Model Accuracy Score

model eval results Model Accuracy Score is the highest level metric for your model’s prediction performance. It is defined as the macro average of the areas under the receiver operating characteristic curve for every concept. This metric does not depend on the Prediction Threshold. This metric is an average across 5 splits.

A score of 1 represents a perfect model; a score of .5 represents a worthless model. As a general rule of thumb, a score above .9 is considered good.

Note that we discourage users from making a final assessment of the model accuracy based on the Model Accuracy Score only.

Concept Analysis

Prediction Threshold

prediction threshold Probability threshold determines the model’s predictions. The default threshold is .5. The input is predicted as (i.e. “counts” as) as a concept, such as “dog”, only if the prediction probability for “dog” is higher than the set threshold, for example, 0.5. You can adjust the threshold depending on how ‘strict’ you want your classification to be.

All prediction binary metrics, such as True Positives, False Negatives, and False Positives, and Total Predicted, Recall Rate, Precision Rate, depend on this threshold.

Evaluation Summary

model eval summary This table summarizes the numerical evaluation results for every concept. For every concept, it calculates:

5-Split Average

  1. ROC AUC (Concept Accuracy Score): concept’s prediction performance score, defined the area under the receiver operating characteristic curve. A score of 1 represents a perfect model; a score of .5 represents a worthless model. As a general rule of thumb, a score above .9 is considered good. Note: ROC AUC is not dependent on the prediction threshold.

1-Split

  1. Total Labeled: Total number of inputs that were originally labeled as the concept in the test set. Total Labeled is the sum of True Positives (correct) and False Negatives (incorrect). Note: Total Labeled is not dependent on the prediction threshold.
  2. Total Predicted: Total number of inputs that were predicted as the concept in the test set. This means these inputs were predicted as a concept with probability greater than the prediction threshold value. Total Predicted is the sum of True Positives (correct) and False Positives (incorrect).
  3. True Positives (TP): Number of inputs that were correctly predicted as the concept they were actually labeled. Also known as “hits”. (E.g. These are the images that were labeled as “dog” and were predicted as “dog”)
  4. False Negatives (FN): Number of inputs that were not predicted as the concept they were actually labeled. Also known as “misses”. (E.g. These are the images that were labeled as “dog” but were not predicted as “dog”)
  5. False Positives (FP): Number of inputs that were predicted as the concept, but they were not labeled as the concept. Also known as “false alarms”. (E.g. These are the images that were predicted as “dog” but were not labeled as “dog”)
  6. Recall Rate: proportion of the images labeled as the concept that were predicted as the concept. It is calculated as True Positives divided by Total Labeled. Also known as “sensitivity” or “true positive rate”.
  7. Precision Rate: proportion of the images predicted as a concept that had been actually labeled as the concept. It is calculated as True Positives divided by Total Predicted. Also known as “positive predictive value”.
Concept by Concept Results (Advanced)

This section has a concept by concept matrix. Note that this is not a confusion matrix; we recommend that you read this through before interpreting the data.

In general, the matrix is meant to be read by fixing each row. Each row represents a subset of the test set that was actually labeled as a particular concept. For this subset, each cell across the row represents either

  1. The number of inputs that were predicted as a concept (i.e. “counts”), or
  2. The average prediction probability for each concept, noted by the column name, for all inputs in this subset, across all 5 splits (i.e. “probabilities”).

Concepts that co-occur, or are similar, may form a visual cluster on the matrix. On the other hand, exclusive or dissimilar concepts should not form a cluster.

Counts (1-Split) concept by concept results Each row represents the subset of the test set that were actually labeled as a concept, e.g. “dog”. As you go across the row, each cell shows the number of times those images were predicted as each concept, noted by the column name.

The diagonal cells represent True Positives, i.e. correctly predicted inputs. You’d want this number to be as close to the Total Labeled as possible.

Depending on how your model was trained, the off-diagonal cells could include both correct and incorrect predictions. In a non-mutually exclusive concepts environment, you can label an image with more than 1 concept. For example, an image is labeled as both “dog” and “cat”, this image would be counted in both “dog” row and “cat” row. If the model correctly predicts this image to be both “dog” and “cat”, then this input will be counted in both on and off-diagonal cells.

Few things to note:

  1. Remember that the prediction counts depend on the threshold. This means the images are counted toward a predicted concept only if the prediction probability for this concept is higher than the threshold.
  2. This means the sum of the row may be less or greater than the # of total labeled inputs that were labeled as the concept.
  3. You can click on each cell to view the actual prediction results for every input that was counted in this cell.
  4. This represents the test set data of a single split.

Probabilities (5-Split Average) concept by concept five split results Each row represents the subset of the test set that were actually labeled as a concept, e.g. “dog”. As you go across the row, each cell shows the average prediction probability for each concept, noted by the column name, for all inputs in this subset. In short, the cell shows the average prediction probability for a concept given the images labeled as a concept.

Few things to note:

  1. This matrix does not depend on the Prediction Threshold.
  2. You can click on each cell to view the actual prediction results for every input that were used to calculate this cell.
  3. This represents the average across all 5 splits.

Input Analysis

Selection Details

not expanded This Selection Details table shows the input-level details of the selection you made on the Summary Table or Concept by Concept Results. It shows the image input and prediction probabilities for a specific concept.

Note: the prediction probabilities on this table may seem different from your actual model’s probabilities. The reason is that all the evaluation results are based on the new model that was built for evaluation purposes during the cross validation process.

Expanded Selection Details

expanded You can click on the expand button to view the prediction probabilities for every concept for each image. The blue dot denotes the concept(s) the input was actually labeled, i.e. true positives. The red dot denotes the concept(s) that the input was not labeled.

Improving your Model

The evaluation metrics are meant to help you diagnose the quality of your model. Your model may belong to one or more of many categories, including, but not limited to:

  1. Good model with all great concepts.
  2. OK model with a few bad concepts.
  3. Bad model: with all bad concepts.
  4. Biased model: the model is consistently picking up certain visual cues other than what you’d like to pick up. This is sometimes obvious to pick up in the Explorer here .
  5. Model with Variance: there is no consistency in the way the model is predicting on inputs.

Possible Areas of Improvement

The performance of your model depends on the performance of each concept, which is trained on a set of inputs. We’d recommend that you look at both inputs and concepts when diagnosing areas of improvement.

Inputs

  1. Diversity: try to include all perspectives of the concept, e.g. include all angles of a “dog”, if you’re building a “dog” concept.
  2. Strong positives: Images that are the true representation of your concept.
  3. Training data should be representative of the real world data -- avoid making models where the data is too ‘easy’, i.e. unrealistic set of data.
  4. Number: minimum 50 inputs per concept; more inputs the better.
  5. File dimensions: minimum 512px x 512px.

Concepts

  1. Concepts: avoid concepts that do not rely on visual cues within the image. Also, current custom training does not perform well on training to identify faces.
  2. Labels: check to see if any inputs are labeled with wrong concepts.

Tips

When improving your model, there is no one-size-fits-all answer. Here are some tips to keep in mind:

  1. Although we use ROC AUC as a general top-level ‘score’ for both concept and model, we do not recommend that you rely on 1 metric only to draw your final conclusion on your model performance.
  2. Refer to both Concepts by Concepts Results as well as Selection Details to get a better grasp of your model.
  3. When interpreting the evaluation results, keep in mind the nature of your model. Specifically, pay attention to whether or not you have labeled the inputs with more than 1 concept (i.e. non-mutually exclusive concepts environment), vs. only 1 concept per image.
  4. Remember, the rule of diminishing returns may also apply to training models. After a certain point, the changes may not make a big difference in the model quality.

Predictions

You can set additional parameters to gain flexibility in the predict operation.

Select Concepts

By putting this additional parameter on your predict calls, you can receive predict value(s) for only the concepts that you want to. You can specify particular concepts by either their id and/or their name. The concept names and ids are case sensitive, and so, these must be exact matches.

To retrieve an entire list of concepts from a given model use the GET /v2/models/{model_id}/output_info endpoint. Check out the Advanced Models section for how to use with any of the API clients!

If you submit a request with not an exact match of the concept id or name, you will receive an invalid model argument error. However, if one or more matches while one or more do not, the API will respond with a Mixed Success.


clarifai.models.predict(Clarifai.GENERAL_MODEL, 'https://samples.clarifai.com/metro-north.jpg', {
  selectConcepts: [
    {name: 'train'},
    {id: 'ai_6kTjGfF6'}
  ]
})
from clarifai.rest import Concept

model = app.models.get('general-v1.3')

select_concept_list = [Concept(concept_name='train'), Concept(concept_id='ai_6kTjGfF6')]
model.predict_by_url(url='https://samples.clarifai.com/metro-north.jpg', select_concepts=select_concept_list)
client.predict(client.getDefaultModels().generalModel().id())
    .withInputs(ClarifaiInput.forImage("https://samples.clarifai.com/metro-north.jpg"))
    .selectConcepts(Concept.forID("dog"), Concept.forID("cat"))
    .executeSync();
// Coming Soon
curl -X POST \
  -H 'authorization: Key YOUR_API_KEY' \
  -H 'content -type: application/json' \
  -d '{
  "inputs": [
    {
      "data": {
        "image": {
          "url": "https://samples.clarifai.com/metro-north.jpg"
        }
      }
    }
  ],
  "model": {
    "output_info": {
      "output_config": {
        "select_concepts": [
          {"name": "train"},
          {"id": "ai_6kTjGfF6"}
        ]
      }
    }
  }
}'\
https://api.clarifai.com/v2/models/aaa03c23b3724a16a56b629203edc62c/outputs
Show All
Response{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "outputs": [
    {
      "id": "c8abf5cbe52746efa9df8a2319d49d0a",
      "status": {
        "code": 10000,
        "description": "Ok"
      },
      "created_at": "2017-06-27T13:31:57.493797045Z",
      "model": {
        "id": "aaa03c23b3724a16a56b629203edc62c",
        "name": "general-v1.3",
        "created_at": "2016-03-09T17:11:39.608845Z",
        "app_id": "main",
        "output_info": {
          "message": "Show output_info with: GET /models/{model_id}/output_info",
          "type": "concept",
          "type_ext": "concept"
        },
        "model_version": {
          "id": "aa9ca48295b37401f8af92ad1af0d91d",
          "created_at": "2016-07-13T01:19:12.147644Z",
          "status": {
            "code": 21100,
            "description": "Model trained successfully"
          }
        }
      },
      "input": {
        "id": "c613b3254da34382b2fca65365da7c49",
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/metro-north.jpg"
          }
        }
      },
      "data": {
        "concepts": [
          {
            "id": "ai_HLmqFqBf",
            "name": "train",
            "value": 0.9989112,
            "app_id": "main"
          },
          {
            "id": "ai_6kTjGfF6",
            "name": "station",
            "value": 0.992573,
            "app_id": "main"
          }
        ]
      }
    }
  ]
}

Maximum Concepts

Setting the max concepts parameter will customize how many concepts and their corresponding probability scores the predict endpoint will return. If not specified, the predict endpoint will return the top 20 concepts. You can currently set the max concepts parameter to any number in the range: [1-200]. If your use case requires more concepts, please contact Support.


clarifai.models.predict(Clarifai.GENERAL_MODEL, 'https://samples.clarifai.com/metro-north.jpg', { maxConcepts: 3 })
  .then(log)
  .catch(log);

from clarifai.rest import ClarifaiApp

app = ClarifaiApp()

model = app.models.get('general-v1.3')

model.predict_by_url(url='https://samples.clarifai.com/metro-north.jpg', max_concepts=3)

client.predict(client.getDefaultModels().generalModel().id())
    .withInputs(ClarifaiInput.forImage("https://samples.clarifai.com/metro-north.jpg"))
    .withMaxConcepts(3)
    .executeSync();

//Coming soon

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/metro-north.jpg"
          }
        }
      }
    ],
    "model":{
      "output_info":{
        "output_config":{
          "max_concepts": 3
        }
      }
    }
  }'\
  https://api.clarifai.com/v2/models/aaa03c23b3724a16a56b629203edc62c/outputs
Show All
Response
{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "outputs": [
    {
      "id": "c8c400234b0d47df9084857df0d69efb",
      "status": {
        "code": 10000,
        "description": "Ok"
      },
      "created_at": "2017-06-15T16:09:48.984389535Z",
      "model": {
        "id": "aaa03c23b3724a16a56b629203edc62c",
        "name": "general-v1.3",
        "created_at": "2016-02-26T23:38:40.086101Z",
        "app_id": "main",
        "output_info": {
          "message": "Show output_info with: GET /models/{model_id}/output_info",
          "type": "concept",
          "type_ext": "concept"
        },
        "model_version": {
          "id": "aa9ca48295b37401f8af92ad1af0d91d",
          "created_at": "2016-07-13T00:58:55.915745Z",
          "status": {
            "code": 21100,
            "description": "Model trained successfully"
          }
        }
      },
      "input": {
        "id": "fd99d9e345f3495a8bd2802151d09efa",
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/metro-north.jpg"
          }
        }
      },
      "data": {
        "concepts": [
          {
            "id": "ai_HLmqFqBf",
            "name": "train",
            "value": 0.9989112,
            "app_id": "main"
          },
          {
            "id": "ai_fvlBqXZR",
            "name": "railway",
            "value": 0.9975532,
            "app_id": "main"
          },
          {
            "id": "ai_Xxjc3MhT",
            "name": "transportation system",
            "value": 0.9959158,
            "app_id": "main"
          }
        ]
      }
    }
  ]
}

Minimum Prediction Value

This parameter lets you set a minimum probability threshold for the outputs you want to view for the Predict operation. For example if you want to see all concepts with a probability score of .90 or higher, this parameter will allow you to accomplish that. Also note that if you don't specify the number of max concepts, you will only see the top 20. If your result can contain more values you will have to increase the number of maximum concepts as well.


clarifai.models.predict(Clarifai.GENERAL_MODEL, 'https://samples.clarifai.com/metro-north.jpg', { minValue: 0.97 })
  .then(log)
  .catch(log);

from clarifai.rest import ClarifaiApp

app = ClarifaiApp()

model = app.models.get('general-v1.3')

model.predict_by_url(url='https://samples.clarifai.com/metro-north.jpg', min_value=0.97)

client.predict(client.getDefaultModels().generalModel().id())
    .withInputs(ClarifaiInput.forImage("https://samples.clarifai.com/metro-north.jpg"))
    .withMinValue(0.9)
    .executeSync();

//Coming soon

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/metro-north.jpg"
          }
        }
      }
    ],
    "model":{
      "output_info":{
        "output_config":{
          "min_value": 0.97
        }
      }
    }
  }'\
  https://api.clarifai.com/v2/models/aaa03c23b3724a16a56b629203edc62c/outputs
Show All
Response
{
  "status": {
    "code": 10000,
    "description": "Ok"
  },
  "outputs": [
    {
      "id": "b2027bccf4964d03b062ce653cff85b6",
      "status": {
        "code": 10000,
        "description": "Ok"
      },
      "created_at": "2017-06-15T20:22:05.841603659Z",
      "model": {
        "id": "aaa03c23b3724a16a56b629203edc62c",
        "name": "general-v1.3",
        "created_at": "2016-02-26T23:38:40.086101Z",
        "app_id": "main",
        "output_info": {
          "message": "Show output_info with: GET /models/{model_id}/output_info",
          "type": "concept",
          "type_ext": "concept"
        },
        "model_version": {
          "id": "aa9ca48295b37401f8af92ad1af0d91d",
          "created_at": "2016-07-13T00:58:55.915745Z",
          "status": {
            "code": 21100,
            "description": "Model trained successfully"
          }
        }
      },
      "input": {
        "id": "f7640568d37f47fbba9d6fdc892ec64d",
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/metro-north.jpg"
          }
        }
      },
      "data": {
        "concepts": [
          {
            "id": "ai_HLmqFqBf",
            "name": "train",
            "value": 0.9989112,
            "app_id": "main"
          },
          {
            "id": "ai_fvlBqXZR",
            "name": "railway",
            "value": 0.9975532,
            "app_id": "main"
          },
          {
            "id": "ai_Xxjc3MhT",
            "name": "transportation system",
            "value": 0.9959158,
            "app_id": "main"
          },
          {
            "id": "ai_6kTjGfF6",
            "name": "station",
            "value": 0.992573,
            "app_id": "main"
          },
          {
            "id": "ai_RRXLczch",
            "name": "locomotive",
            "value": 0.992556,
            "app_id": "main"
          },
          {
            "id": "ai_VRmbGVWh",
            "name": "travel",
            "value": 0.98789215,
            "app_id": "main"
          },
          {
            "id": "ai_SHNDcmJ3",
            "name": "subway system",
            "value": 0.9816359,
            "app_id": "main"
          },
          {
            "id": "ai_jlb9q33b",
            "name": "commuter",
            "value": 0.9712483,
            "app_id": "main"
          }
        ]
      }
    }
  ]
}

Searches

By Public Concepts

When you add an input, it automatically gets predictions from the general model. You can search for those predictions.


app.inputs.search([
  {
    concept: {
      name: 'cat'
    }
  },
  {
    concept: {
      name: 'dog'
    }
  }
]).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# search by single concept name
app.inputs.search_by_predicted_concepts(concept='cat')

# search by single concept id
app.inputs.search_by_predicted_concepts(concept_id='ai_mFqxrph2')

# search by multiple concepts with name
app.inputs.search_by_predicted_concepts(concepts=['cat', 'cute'])

# search by multiple concepts with ids
app.inputs.search_by_predicted_concepts(concept_ids=['ai_mFqxrph2', 'ai_4CRlSvbV'])

# search by multiple concepts with not logic
app.inputs.search_by_predicted_concepts(concepts=['cat', 'dog'], values=[True, False])

// Search concept by name
client.searchInputs(SearchClause.matchConcept(Concept.forName("cat")))
    .getPage(1)
    .executeSync();

// Search concept by ID
client.searchInputs(SearchClause.matchConcept(Concept.forID("ai_mFqxrph2")))
    .getPage(1)
    .executeSync();

// Search multiple concepts
client.searchInputs(SearchClause.matchConcept(Concept.forID("cat")))
    .and(SearchClause.matchConcept(Concept.forID("cute")))
    .getPage(1)
    .executeSync();

// Search NOT by concept
client.searchInputs(SearchClause.matchConcept(Concept.forID("cat").withValue(false)))
    .getPage(1)
    .executeSync();

// First create a search term with a concept you want to search.
ClarifaiConcept *conceptFromGeneralModel = [[ClarifaiConcept alloc] initWithConceptName:@"fast"];
ClarifaiSearchTerm *searchTerm = [ClarifaiSearchTerm searchByPredictedConcept:conceptFromGeneralModel];

[app search:@[searchTerm] page:@1 perPage:@20 completion:^(NSArray<ClarifaiSearchResult *> *results, NSError *error) {
  // Print output of first search result.
  NSLog(@"inputID: %@", results[0].inputID);
  NSLog(@"URL: %@", results[0].mediaURL);
  NSLog(@"probability of input matching search query: %@", results[0].score);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "query": {
      "ands": [
        {
          "output": {
            "data": {
              "concepts": [
                {
                  "name":"dog"
                }
              ]
            }
          }
        }
      ]
    }
  }'\
  https://api.clarifai.com/v2/searches

By Custom Concepts

After you have added inputs with concepts, you can search by those concepts.


app.inputs.search([
  {
    concept: {
      type: 'input',
      name: 'cat'
    }
  },
  {
    concept: {
      type: 'input',
      name: 'dog'
    }
  }
]).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# search by single concept name
app.inputs.search_by_annotated_concepts(concept='cat')

# search by single concept id
app.inputs.search_by_annotated_concepts(concept_id='ai_mFqxrph2')

# search by multiple concepts with name
app.inputs.search_by_annotated_concepts(concepts=['cat', 'cute'])

# search by multiple concepts with ids
app.inputs.search_by_annotated_concepts(concept_ids=['ai_mFqxrph2', 'ai_4CRlSvbV'])

# search by multiple concepts with not logic
app.inputs.search_by_annotated_concepts(concepts=['cat', 'dog'], values=[True, False])

// Search concept by name
client.searchInputs(SearchClause.matchUserTaggedConcept(Concept.forName("cat")))
    .getPage(1)
    .executeSync();

// Search concept by ID
client.searchInputs(SearchClause.matchUserTaggedConcept(Concept.forID("ai_mFqxrph2")))
    .getPage(1)
    .executeSync();

// Search multiple concepts
client.searchInputs(SearchClause.matchUserTaggedConcept(Concept.forID("cat")))
    .and(SearchClause.matchUserTaggedConcept(Concept.forID("cute")))
    .getPage(1)
    .executeSync();

// Search NOT by concept
client.searchInputs(SearchClause.matchUserTaggedConcept(Concept.forID("cat").withValue(false)))
    .getPage(1)
    .executeSync();

// If you have previously added inputs tagged with "dog", you can search for them by the same tag.
ClarifaiConcept *concept = [[ClarifaiConcept alloc] initWithConceptName:@"dog"];
ClarifaiSearchTerm *term = [ClarifaiSearchTerm searchInputsByConcept:concept];

[app search:@[term] page:@1 perPage:@20 completion:^(NSArray<ClarifaiSearchResult *> *results, NSError *error) {
  // Print output of first search result.
  NSLog(@"inputID: %@", results[0].inputID);
  NSLog(@"URL: %@", results[0].mediaURL);
  NSLog(@"probability of input matching search query: %@", results[0].score);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "query": {
      "ands": [
        {
          "input": {
            "data": {
              "concepts": [
                {
                  "name":"dog"
                }
              ]
            }
          }
        }
      ]
    }
  }'\
  https://api.clarifai.com/v2/searches

By Image

You can use images to search through your collection. The API will return ranked results based on how similar the results are to the image you provided in your query.


app.inputs.search(
  {
    input: {
      url: 'https://samples.clarifai.com/puppy.jpeg'
    }
  }
).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# search by image url
app.inputs.search_by_image(url="https://samples.clarifai.com/metro-north.jpg")

# search by existing input id
input_id = "some_existing_input_id"
app.inputs.search_by_image(image_id=input_id)

# search by raw bytes
data = "image_raw_bytes"
app.inputs.search_by_image(imgbytes=data)

# search by base64 bytes
base64_data = "image_bytes_encoded_in_base64"
app.inputs.search_by_image(base64bytes=base64_data)

# search by local filename
filename="filename_on_local_disk.jpg"
app.inputs.search_by_image(filename=filename)

# search from fileio
fio = open("filename_on_local_disk.jpg", 'rb')
app.inputs.search_by_image(fileobj=fio)

// Search by image URL (String or java.net.URL)
client.searchInputs(SearchClause.matchImageVisually(ClarifaiImage.of("https://samples.clarifai.com/metro-north.jpg")))
    .getPage(1)
    .executeSync();

// Search by local image (java.io.File or byte[])
client.searchInputs(SearchClause.matchImageVisually(ClarifaiImage.of(new File("image.png"))))
    .getPage(1)
    .executeSync();

ClarifaiSearchTerm *searchTerm = [ClarifaiSearchTerm searchVisuallyWithImageURL:@"https://samples.clarifai.com/metro-north.jpg"];

[app search:@[searchTerm] page:@1 perPage:@20 completion:^(NSArray<ClarifaiSearchResult *> *results, NSError *error) {
  // Print output of first search result.
  NSLog(@"inputID: %@", results[0].inputID);
  NSLog(@"URL: %@", results[0].mediaURL);
  NSLog(@"probability of input matching search query: %@", results[0].score);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "query": {
      "ands": [
        {
          "output":{
            "input":{
              "data": {
                "image": {
                  "url": "https://samples.clarifai.com/metro-north.jpg"
                }
              }
            }
          }
        }
      ]
    }
  }'\
  https://api.clarifai.com/v2/searches

By Image With Crop

You can search using a crop of an image through your collection. The API will still return ranked results based upon on how similar the results are to the crop of the image you provide in your query.

app.inputs.search(
  {
    input: {
      url: 'https://samples.clarifai.com/puppy.jpeg',
      crop: [0.1, 0.1, 0.9, 0.9]
    }
  }
).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);
from clarifai.rest import ClarifaiApp

app = ClarifaiApp(api_key='YOUR_API_KEY')

search = app.inputs.search_by_image(url='https://samples.clarifai.com/puppy.jpeg', crop=[0.1, 0.1, 0.9, 0.9])

print search
client.searchInputs(SearchClause.matchImageVisually(ClarifaiImage.of("https://samples.clarifai.com/puppy.jpeg")
    .withCrop(Crop.create()
        .top(0.1F)
        .left(0.1F)
        .bottom(0.9F)
        .right(0.9F)
    )
))
.getPage(1)
.executeSync();
// Coming Soon
curl -X POST \
  -H 'authorization: Key YOUR_API_KEY' \
  -H 'content-type: application/json' \
  -d '{
  "query": {
    "ands": [
      {
        "output": {
          "input": {
              "data": {
                "image": {
                    "url": "https://samples.clarifai.com/puppy.jpeg",
                    "crop": [0.1,0.1,0.9,0.9]
                }
            }
          }
        }  
      }
    ]
  }
}'\
https://api.clarifai.com/v2/searches

By Public and Custom Concepts

You can combine a search to find inputs that have concepts you have supplied as well as predictions from your model.


app.inputs.search([
  // this is the public concept
  {
    concept: {
      name: 'cat'
    }
  },
  // this is the user-supplied concept
  {
    concept: {
      type: 'input',
      name: 'dog'
    }
  }
]).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

from clarifai.rest import InputSearchTerm, OutputSearchTerm, SearchQueryBuilder

term1 = InputSearchTerm(concept='cat')
term2 = OutputSearchTerm(concept='dog', value=False)
query = SearchQueryBuilder()
query.add_term(term1)
query.add_term(term2)

app.inputs.search(query)

client.searchInputs()
    // Matches images we tagged as "cat", and that the API tagged as not having "dog"
    .ands(
        SearchClause.matchUserTaggedConcept(Concept.forName("cat")),
        SearchClause.matchConcept(Concept.forName("dog").withValue(false))
    )
    .getPage(1)
    .executeSync();

ClarifaiConcept *conceptFromGeneralModel = [[ClarifaiConcept alloc] initWithConceptName:@"fast"];
ClarifaiConcept *conceptFromTrainedCustomModel = [[ClarifaiConcept alloc] initWithConceptName:@"dog"];

ClarifaiSearchTerm *term1 = [ClarifaiSearchTerm searchByPredictedConcept:conceptFromGeneralModel];
ClarifaiSearchTerm *term2 = [ClarifaiSearchTerm searchByPredictedConcept:conceptFromTrainedCustomModel];

[_app search:@[term1, term2] page:@1 perPage:@20 completion:^(NSArray<ClarifaiSearchResult *> *results, NSError *error) {
  // Print output of first search result.
  NSLog(@"inputID: %@", results[0].inputID);
  NSLog(@"URL: %@", results[0].mediaURL);
  NSLog(@"probability of input matching search query: %@", results[0].score);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
-d '
{
  "query": {
    "ands": [
      {
        "output": {
          "data": {
            "concepts": [
              {
                "name": "fast"
              }
            ]
          }
        }
      },
      {
        "input": {
          "data": {
            "concepts": [
              {
                "name": "ferrari23",
                "value": true
              }
            ]
          }
        }
      }
    ]
  }
}'\
https://api.clarifai.com/v2/searches

By Custom Metadata

After you have added inputs with custom metadata, you can search by that metadata.

Below is an example of searching over custom metadata. You can exact match any key: value pair no matter how nested it is. For example, if the metadata on an input is:

{
  "keyname": "value1",
  "somelist": [1,2,3],
  "somenesting": {
     "keyname2":"value2",
     "list2":[4,5]
   }
}

Then the following searches will find this:

{
  "keyname": "value1"
}
{
  "somelist": [1,2,3]
}
{
  "somelist": [1,2]
}
{
  "somenesting": {"keyname2":"value2"}
}
{
  "somenesting": {"list2":[5]}
}

How to perform searches:


// Search with only metadata
app.inputs.search({
  input: {
    metadata: {
      key: 'value'
    }
  }
}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

// Search with nested metadata
app.inputs.search({
  input: {
    metadata: {
      parent: {
        key: 'value'
      }
    }
  }
}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

// Search with metadata and concepts or input source
app.inputs.search([
  {
    input: { metadata: { key: 'value' } }
  },
  {
    concept: { name: 'cat' }
  },
  {
    concept: { type: 'output', name: 'group', value: false }
  }
]).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

# search with simple metadata only
app.inputs.search_by_metadata(metadata={'name':'bla'})

# search with nested metadata only
app.inputs.search_by_metadata(metadata={'my_class1': { 'name' : 'bla' }})

# search with metadata combined with others

from clarifai.rest import InputSearchTerm
from clarifai.rest import OutputSearchTerm
from clarifai.rest import SearchQueryBuilder

query = SearchQueryBuilder()
query.add_term(InputSearchTerm(concept='cat'))
query.add_term(InputSearchTerm(metadata={'name':'value'}))
query.add_term(OutputSearchTerm(concept='group', value=False))

app.inputs.search(query)

JsonObject metadata = new JsonObject();
metadata.addProperty("isPuppy", true);

List<SearchHit> hits = client
  .searchInputs(SearchClause.matchMetadata(metadata))
  .executeSync();

// Search by metadata only.
[_app searchByMetadata:@{@"my_key": @[@"my", @"values"]} page:@1 perPage:@20 completion:^(NSArray<ClarifaiSearchResult *> *results, NSError *error) {
  // Print output of first search result.
  NSLog(@"inputID: %@", results[0].inputID);
  NSLog(@"URL: %@", results[0].mediaURL);
  NSLog(@"probability of input matching search query: %@", results[0].score);
}];

// Search metadata in conjunction with other ClarifaiSearchTerms. For example, the
// following will search for inputs with predicted tag "fast" and matching metadata.
ClarifaiConcept *conceptFromGeneralModel = [[ClarifaiConcept alloc] initWithConceptName:@"fast"];
ClarifaiSearchTerm *searchTerm1 = [ClarifaiSearchTerm searchByPredictedConcept:conceptFromGeneralModel];

ClarifaiSearchTerm *searchTerm2 = [ClarifaiSearchTerm searchInputsWithMetadata:@{@"my_key": @[@"my", @"values"]}];

[app search:@[searchTerm1, searchTerm2] page:@1 perPage:@20 completion:^(NSArray<ClarifaiSearchResult *> *results, NSError *error) {
  // Print output of first search result.
  NSLog(@"inputID: %@", results[0].inputID);
  NSLog(@"URL: %@", results[0].mediaURL);
  NSLog(@"probability of input matching search query: %@", results[0].score);
}];

curl -X POST \
  -H "Authorization: Key {api-key}" \
  -H "Content-Type: application/json" \
  -d '
  {
    "query": {
      "ands": [
        {
          "input":{
            "data": {
              "metadata": {
                "key": "value"
              }
            }
          }
        }
      ]
    }
  }'\
  https://api.clarifai.com/v2/searches

By Url

You can also search for an input by URL.


app.inputs.search(
  {
    input: {
      type: 'input',
      url: 'https://samples.clarifai.com/puppy.jpeg'
    }
  }
).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

meta = {"url":"https://samples.clarifai.com/metro-north.jpg"}
app.inputs.search_by_metadata(meta)

// Lookup images with this URL
client.searchInputs(SearchClause.matchImageURL(ClarifaiImage.of("https://samples.clarifai.com/puppy.jpeg")))
    .getPage(1)
    .executeSync();


// Lookup images with this URL
ClarifaiSearchTerm *term = [ClarifaiSearchTerm searchInputsWithImageURL:@"https://samples.clarifai.com/metro-north.jpg"];

[app search:@[term] page:@1 perPage:@20 completion:^(NSArray<ClarifaiSearchResult *> *results, NSError *error) {
  // Print output of first search result.
  NSLog(@"inputID: %@", results[0].inputID);
  NSLog(@"URL: %@", results[0].mediaURL);
  NSLog(@"probability of input matching search query: %@", results[0].score);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "query": {
      "ands": [
        {
          "input":{
            "data": {
              "image": {
                "url": "https://samples.clarifai.com/metro-north.jpg"
              }
            }
          }
        }
      ]
    }
  }'\
  https://api.clarifai.com/v2/searches

By Geo Location

Search by geo location allows you to restrict your search results to a bounding box based on longitude and latitude points. There are two ways you can provide longitude/latitude points. You can provide one point and a radius or you can provide two points.

It is important to note that a search by geo location acts as a filter and returns results ranked by any other provided search criteria, whether that is a visual search, concept search or something else. If no other criteria is provided, results will return in the order the inputs were created, NOT by their distance to center of the search area.

If you are providing one point and a radius, the radius can be in "mile", "kilometer", "degree", or "radian", marked by keywords withinMiles, withinKilometers, withinDegrees, withinRadians.

If you are providing two points, a box will be drawn from the uppermost point to the lowermost point and the leftmost point to the rightmost point.

Before you perform a search by geo location, make sure you have added inputs with longitude and latitude points.

Add inputs with longitiude and latitude points

Provide a geo point to an input. The geo point is a JSON object consisting of a longitude and a latitude in GPS coordinate system (SRID 4326). There can be at most one single geo point associated with each input.

app.inputs.create({
  url: "https://samples.clarifai.com/puppy.jpeg",
  geo: { longitude: 116.2317, latitude: 39.5427},
}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);
from clarifai.rest import Geo, GeoPoint
from clarifai.rest import ClarifaiApp

geo_p1 = Geo(geo_point=GeoPoint(116.2317,39.5427))

app.inputs.create_image_from_url(url="https://samples.clarifai.com/puppy.jpeg", geo=geo_p1)
client.addInputs().plus(ClarifaiInput.forImage("https://samples.clarifai.com/puppy.jpeg")
    .withGeo(PointF.at(116.2317F, 39.5427F))).executeSync();
ClarifaiImage *image = [[ClarifaiImage alloc] initWithURL:@"https://samples.clarifai.com/metro-north.jpg"];
image.location = [[ClarifaiLocation alloc] initWithLatitude:116.2317 longitude:39.5427];

[_app addInputs:@[image] completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
  NSLog(@"%@",inputs);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "https://samples.clarifai.com/dog.tiff",
            "allow_duplicate_url": true
          },
          "geo": {
            "geo_point": {
              "longitude": -30,
              "latitude": 40
            }
          }
        }
      }
    ]
  }'\
  https://api.clarifai.com/v2/inputs

Perform a search with one geo point and radius in kilometers

app.inputs.search({
  input: {
    geo: {
      longitude: 116.2317,
      latitude: 39.5427,
      type: 'withinKilometers',
      value: 1
    }
  }
}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);
from clarifai.rest import GeoPoint, GeoLimit

geo_p = GeoPoint(116.2317, 39.5427)
geo_l = GeoLimit(limit_type='kilometer', limit_range=1)

imgs = app.inputs.search_by_geo(geo_point=geo_p, geo_limit=geo_l)
client.searchInputs(SearchClause.matchGeo(PointF.at(59F, 29.75F), Radius.of(500, Radius.Unit.KILOMETER)))
            .getPage(1)
            .executeSync();

ClarifaiLocation *loc = [[ClarifaiLocation alloc] initWithLatitude:116.2317 longitude:39.5427];
ClarifaiGeo *geoFilterKilos = [[ClarifaiGeo alloc] initWithLocation:loc radius:50.0 andRadiusUnit:ClarifaiRadiusUnitKilometers];
ClarifaiSearchTerm *term = [ClarifaiSearchTerm searchInputsWithGeoFilter:geoFilterKilos];

[_app search:@[term] page:@1 perPage:@20 completion:^(NSArray<ClarifaiSearchResult *> *results, NSError *error) {
  NSLog(@"inputID: %@", results[0].inputID);
  NSLog(@"URL: %@", results[0].mediaURL);
  NSLog(@"probability of predicted concept: %@", results[0].score);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "query": {
      "ands": [
        {
          "input": {
            "data": {
              "image":{
                "url":"https://samples.clarifai.com/metro-north.jpeg"
              },
              "geo": {
                "geo_point": {
                  "longitude": -1,
                  "latitude": 1.5
                },
                "geo_limit": {
                  "type": "withinKilometers",
                  "value": 1
                }
              }
            }
          }
        }
      ]
    }
  }'\
  https://api.clarifai.com/v2/searches

Perform a search with two geo points

app.inputs.search({
  input: {
    geo: [{
      latitude: 116.2316,
      longitude: 39.5426
    }, {
      latitude: 116.2318,
      longitude: 39.5428
    }]
  }
}).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);
from clarifai.rest import GeoBox, GeoPoint, GeoLimit

p1 = GeoPoint(116.2316, 39.5426)
p2 = GeoPoint(116.2318, 39.5428)
box1 = GeoBox(point1=p1, point2=p2)

imgs = app.inputs.search_by_geo(geo_box=box1)
client.searchInputs(SearchClause.matchGeo(PointF.at(3F, 0F), PointF.at(70, 30F)))
            .getPage(1)
            .executeSync()

ClarifaiLocation *startLoc = [[ClarifaiLocation alloc] initWithLatitude:50 longitude:58];
ClarifaiLocation *endLoc = [[ClarifaiLocation alloc] initWithLatitude:32 longitude:-30];
ClarifaiGeo *geoBox = [[ClarifaiGeo alloc] initWithGeoBoxFromStartLocation:startLoc toEndLocation:endLoc];

[_app search:@[term] page:@1 perPage:@20 completion:^(NSArray<ClarifaiSearchResult *> *results, NSError *error) {
  NSLog(@"inputID: %@", results[0].inputID);
  NSLog(@"URL: %@", results[0].mediaURL);
  NSLog(@"probability of predicted concept: %@", results[0].score);
}];

curl -X POST \
  -H "Authorization: Key YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '
  {
    "query": {
      "ands": [
        {
          "input": {
            "data": {
              "geo": {
                "geo_box": [
                  {  
                    "geo_point": {
                      "latitude": 35,
                      "longitude": -30
                    }
                  },
                  {
                    "geo_point": {
                      "latitude": 50,
                      "longitude": -35
                    }
                  }
                ]
              }
            }
          }
        }
      ]
    }
  }'\
  https://api.clarifai.com/v2/searches

Search ANDing

You can also combine searches using AND.


app.inputs.search([
  { input: { url: 'https://samples.clarifai.com/puppy.jpeg' } },
  { concept: { name: 'cat', type: 'input' } },
  { concept: { name: 'dog' } }
]).then(
  function(response) {
    // do something with response
  },
  function(err) {
    // there was an error
  }
);

from clarifai.rest import InputSearchTerm, OutputSearchTerm, SearchQueryBuilder

term1 = InputSearchTerm(concept='cat')
term2 = OutputSearchTerm(concept='dog', value=False)
term3 = OutputSearchTerm(url="https://samples.clarifai.com/metro-north.jpg")

query = SearchQueryBuilder()
query.add_term(term1)
query.add_term(term2)
query.add_term(term3)

app.inputs.search(query)

client.searchInputs()
    .ands(
        SearchClause.matchUserTaggedConcept(Concept.forName("cat")),
        SearchClause.matchConcept(Concept.forName("dog").withValue(false)),
        SearchClause.matchImageVisually(ClarifaiImage.of("https://samples.clarifai.com/metro-north.jpg"))
    )
    .getPage(1)
    .executeSync();

//Search for inputs that are predicted as "fast" and visually similar to the given image.
ClarifaiConcept *conceptFromGeneralModel = [[ClarifaiConcept alloc] initWithConceptName:@"fast"];
ClarifaiSearchTerm *term1 = [ClarifaiSearchTerm searchByPredictedConcept:conceptFromGeneralModel];

ClarifaiSearchTerm *term2 = [ClarifaiSearchTerm searchVisuallyWithImageURL:@"https://samples.clarifai.com/metro-north.jpg"];

[_app search:@[term1, term2] page:@1 perPage:@20 completion:^(NSArray<ClarifaiSearchResult *> *results, NSError *error) {
  // Print output of first search result.
  NSLog(@"inputID: %@", results[0].inputID);
  NSLog(@"URL: %@", results[0].mediaURL);
  NSLog(@"probability of input matching search query: %@", results[0].score);
}];

curl -X POST \
  -H "Authorization: Key {api-key}" \
  -H "Content-Type: application/json" \
-d '
{
    "query": {
        "ands": [
            {
                "output": {
                    "input":{
                        "data": {
                            "image": {
                                "url": "http://i.imgur.com/HEoT5xR.png"
                            }
                        }
                    }
                }
            },
            {
                "output": {
                    "data": {
                        "concepts": [
                            {"name":"fast", "value":true}
                        ]
                    }
                }
            }
        ]
    }
}'\
https://api.clarifai.com/v2/searches

Pagination

Many API calls are paginated. You can provide page and per_page params to the API. In the example below we are getting all inputs and specifying to start at page 2 and get back 20 results per page.


app.inputs.list({page: 2, perPage: 20});

app.inputs.get_by_page(page=2, per_page=20)

client.getInputs()
    .perPage(20) // OPTIONAL, to specify how many results should be on one page
    .getPage(2)
    .executeSync();

[app getInputsOnPage:2 pageSize:20 completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
    NSLog(@"inputs: %@", inputs);
}];

curl -X GET \
  -H "Authorization: Key YOUR_API_KEY" \
  https://api.clarifai.com/v2/inputs?page=2&per_page=20

Patching

We designed PATCH to work over multiple resources at the same time (bulk) and be flexible enough for all your needs to minimize round trips to the server. Therefore it might seem a little different to any PATCH you've seen before, but it's not complicated. All three actions that are supported do overwrite by default, but have special behaviour for lists of objects (for example lists of concepts).

Merge

merge action will overwrite a key:value with key:new_value or append to an existing list of values, merging dictionaries that match by a corresponding id field.

In the following examples A is being patched into B to create the Result:


*Merges different key:values*
A = `{"a":[1,2,3]}`
B = `{"blah":true}`
Result = `{"blah":true, "a":[1,2,3]}`

*For id lists, merge will append*
A = `{"a":[{"id": 1}]}`
B = `{"a":[{"id": 2}]}`
Result = `{"a":[{"id": 2}, {"id":1}]}`

*Simple merge of key:values and within a list*
A = `{"a":[{"id": "1", "other":true}], "blah":1}`
B = `{"a":[{"id": "2"},{"id":"1", "other":false}]}`
Result = `{"a":[{"id": "2"},{"id": "1"}], "blah":1}`

*Different types should overwrite fine*
A = `{"a":[{"id": "1"}], "blah":1}`
B = `{"a":[{"id": "2"}], "blah":"string"}`
Result = `{"a":[{"id": "2"},{"id": "1"}], "blah":1}`

*Deep merge, notice the "id":"1" matches, so those dicts are merged in the list*
A = `{"a":[{"id": "1","hey":true}], "blah":1}`
B = `{"a":[{"id": "1","foo":"bar","hey":false},{"id":"2"}], "blah":"string"}`
Result = `{"a":[{"hey":true,"id": "1","foo":"bar"},{"id":"2"}], "blah":1}`

*For non-id lists, merge will append*
A = `{"a":[{"blah": "1"}], "blah":1}`
B = `{"a":[{"blah": "2"}], "blah":"string"}`
Result = `{"a":[{"blah": "2"}, {"blah":"1"}], "blah":1}`

*For non-id lists, merge will append*
A = `{"a":[{"blah": "1"}], "blah":1, "dict":{"a":1,"b":2}}`
B = `{"a":[{"blah": "2"}], "blah":"string"}`
Result = `{"a":[{"blah": "2"}, {"blah":"1"}], "blah":1, "dict":{"a":1,"b":2}}`

*Simple overwrite root element*
A = `{"key1":true}`
B = `{"key1":{"key2":"value2", "key3":"value3"}}`
Result = `{"key1":true}`

*Overwrite a sub element*
A = `{"key1":{"key2":true}}`
B = `{"key1":{"key2":"value2", "key3":"value3"}}`
Result = `{"key1":{"key2":true, "key3":"value3"}}`

*Merge a sub element*
A = `{"key1":{"key2":{"key4":"value4"}}}`
B = `{"key1":{"key2":"value2", "key3":"value3"}}`
Result = `{"key1":{"key2":{"key4":"value4"}, "key3":"value3"}}`

*Merge multiple trees*
A = `{"key1":{"key2":{"key9":"value9"}, "key3":{"key4":"value4", "key10":[1,2,3]}}, "key6":{"key11":"value11"}}`
B = `{"key1":{"key2":"value2", "key3":{"key4":{"key5":"value5"}}}, "key6":{"key7":{"key8":"value8"}}}`
Result = `{"key1":{"key2":{"key9":"value9"}, "key3":{"key4":"value4", "key10":[1,2,3]}}, "key6":{"key7":{"key8":"value8"}, "key11":"value11"}}`

*Merge {} element will replace*
A = `{"key1":{"key2":{}}}`
B = `{"key1":{"key2":"value2", "key3":"value3"}}`
Result = `{"key1":{"key2":{}, "key3":"value3"}}`

*Merge a null element does nothing*
A = `{"key1":{"key2":null}}`
B = `{"key1":{"key2":"value2", "key3":"value3"}}`
Result = `{"key1":{"key2":"value2", "key3":"value3"}}`

*Merge a blank list [] will replace root element*
A = `{"key1":[]}`
B = `{"key1":{"key2":"value2", "key3":"value3"}}`
Result = `{"key1":[]}`

*Merge a blank list [] will replace single element*
A = `{"key1":{"key2":[]}}`
B = `{"key1":{"key2":"value2", "key3":"value3"}}`
Result = `{"key1":{"key2":[], "key3":"value3"}}`

*Merge a blank list [] will remove nested objects*
A = `{"key1":{"key2":[{"key3":"value3"}]}}`
B = `{"key1":{"key2":{"key3":"value3"}}}`
Result = `{"key1":{"key2":[{"key3":"value3"}]}}`

*Merge an existing list with some other struct*
A = `{"key1":{"key2":{"key3":[{"key4":"value4"}]}}}`
B = `{"key1":{"key2":[]}}`
Result = `{"key1":{"key2":{"key3":[{"key4":"value4"}]}}}`

Remove

remove action will overwrite a key:value with key:new_value or delete anything in a list that matches the provided values' ids.

In the following examples A is being patched into B to create the Result:

*Remove from list*
A = `{"a":[{"id": "1"}], "blah":1}`
B = `{"a":[{"id": "2"},{"id": "3"}, {"id":"1"}], "blah":"string"}`
Result = `{"a":[{"id": "2"},{"id":"3"}], "blah":1}`

*For non-id lists, remove will append*
A = `{"a":[{"blah": "1"}], "blah":1}`
B = `{"a":[{"blah": "2"}], "blah":"string"}`
Result = `{"a":[{"blah": "2"}, {"blah":"1"}], "blah":1}`

*Empty out a nested dictionary*
A = `{"key1":{"key2":true}}`
B = `{"key1":{"key2":"value2"}}`
Result = `{"key1":{}}`

*Remove the root element, should be empty*
A = `{"key1":true}`
B = `{"key1":{"key2":"value2", "key3":"value3"}}`
Result = `{}`

*Remove a sub element*
A = `{"key1":{"key2":true}}`
B = `{"key1":{"key2":"value2", "key3":"value3"}}`
Result = `{"key1":{"key3":"value3"}}`

*Remove a multiple sub elements*
A = `{"key1":{"key2":{"key3":true}, "key4":true}}`
B = `{"key1":{"key2":{"key3":{"key5":"value5"}}, "key4":{"key6":{"key7":"value7"}}}}`
Result = `{"key1":{"key2":{}}}`

*Remove one of the root elements if there are more than one*
A = `{"key1":true}`
B = `{"key1":{"key2":"value2", "key3":"value3"}, "key4":["a", "b", "c"]}`
Result = `{"key4":["a", "b", "c"]}`

*Remove with false should over write*
A = `{"key1":{"key2":false, "key3":true}, "key4":false}`
B = `{"key1":{"key2":"value2", "key3":"value3"}, "key4":[{"key5":"value5", "key6":"value6"}, {"key7": "value7"}]}`
Result = `{"key1":{"key2":false}, "key4":false}`

*Only objects with id's can be put into lists*
A = `{"key1":[{"key2":true}]}`
B = `{"key1":[{"key2":"value2"}, {"key3":"value3"}]}`
Result = `{}`

*Elements with {} should do nothing*
A = `{"key1":{}}`
B = `{"key1":{"key2":"value2", "key3":"value3"}}`
Result = `{"key1":{"key2":"value2", "key3":"value3"}}`

*Elements with nil should do nothing*
A = `{"key1":{"key2":null}}`
B = `{"key1":{"key2":"value2", "key3":"value3"}}`
Result = `{"key1":{"key2":"value2", "key3":"value3"}}`

Overwrite

overwrite action will overwrite a key:value with key:new_value or overwrite a list of values with the new list of values. In most cases this is similar to merge action.

In the following examples A is being patched into B to create the Result:

*Overwrite whole list*
A = `{"a":[{"id": "1"}], "blah":1}`
B = `{"a":[{"id": "2"}], "blah":"string"}`
Result = `{"a":[{"id": "1"}], "blah":1}`

*For non-id lists, overwrite will overwrite whole list*
A = `{"a":[{"blah": "1"}], "blah":1}`
B = `{"a":[{"blah": "2"}], "blah":"string"}`
Result = `{"a":[{"blah": "1"}], "blah":1}`

Supported Types

The API supports the following image formats:

  • JPEG
  • PNG
  • TIFF
  • BMP

The API supports the following video formats:

  • AVI
  • MP4
  • WMV
  • MOV
  • GIF
  • 3GPP
>