Clarifai API Conventions

The goal for this post in our series on designing the Clarifai API is to cover some of the conventions that we instituted when designing v2 of our API. Sharing these conventions should not only provide a better understanding of how to easily use a copy and paste guide to your first api project with clarifai but can also help in designing your own set of APIs.

Our API is designed to operate as both HTTP and gRPC as mentioned in Part 2 of this series and with that in mind we had to think about conventions that apply to both regimes. We have developed a straightforward REST architecture for our API surface that hides the underlying complexity of our machine learning algorithms so that you can integrate AI functionality within your application as easily as any other kind of API.

Endpoint Conventions

Clarifai resources can be manipulated using standard HTTP methods. With the HTTP protocol, the resource names map to URLs, and methods naturally map to HTTP methods POST, GET, PATCH, and DELETE. 

POST

Used when creating a new object or conducting some complex operation that needs an extensive request body. For example, you will use the POST method when adding inputs to your application or having a model make predictions on your data.

POSTalways creates a new resource, so the returned object should have an ID. The ID of an object should just be called "id" at the top-level of that object.

POST  should include the object you are posting in a named field. For example POST /models should be {“model”:{MODEL_OBJECT}}. 

POST/ PATCH should always be a plural request/response as what we learned by rewriting our API in most cases allows for bulk processing whenever possible.

When needing to conduct a complex operation like making a prediction we also use POST so that a complete body of the request can be filled in. Additionally, since the model prediction request returns outputs, the URL endpoint ends with outputs like:

POST https://api.clarifai.com/api-doc/#/V2/V2_PostModelOutputs4

GET

Use the GET method to return an object at a given endpoint. For example, GET /inputs allows you to paginate through all the inputs in your app. As you’ll see below, GET for a specific ID will return just that one object. 

GET returns a dictionary where the type of the get is the key inside the dictionary and the object is present. For example GET /concepts/{concept_id} returns {"concept":{CONCEPT OBJECT}}.

GET will give you a list of endpoints, on specific IDs, it will return a single item. Putting this into practice, we might use the GET method to return an object at a specific endpoint. For example, to get a specific concept within an app we would use

GET  https://api.clarifai.com/v2/concepts/{concept_id}

Get multiple objects should support pagination and it should return an empty list if there is nothing. For example, GET /concepts returns {“concepts”:[{CONCEPT OBJECT 1},{CONCEPT OBJECT 2}]}.In the case where there are no concepts, GET /concepts returns an empty list {“concepts”:[]}.

PATCH

Use PATCH when you need to change something that you have already created. For example, PATCH /models/{model_id} would be used to modify that given model specific by model_id. You can find additional information about patching in our API documentation here.

PATCH should always be a plural request/response as that is needed for patching many concepts or multiple concepts to a model for example. Therefore something like PATCH /concepts has a request {“concepts”:[{CONCEPT OBJECT 1},{CONCEPT OBJECT 2}]}.

PATCH request should have a string field called ”action”. There are three types of action supported now. merge, overwrite, remove.

Patching has been designed to effectively handle multiple objects in one request (this becomes useful when you have a large number of concepts for example).Patching relies on each object having that ”id” field to know which part of your request body should be matched (recursively) and then modified according to the ”action”.Please see additional documentation here:

https://docs.clarifai.com/api-guide/data-management/patching

DELETE

Delete simply removes that resource from your app. Some endpoints support deleting an individual object by id, deleting a batch of objects listed in the ”ids” field of the request or iterate over and delete all data in an endpoint with a plural name when ”delete_all” is true. As an example, DELETE /models/{model_id} would delete that model. 

Example Endpoints:

Since we support both grpc and HTTP we define endpoints as RPC names with the following convention:

{HTTPMethod}{Resource}

For example, GetConcept is the GET /concepts/{concept_id} endpoint and ListConcepts is the GET /concepts rpcs which we define in a protobuf file as:

The last name in an endpoint is the type of object that you should expect from that endpoint.

The path of an endpoint should always be:

/v2/users/{user_app_id.user_id}/apps/{user_app_id.app_id}/* or /v2/users/{user_app_id.user_id}/*

/v2/* should be only used in additional bindings.

You’ll see soon we will switch only using the fully qualified URL with the user_id and app_id in them so we highly recommend you use those starting now. You must not use hyphens in the path. Below you will see how the Request and Response objects you see above are defined.

Apps 

Apps are the fundamental top-level resource of much of what you will do in our API. Apps contain important objects such as models, concepts and input data. Assuming your user id is user_id, then the path to your Apps and their functionality are simply:

And the path to resources held under an app (specified by app_id) will be:

https://api.clarifai.com/api-doc/#/V2/V2_ListConcepts

https://api.clarifai.com/api-doc/#/V2/V2_ListModels

https://api.clarifai.com/api-doc/#/V2/V2_ListInputs

For concepts, models, and inputs respectively as a few examples.

Side note: the URLs shown above are the fully qualified URLs that you should use if using personal access tokens to access other user’s apps. If you’re using an app-specific key we know which app that key is tied to and so you can actually skip providing both the user_id and the app_id in the URLs if you want such as:

However, since it doesn’t hurt to include both the user_id and app_id in the URLs with app-specific keys, and we are introducing more and more collaboration features, you should get comfortable with always providing that information. To keep the URLs short, for the remainder of this post we will not show the users or apps portion for simplicity.

You may notice another naming convention at this point - that these resources are all plural. Each object defined by the resources would return a list of items. In common RESTful style, if you would like to navigate to an individual resource you would simply add the id of the individual resource:

https://api.clarifai.com/api-doc/#/V2/V2_GetConcept2

https://api.clarifai.com/v2/models/{model_id}

https://api.clarifai.com/api-doc/#/V2/V2_GetInput2 

Responses with *_id are used to denote a reference to that object type (without filling in the heavyweight object in that response). For example:

app_id which is present in many of the request/response objects can later be used to get that app object with GET /apps/{app_id}.

Object Conventions

One of the benefits of building the underlying structure of our API in gRPC protobuffers, is that objects that you can GET out of the API are also valid requests. Protobuffers are object-oriented when compiled to your given API client language, giving you an object in your language that you can interact with as a response or a request. While it’s possible to do this in RESTful HTTP API as well, the benefits of having the object-oriented proto message support of gRPC makes this really seamless to implement and even more beneficial to use as an API you can essentially read/write from.

All fields in the JSON requests and responses should be snake_case.

The API follows “object_name”:{OBJECT} layout in the JSON where “object_name” denotes that the OBJECT for that key is of type “object_name”. For example: 

{“concept”:{CONCEPT OBJECT}}

Therefore whenever you see the same object_name throughout our API documentation you should see the same object (not all fields may be present but the same object). This gives you some information about the types to use in JSON format if you’re using HTTP request (which isn’t a problem using the gRPC interface which is strongly typed).

Here is an example of how the underlying object might be defined:

How we define requests and responses

In general requests and responses of the same object type should be interchangeable.

Request name should always be [Get|Post|Patch|Delete|List]*Request

Response name should always be [Single|Multiple]*Response

Plural keys names in requests/responses should have a list as the value type where each entry of the list is the object of that type. For example "concepts" is a list of CONCEPT objects.

Here is how we might define a requests and responses in Go:

And to *Response objects where we typically have Single* and Multi* variants:

Conclusion

We spent years developing these conventions, learning a lot from our v1 APIs so that our v2 APIs could have a much more extensible and consistent experience for our users. We hope you can take some learnings from this to build better APIs on your own and benefit from a better understanding of the Clarifai API which you can find out more about at docs.clarifai.com or sign up at portal.clarifai.com.