Last week, we were excited to announce the launch our new unified user interface, Clarifai Portal and Search demos. Now, we talk to Amy Kim, the trailblazing product manager who, alongside our team of engineers, lead the development of these amazing new features. Check out our interview below to find out more about what makes Clarifai Search stand out and how you can sign up for it!
Obviously, we just launched a new demo on the site showcasing Clarifai’s search capabilities, but first, can you explain what the components of computer vision search are?
Sure! To begin, Clarifai Search has 2 main states:
1) index preparation
2) search query
Once the index is prepared, you can input a query to our Search service, which will use the Clarifai’s search algorithm to rank and retrieve the search results from the index for that specific query.
To define the bolded keywords above:
Index: Index is the data source for all searches; at Clarifai, the index is your Clarifai App. All of the inputs in your app are prepared for future search queries. The indexing happens when you add inputs to your app.
Query: A Clarifai search query is a set of images, concepts, and/or other metadata that is used to search in the index. The query designates the ranking algorithm and/or filtering that will be used to retrieve the search results. An example query could look like:
AND my-vacation-home (custom concept)
The results will be images that are visually similar to 1) the query image of the dog and 2) the visual representation of the concept 'my-vacation-home.'
Ranking Algorithm: The Clarifai search ranking algorithm is based on visual relevance. For example, a custom concept search or image search would rank the index of images in the order of which is most visually similar to the visual representation of the query concept or image. Based on the example above, the search engine will rank the results based on which looks most like both the image of the dog and the custom concept (my vacation home). When the query designates a non-ranking filtering search, the retrieved results are sorted in the order of the created_at date of the input. For example, if you’re only looking to filter the index by certain metadata (e.g., “user-id”: “1234”), your top search results will be the most recently added inputs that had the json metadata “user-id”: “1234” associated with them.
Search Results: The search results refer to the ranked list of inputs that the search engine has found based on the query. If you designated a filter (e.g., “image-source”: “my-iphone”), your search results would only contain the filtered data (e.g., images from my index that are similar to the image of the dog and my vacation home that are only from my phone).
(Need help from an expert? Talk to us!)
Okay, so what makes Clarifai Search unique?
What makes Clarifai Search unique is that we are leveraging computer vision AI at the core to power our search engine. This allows our customers to use visual data in both the query and the index and automatically surface the most visually relevant hits in the search results, even when the index isn’t thoroughly labeled by humans. The end result is a sophisticated search product with which you can explore your image dataset, opening up what was traditionally unstructured — and therefore unsearchable — data (i.e., image pixels) into automatically searchable dataset without having to manually tag or store all these tags in your database. For example, by using Clarifai Search, you can automatically search your product catalog images by using various keyword concepts that belong to our pre-built models (e.g., “pants”), even if your catalog data doesn’t contain any of these fields.
What really sets Clarifai Search apart, though, is the fact that you can custom define the keywords yourself (e.g., “Dodo’s favorite pants”) very easily by using our Custom Concept Search functionality. To do so, you need to give only a few visual examples of the custom keyword to teach the model what the custom keyword looks like, and you’ll be able to immediately search using the custom keyword without having to re-index your dataset. This is a very powerful feature that allows customers to further customize their search needs for their visual dataset on the fly.
In addition to the visual attribute-based searches, you can also take advantage of the custom metadata filtering feature, with which you can further customize your search results using non-visual attributes (e.g., “best-seller”: “yes”).
What are some of the current major use cases for visual search?
The most prominent use cases we’ve seen are:
- General image search: The customer has unlabeled images and uses Clarifai to retrieve similar images from their album.
- Retail catalog search: The customer has catalog images of various retail products and uses Clarifai Search to look for visually similar products/product images. "Learn more about how computer vision is impacting retail here!
How do you see visual search being used in the future?
In the future, I see a lot of potential in applying Visual Search in the security domain — there is a ton of visual data from streaming videos that we could apply Visual Search to, facilitating custom object and/or frame retrievals. This is an area that we are actively looking into.
Does Clarifai Search have any limitations?
Of course, Clarifai Search is not perfect and has many limitations, most of which apply to our models. Some examples of the limitations include:
- Ability to index crops of images
- Ability to index videos
- Ability to customize different visual attributes for the ranking algorithm
- Ability to retrieve images from a domain that is different from the query’s domain, e.g., trying to map the same objects in different visual backgrounds
That said, we are actively working on the limitations above to better serve our customers’ needs. If you have uncovered a bug or a deficiency that you’d like to see in our Search product, please let us know!
If a business wanted to implement Clarifai Search, what would be their first steps?
First, they should check out our Search Demo to see the different functionalities in action. If they’re interested in moving forward, they can either sign up on our Portal to add their inputs to a new app to see Clarifai Search in action, or talk to our Sales team to get a more catered approach to their needs.
Either way, we’d love to hear from you on how you’re using Clarifai Search! Please feel free to send us your product feedback to feedback@clarifai.com.
