DOCUMENTATION

ImageChat

ImageChat Overview

ImageChat is Chooch’s latest cutting-edge model that provides more detailed insights into visual images with staggering accuracy. Our new ensemble model combines computer vision and large language models to provide you with the best possible results. ImageChat-2 has over 11 billion parameters and was trained on 400 million images. For more information on the release and an example use case, read our blog.

ImageChat includes functionality such as:

Easy to use image analysis and descriptive chat
Customized model creation and prompting
Model integration with ReadyNow models for additional accuracy and localization
Model integration with custom models
Hosted API usage and automated image analysis

ImageChat can be tested and demonstrated via Chooch’s iOS and Android apps or online.

ImageChat Examples

With ImageChat you can upload an image and have a continuous discussion, allowing you to provide multiple prompts for further analysis. The example conversations below showcase expected usage of ImageChat.

The Web interface is easy to use and fun to chat with. In addition, the API interface allows you to query an image and automate a prompt. ImageChat then returns the analyzed result in less than 0.2 seconds.

Please note you will need a free account on the Chooch AI Vision Studio. If you don’t yet have one, please sign up here. In order to use the ImageChat API and ReadyNow/Customized model integration, you will also need an Enterprise account. Please request one here.

Using ImageChat

Web Interface
- ImageChat in the Web Interface allows for one line prompts to be provided to ImageChat at a time. ImageChat will respond rapidly in a familiar-style chat box.

Custom Model interface in the Chooch AI Vision Studio
- Customers can integrate an Object Detector with ImageChat for better analysis on custom objects and higher accuracy with localization in the form of bounding boxes. Create integrated models in the Chooch AI Vision Studio as described in the section below.

Hosted API Interface
- Customers can use the ImageChat API to access ImageChat or an integrated ImageChat/Object Detector model automatically from their systems. This feature is great for continuously analyzing camera feeds, iterating through saved images or videos, or integrating into an existing system. See usage of the ImageChat API.

iOS and Android Applications
- Chooch’s iOS and Android applications are perfect for testing functionality with your phone’s camera or any saved photos. In addition, these apps are great for showcasing ImageChat functionality to your friend, colleague, or pet.

Creating Custom Models with ImageChat

You can add an object detector to ImageChat for additional accuracy and localization within a frame. ImageChat combined with an object detector allows for unparalleled accuracy in detections and analysis.

In order to create your custom ImageChat model, navigate to the Chooch AI Vision Studio and login.

Select Custom Models from the left side navigation bar.
Make sure you are in the ImageChat tab and select Add ImageChat Model.
Provide a name for your new model.
(Optional) Select an Object Detector from the Chooch ReadyNow catalog or from your custom models.
(Optional) Customize additional parameters of your object detection model such as padding, confidence, and non-max suppression. These are useful for additional fine tuning during model testing. If you did not select an object detector, these parameter options will not appear.
Add a prompt (or multiple) to automate detection with your new ImageChat model. Make sure to follow the best practices described in Prompting ImageChat to ensure the highest accuracy analysis.
Once you have created your custom ImageChat model, you can select your new model and begin testing. Select Test Model to upload one or more images for analysis.
View your results and fine-tune as needed by selecting Edit Model to adjust any additional parameters.

Prompting ImageChat and Best Practices

Creating high-quality prompts for ImageChat is crucial to generate accurate results. Whether you are using the web interface or creating your own model integrated with an Object Detector, ensure your prompts are free of typos, grammar errors, and straight-forward to understand.

For the best results we recommend using prompts that are full sentences and end in a question mark.

Instead of:

is there fire

Try:

Is there fire or smoke in this image?

Instead of:

do you see corrosion

Try:

Is there corrosion in this image?

In addition, refrain from making run-on prompts that can be confusing to ImageChat, for instance:

Is there fire or smoke or anything that might be dangerous in this image?

Instead, try multiple specific prompts such as:

Is there fire or smoke in this image?

Is there dry brush in this image?

As you test and prompt ImageChat you will find that high-quality prompts can positively impact the accuracy of the results. If you are not seeing the results you are expecting, try adjusting your prompts slightly and test again.

For instance, you can add the keyword “possible” to your prompt to allow a more sensitive detection from ImageChat:

Is there possible rust in this image?

In addition, including the phrase in this image? can help ImageChat understand your prompt better.
Instead of:

Is there fire?

Try:

Is there fire in this image?

If you have any questions on prompting or would like additional help from the Chooch Team, don’t hesitate to reach out at [email protected]

ImageChat API

Our cutting-edge ensemble model combines computer vision and large language models to provide you with the best possible results. Both ImageChat and Custom ImageChat models are available via API access for automating prompting and analysis in your pipeline.

ImageChat API

Below are code examples of using the ImageChat API. The API includes additional parameters to fine-tune your API call and ensure the best analysis.

ImageChat Parameters:

model_id: The ID of the ImageChat-2 model used to generate the predictions.

model_title: The title of the ImageChat-2 model used to generate the predictions.

predictions: The predictions made on the input image, provided as a list. Each prediction includes a class title, score, prediction keywords, and coordinates.

prompt: The prompt given to the model to generate the predictions.

image_width: The width of the input image in pixels.

image_height: The height of the input image in pixels.

image_id: The ID of the input image.

workflow_predict_objects_only: (Optional) When set to True, ImageChat will only look at the objects detected rather than the whole image.

workflow_class_filter: (Optional) This will only include the classes you explicitly state in the results.

workflow_padding: (Optional) Padding percentage will add extra space within the bounding box detections.

workflow_conf_thresh: (Optional) This will adjust the confidence threshold of the object detector and only show objects above the set threshold. Accepts values between 0.01 – 1.0.

workflow_nms_thresh: (Optional) This will adjust the non-max suppression threshold of the object detector. Non-max suppression removes possible duplicate bounding boxes if they overlap with other detections Accepts values between 0.01 – 1.0.

URL Post

Python
Javascript (REST)
cURL


														
														# Inference an Image on ImageChat-2, Image File Url
import requests
import json
from io import BytesIO

def predict_api(model_id, image_file_url, api_key=None, parameters=None):

    payload = {
        "image_file_url": image_file_url,
        "model_id": model_id,
        "parameters": parameters,
    }

    url = "https://apiv2.chooch.ai/predict?api_key={}".format(api_key)
    response = requests.put(url, data=json.dumps(payload))
    json_data = json.loads(response.content)
    return json_data


# Chooch ImageChat-2 model_id
model_id_image_chat_pt = "ad420c2a-d565-48eb-b963-a8297a0e4000"
image_file_url = ""

parameters = {}

# Default is True. If a prompt is given only 1 class will be returned and deep_detection will be turned off
parameters["deep_inference"] = True

parameters["prompt"] = "is there smoke in this image ?"

# Adding workflow, this is optional.  It will apply the prompt to given objected model predictions as well
smoke_fire_object_detection_model_id = "1b626ec2-69e5-44b8-9cec-587c87fa8b0f"

parameters["workflow_model_id"] = smoke_fire_object_detection_model_id

parameters["workflow_predict_objects_only"] = True

parameters["workflow_padding"] = 0.6

parameters["workflow_conf_thresh"]  = 0.3
parameters["workflow_nms_thresh"]  = 0.45


# Replace with your own api key
api_key = "YOUR_API_KEY"

return_val = predict_api(
    model_id_image_chat_pt, image_file_url, api_key=api_key, parameters=parameters
)
print(return_val)


														
														const axios = require('axios');

async function predictApi(modelId, imageFileUrl, apiKey=null, parameters=null) {
  const url = "https://apiv2.chooch.ai/predict?api_key=${apiKey}";
  const payload = {
    image_file_url: imageFileUrl,
    model_id: modelId,
    parameters: parameters
  };
  const response = await axios.put(url, payload, {
    headers: { 'Content-Type': 'application/json' }
  });
  return response.data;
}

// Chooch ImageChat-2 model_id
const modelIdImageChatPT = 'ad420c2a-d565-48eb-b963-a8297a0e4000';
const imageFileUrl = '';

const parameters = {
  deep_inference: true,
  prompt: 'is there smoke in this image ?',
  workflow_model_id: '1b626ec2-69e5-44b8-9cec-587c87fa8b0f',
  workflow_predict_objects_only: true,
  workflow_class_filter: '',
  workflow_padding: 0.6,
  workflow_conf_thresh: 0.3,
  workflow_nms_thresh: 0.45
};

// Replace with your own api key
const apiKey = 'YOUR_API_KEY';

// This is Chooch Worklow. A the smoke fire object detection model is merged Chooch IC2 PT. If the class_title is 'yes' in any of the predictions
// smoke / fire has been detected so you need to raise an alert
predictApi(modelIdImageChatPT, imageFileUrl, apiKey, parameters)
  .then(result => {
    console.log(result);
  })
  .catch(error => {
    console.error(error);
  });


														
														curl -X PUT "https://apiv2.chooch.ai/predict?api_key=" \
-H "Content-Type: application/json" \
-d '{
        "image_file_url": "",
        "model_id": "ad420c2a-d565-48eb-b963-a8297a0e4000",
        "parameters": {
            "deep_inference": true,
            "prompt": "is there smoke in this image ?",
            "workflow_model_id": "1b626ec2-69e5-44b8-9cec-587c87fa8b0f",
            "workflow_predict_objects_only": true,
            "workflow_class_filter": "",
            "workflow_padding": 0.6,
            "workflow_conf_thresh": 0.3,
            "workflow_nms_thresh": 0.45
        }
    }'

Base64 URL Post

Python
Javascript(REST)
cURL


														
														import base64
import json 
import requests
from io import BytesIO
from base64 import b64encode
import urllib.request


def predict_api(model_id, image_file, api_key=None, parameters=None):

    ENCODING = "utf-8"
    # 1. Reading the binary stuff
    # note the 'rb' flag
    # result: Bytes
    IMAGE_NAME = image_file

    with open(IMAGE_NAME, "rb") as open_file:
        byte_content = open_file.read()

    # 2. Base64 encode read data
    # result: bytes (again)
    base64_bytes = b64encode(byte_content)

    # 3. Decode these bytes to text
    # result: string (in utf-8)
    base64_string = base64_bytes.decode(ENCODING)

    payload = {
        "base64str": base64_string,
        "model_id": model_id,
        "parameters": parameters,
    }

    url = "https://apiv2.chooch.ai/predict?api_key={}".format(api_key)
    response = requests.put(url, data=json.dumps(payload))
    json_data = json.loads(response.content)
    return json_data


# Chooch ImageChat-2 model_id
model_id_image_chat_pt = "ad420c2a-d565-48eb-b963-a8297a0e4000"


# download file
image_file_url = ""

image_file = "test_image.jpg"
urllib.request.urlretrieve(image_file_url, "test_image.jpg")

parameters = {}

# default is True. If a prompt is given only 1 class will be returned and deep_detection will be turned off
parameters["deep_inference"] = True

parameters["prompt"] = "is there smoke this image ?"

# Adding workflow, this is optional.  It will apply the prompt to given objected model predictions as well
smoke_fire_object_detection_model_id = "1b626ec2-69e5-44b8-9cec-587c87fa8b0f"
parameters["workflow_model_id"] = smoke_fire_object_detection_model_id

parameters["workflow_padding"] = 0.6
parameters["workflow_conf_thresh"] = 0.3
parameters["workflow_nms_thresh"] = 0.45

# replace with your own api key
api_key = "YOUR_API_KEY"

return_val = predict_api(
    model_id_image_chat_pt, image_file, api_key=api_key, parameters=parameters
)

print(return_val)


														
														const axios = require('axios');
const fs = require("fs");

const imageStr = fs.readFileSync("your_local_image", { encoding: "base64" });

async function predictApi(modelId, imageStr, apiKey=null, parameters=null) {
  const url = "https://apiv2.chooch.ai/predict?api_key=${apiKey}";
  const payload = {
    base64str: imageStr,
    model_id: modelId,
    parameters: parameters
  };
  const response = await axios.put(url, payload, {
    headers: { 'Content-Type': 'application/json' }
  });
  return response.data;
}

// Chooch ImageChat-2 model_id
const modelIdImageChatPT = 'ad420c2a-d565-48eb-b963-a8297a0e4000';

const parameters = {
  deep_inference: true,
  prompt: 'is there smoke in this image ?',
  workflow_model_id: '1b626ec2-69e5-44b8-9cec-587c87fa8b0f',
  workflow_predict_objects_only: true,
  workflow_class_filter: '',
  workflow_padding: 0.6,
  workflow_conf_thresh: 0.3,
  workflow_nms_thresh: 0.45
};

// Replace with your own api key
const apiKey = 'YOUR_API_KEY';

// This is Chooch Worklow. A the smoke fire object detection model is merged Chooch ImageChat-2. If the class_title is 'yes' in any of the predictions
// smoke / fire has been detected so you need to raise an alert
predictApi(modelIdImageChatPT, imageStr, apiKey, parameters)
  .then(result => {
    console.log(result);
  })
  .catch(error => {
    console.error(error);
  });


														
														base64_string = $(base64 "your_local_image")
curl -X PUT "https://apiv2.chooch.ai/predict?api_key=" \
-H "Content-Type: application/json" \
-d '{
        "base64str": $base64_string,
        "model_id": "ad420c2a-d565-48eb-b963-a8297a0e4000",
        "parameters": {
            "deep_inference": true,
            "prompt": "is there smoke in this image ?",
            "workflow_model_id": "1b626ec2-69e5-44b8-9cec-587c87fa8b0f",
            "workflow_predict_objects_only": true,
            "workflow_class_filter": "",
            "workflow_padding": 0.6,
            "workflow_conf_thresh": 0.3,
            "workflow_nms_thresh": 0.45
        }
    }'

Sample Response


														
														{
  "model_id": "ad420c2a-d565-48eb-b963-a8297a0e4000",
  "model_title": "Chooch_VisionIC2-PT-Deep",
  "predictions": [
    {
      "class_title": "yes",
      "score": 0.486,
      "model_id": 1,
      "model_title": "general_chooch_ic2_pt",
      "prediction_keywords": [],
      "coordinates": {
        "xmin": 1399,
        "ymin": 564,
        "xmax": 1580,
        "ymax": 639
      }
    }
  ],
  "model_type": "Chooch_VisionIC2-PT-Deep",
  "prompt": "is there smoke this image ?.",
  "prediction_type": "Chooch_VisionIC2-PT-Deep",
  "image_width": 1080,
  "image_height": 1920,
  "image_id": "f76d6c0a-ad4a-41ab-a43e-a03bf093a1bc.jpg"
}

The Chooch Team is constantly working on improving the model to ensure optimal performance. Please share your feedback with us at [email protected] to help us improve the accuracy and effectiveness of our technology. We greatly value your input.