Post-processing
The Edge AI Manager is equipped with a number post-processing algorithms triggered contextually based on the ONNX's input/output names. In other words, the output JSON returned after an inference run might differ from the model's raw output, because that output has been processed by one of the algorithms to extract more insights from those raw values.
Below is a list of the currently integrated post-processing algorithms, along with the output JSON structure returned by the AI Manager.
This algorithm is mainly used to count the number of unique entities moving from one side of the image to the other, over a period of time. It can be used for instance to count:
- number of people entering and exiting a building
- number of vehicles passing by a road
The line-crossing algorithm should be fed an output that represents bounding-boxes delimiting entities. Thus, the output should be something like a N by 4 matrix, where N is number of detected entities, and each row contains at least entity's coordinates.
Other than the usual meta-data included in the JSON, additional fields are included, namely:
{ ..., "output":[ a, b], ...}Where:
- a: number of unique entities moving from left to right side of line
- b: number of unique entities moving from right to left side of line
For example, when using this model from the Scailable Free Models catalog, here's a complete JSON output:
[
{
"deviceId": "8e410fb88d924ab2900ffe79cc6bdc5e-145862338700776",
"modelId": "3a21caaa-2ea7-4e32-83cf-9994cb65545a",
"sourceId": "input-0",
"sourceName": "",
"timestamp": 1675421846075,
"output": [
2,
1
]
}
]
The used model is a people locator. The post-processor inferred that 2 people moved from left to right, and 1 person moved left.
For more information regarding the line-crossing parameters and where to specify their parameters, check the AI Manager's page.
This post-processor is capable of decoding characters of a license plate.
- Toll automation
- Restricted area monitoring
- Vehicle tracking
- Unique cars counting
Given the location of a license plate (LP) in an input image, the ANPR algorithm deciphers its characters and returns them as a readable string. In other words, an ML model is needed to locate the LP and return its location in this format: (x1, y1, x2, y2, confidence threshold, the 4 LP corners coordinates). So, the output should should contain 13 values.
Other than the usual meta-data included in the JSON, additional fields are included, namely:
[
{
...
"output": {
"licensePlate": "",
"lastEstimatePlate": "",
"coordinates": [
0,
0,
0,
0
]
},
"size": ,
"direction": ,
...
}
]
Where:
- licensePlate: A string containing the rectified license plate characters based on previous detections.
- lastEstimatePlate: A string representing the last detected license pate characters.
- coordinates: The (x1, y1, x2, y2) coordinates of the detected license plate.
- size: The estimated area of the license plate (Height x Width)
- direction: An integer denoting whether the license plate (and hence the car) is moving away from camera (value = 2), moving closer to camera (value = 1), not moving (value = 0) or unknown when no license plate is detected for instance (value = 3).
For example, when using this model from the Scailable Free Models catalog, here's a complete JSON output:
[
{
"deviceId": "8e410fb88d924ab2900ffe79cc6bdc5e-145862338700776",
"modelId": "1eda1ec4-cb15-4919-9be4-bf32bb87c439",
"sourceId": "input-0",
"sourceName": "",
"timestamp": 1676024792051,
"output": {
"licensePlate": "",
"lastEstimatePlate": "G241GH",
"coordinates": [
0,
0,
0,
0
]
},
"size": 6764,
"direction": 3,
"sourceDate": "",
"outputType": "json"
}
]
This model detected a car with a license plate G241GH.
Since this is the result of the first inference run, the
coordinates are zero-ed and licensePlate is empty.If a model returns boolean values, they can be exploited to trigger an alarm somewhere based on those values. This trigger can serve to:
- detect intruders
- monitor restricted areas
To support alarm trigger for a model, it needs to have an output following a specific pattern. The output name should start with alarm-.
For more information, please check the named IO page.
Other than the usual meta-data included in the JSON, additional fields are included, namely:
[
{
...
"alarm": 0,
"baseRGB": "",
"h": 0,
"w": 0,
"c": 0,
...
}
]
Where:
- alarm: is a boolean value. It's 1 when an alarm is raised, 0 otherwise.
- baseRGB: A base64-encoded string containing the raw pixel values of the image triggering the alarm.
- h, w, and c: stand for height, width and number of channels respectively. They are used to re-construct the original image with the right dimensions.
Please note that
baseRGB, h, w, and c may not be present in the output JSON based on your configuration of the AI Manager.
For information, check the output configuration section.For example, when using the people detector from the Scailable Free Models catalog, here's a complete JSON output:
[
{
"deviceId": "8e410fb88d924ab2900ffe79cc6bdc5e-145862338700776",
"modelId": "6e7cb493-f310-4491-a387-3ef59f16b58e",
"sourceId": "input-0",
"sourceName": "",
"timestamp": 1676035054620,
"alarm": 1,
"outputType": "json",
"outputFormat": "namedObject",
"outputDims": [
[
1
]
],
"outputDataTypes": [
9
],
"output": {
"alarm-0:person": [
1
]
}
}
]
The model detect at least one person, and the alarm is raised but without sending the RGB image to the cloud.
If a model returns two outputs: one containing proposed bounding-boxes and an other for their respective scores, the NMS algorithm is used to only keep pertinent bounding-boxes, ie. remove those which the model is less confident about. NMS serves to:
- Locate and count faces when coupled with the face locator model
Other than the usual meta-data included in the JSON, additional fields are included, namely:
[
{
...
"output": {
"count": 2,
"boxes": [
288,
16,
321,
65,
290,
182,
320,
232
]
}
...
}
]
Where:
- count: is a non-negative integer representing the number of pertinent boxes.
- boxes: a sequence of 4-tuple containing (x1, y1, x2, y2) coordinates.
For example, when using the face locator from the Scailable Free Models catalog, here's a complete JSON output:
[
{
"deviceId": "8e410fb88d924ab2900ffe79cc6bdc5e-145862338700776",
"modelId": "0d1a91bd-1474-11ec-af2f-9600004e79cc",
"sourceId": "input-0",
"sourceName": "",
"timestamp": 1678280224597,
"sourceDate": "",
"outputType": "json",
"outputFormat": "namedObject",
"outputDataTypes": [
1
],
"output": {
"count": 12,
"boxes": [
288,
16,
321,
65,
290,
182,
320,
232,
-1,
8,
30,
68,
64,
17,
102,
67,
139,
11,
178,
68,
215,
10,
258,
71,
26,
101,
70,
150,
110,
100,
149,
155,
217,
98,
253,
152,
56,
180,
97,
234,
143,
182,
182,
232,
220,
179,
257,
235
]
}
}
]
This post-processor blurs the face recognized in a stream and makes them accessible locally for use. This blurring algorithm can serve to:
- Preserve users privacy
- Build a GDPR-compliant application
Other than the usual meta-data included in the JSON, additional fields are included, namely:
[
{
...
"output": {
"count": 2,
"boxes": [
288,
16,
321,
65,
290,
182,
320,
232
]
}
...
}
]
Where:
- count: is a non-negative integer representing the number of pertinent boxes.
- boxes: a sequence of 4-tuple containing (x1, y1, x2, y2) coordinates.
For example, when using the face locator from the Scailable Free Models catalog, here's a complete JSON output:
[
{
"deviceId": "8e410fb88d924ab2900ffe79cc6bdc5e-145862338700776",
"modelId": "0d1a91bd-1474-11ec-af2f-9600004e79cc",
"sourceId": "input-0",
"sourceName": "",
"timestamp": 1678280224597,
"sourceDate": "",
"outputType": "json",
"outputFormat": "namedObject",
"outputDataTypes": [
1
],
"output": {
"count": 12,
"boxes": [
288,
16,
321,
65,
290,
182,
320,
232,
-1,
8,
30,
68,
64,
17,
102,
67,
139,
11,
178,
68,
215,
10,
258,
71,
26,
101,
70,
150,
110,
100,
149,
155,
217,
98,
253,
152,
56,
180,
97,
234,
143,
182,
182,
232,
220,
179,
257,
235
]
}
}
]
This post-processor is tested only on the face locator with blurring model in the Scailable free catalog.
This post-processor determines the dominant color inside each recognized bounding-box. It could be used to:
- detect color of cars
Other than the usual meta-data included in the JSON, additional fields are included, namely:
[
{
...
"colors": [
"Black"
],
...
}
]
Where:
- colors: is a list recognized colors.
This post-processor is used to decode bar and QR codes which are recognized in an image, for instance it can serve to:
- Product counting
- Identify products in store
- Read links, information from images
Other than the usual meta-data included in the JSON, additional fields are included, namely:
[
{
...
"output": {
"barCodes": [
"welcome to scailable!",
"Scailable-2023"
]
},
...
}
]
Where:
- barCodes: is a list of data encoded in each detect bar/qr code.
This post-processor is used to send input images with the output JSON based on the model's output on that image. More precisely, this works with object detection models where an image is uploaded if the model is less confident about a certain image.
To configure this post-processor on the device, check out this page.
This post-processor can be used to:
- Grab images from deployment environment to finetune the model.
Other than the usual meta-data included in the JSON, additional fields are included, namely:
[
{
...
"uncertain": 0,
"baseRGB": "",
"h": 512,
"w": 512,
"c": 3,
...
}
]
Where:
- uncertain: is a boolean value. It's 1 when model is uncertain about the input, and 0 otherwise.
- baseRGB: A base64-encoded string containing the raw pixel values of the image triggering the uncertainty condition. It's empty if the condition is not met.
- h, w, and c: stand for height, width and number of channels respectively. They are used to re-construct the original image with the right dimensions.
This post-processor is only available for EdgeImpulse models.
Last modified 6d ago