Synthetic Bounding Box Generation

A step further to generate artificial bounding boxes for balanced training in object detectors

Photo by Rumman Amin on Unsplash

While training object detection networks, almost all of us must have gone through the pain of dataset balancing. Since an imbalance dataset can create a lot of issues in accuracy numbers during inference. This article is dedicated to one such method which generates synthetic bounding boxes based on given ground truth boxes.

Generating Positive Bounding Boxes for Balanced Training of Object Detectors¹ focuses on :

1. Bounding box generation and pRoI generation
2. Imbalance problems and analysis of RPN ROIs,
3. Practical improvements.

However, this article will cover only the main algorithm of the bounding box generation and its code snippets.

Motivation

An imbalanced dataset are mostly is Foreground-Background and the other is Foreground-Foreground imbalance.

1. Foreground-Background: The issue occurs as negative class samples are comparatively more than that of foreground classes. Few solutions to overcome this issue are listed below:
   -> Hard Sampling
   -> Soft Sampling
   -> Prime Sampling
   -> Gradient harmonizing mechanism
2. Foreground-Foreground: The issue mostly occurs when one class has more number of ground truth samples than others. To overcome this issue we do class-aware Sampling.
3. BB IoU imbalance: The IoU distribution of the ROIs generated by the region proposal network is imbalanced, which biases the BB regressor in favor of the IoU that the distribution is skewed towards.

Bounding Bounding Generator

The main goal of this algorithm is to generate a spatial distribution in which generated synthetic box results in IoU > T(threshold).

Algorithm: Bounding Box Generator

The proposed algorithms flow given in Fig-2.

Fig-2: Flow of the Bounding Box generation

Stepwise function explanation:

1. findTopLeftPointBorders: Calculates boundary points of the top left corner based on provided ground truth box and threshold IoU (Detailed explanation is given using code and visualization in the 1.1 section).
2. Sample polygon: Given above boundary points function defines a polygon and returns sample point that lies in the polygon.
3. findBottomRightBorders: Function used top-left sample point and calculates boundary points for the bottom-right corner (Detailed explanation is given using code and visualization in the 1.2 section).
4. Sample polygon: Defines polygon given bottom-right boundary points and generates bottom right boundary points.

The above flow is performed on each ground truth bounding boxes of every IoU threshold that we have decided.

Definitions and Notations

B(bounding box) :- [x1,y1,x2,y2]
LT(top left)-[x1,y1] and BR(bottom right)-[x2,y2]
B’:- Generated bounding box
B:- Ground truth bounding box

Fig-3: Regions for top-left and bottom-right coordinate referencing; Source[link]

Fig-3 Depicts different regions. Boundary points are generated for each region separately.

1.1 findTopLeftPointBorders

The top left boundary points are generated separately in four regions(refer to Fig-3). Each region produces an array of points(x1’,y1’) given ground truth bounding box and T(IoU threshold).

Fig-4 [a,b,c,d] provides region-wise equations for boundary point generation based on which code is written and output is generated.

Atributes Required:
box, ##Ground Truth box
IoU, ##T- IoU threshould
boxArea ## area of gound truth box

Fig-4(a): Output and Equation (source) of Quadrant 1

#1st Quadrant BoundingPoints Generation 
x1TR = np.arange(box[0], box[2] - IoU * (box[2] - box[0]), step=IoULimitPrecision)
I = (box[2] - x1TR) * (box[3] - box[1])
 
y1TR = box[3] - (I / IoU - boxArea + I) / (box[2] - x1TR)

Fig-4(b): Output and Equation (source) of Quadrant 2

#2nd Quadrant Boundary Points Generation
y1BR = np.arange(box[1], box[3] - (boxArea * IoU) / (box[2] - box[0]), step=IoULimitPrecision)
x1BR = box[2] - ((boxArea * IoU) / ((box[3] - y1BR)))

Fig-4(c): Output and Equation (source) of Quadrant 3

#3rd Quadrant Boundary Points Generation
y1BL = np.arange(box[1], box[3] - (boxArea * IoU) / (box[2] -       box[0]), step=IoULimitPrecision)
I = (box[2] - box[0]) * (box[3] - y1BL)
x1BL = box[2] - (I / IoU - boxArea + I) / (box[3] - y1BL)
inv_idx = np.arange(y1BL.shape[0] - 1, -1, -1).astype(int)
y1BL = y1BL[inv_idx]
x1BL = x1BL[inv_idx]

Fig-4(d): Output and Equation (source) of Quadrant 4

#4th Quadrant Boundary Points Generation
y1TL = np.arange((((box[3] * (IoU - 1)) + box[1]) / IoU), box[1], step=IoULimitPrecision)
x1TL = box[2] - (boxArea / (IoU * (box[3] - y1TL)))
inv_idx = np.arange(y1TL.shape[0] - 1, -1, -1).astype(int)
y1TL = y1TL[inv_idx]
x1TL = x1TL[inv_idx]

Fig-4(e): Example of Sample Point for Top left

Sample Point Generation: Once we have boundary points, it is easy to define a polygon(spatial distribution) with sample space in which any point is considered valid. As shown in Fig-4 Top left feasible point.

1.2 findBottomRightBorders

Since we have Top left sample point ((x1',y1') coordinates) we can now derive bottom right boundary points (as shown in Fig-5[a,b,c,d])given T(IoU threshold).

Fig-5[a,b,c,d] provides region-wise equations for boundary point generation based on which code is written and output is generated.

Attributes Required: 
box,IoU,boxArea,
### Top Left sample points
proposedx1, proposedy1,
###Calculated based on TL sample point and IoU threshoumd
limitLeftX, limitRightX, limitTopY, limitBottomY

Fig-5(a): Output and Equation (source) of Quadrant 1

#1st Quadrant Boundary Points Generation
y2TR = np.arange(limitTopY, box[3] + IoULimitPrecision, step=IoULimitPrecision)
yBnew = np.minimum(y2TR, box[3])
Inew = np.clip(xB - xA, a_min=0, a_max=sys.maxsize) * np.clip(yBnew - yA, a_min=0, a_max=sys.maxsize)
x2TR = (Inew / IoU - boxArea + Inew) / (y2TR - proposedy1)
x2TR += proposedx1

Fig-5(b): Output and Equation (source) of Quadrant 2

#2nd Quadrant Boundary Points Generation
x2BR = np.arange(limitRightX, box[2] - IoULimitPrecision, step=-IoULimitPrecision)
y2BR = (I / IoU - boxArea + I) / (x2BR - proposedx1)
y2BR += proposedy1

Fig-5(c): Output and Equation (source) of Quadrant 3

#3rd Quadrant Boundary Points Generation
y2BL = np.arange(limitBottomY, box[3] - IoULimitPrecision, step=-IoULimitPrecision)
yBnew = np.minimum(y2BL, box[3])
x2BL = IoU * boxArea + xA * IoU * (yBnew - yA) + xA * (yBnew - yA) - IoU * proposedx1 * (y2BL - proposedy1)
x2BL /= ((IoU + 1) * (yBnew - yA) - IoU * (y2BL - proposedy1))

Fig-5(d): Output and Equation (source) of Quadrant 4

#4th Quadrant Boundary Points Generation
x2TL = np.arange(limitLeftX, box[2] + IoULimitPrecision, step=IoULimitPrecision)
xBnew = np.minimum(x2TL, box[2])
y2TL = IoU * boxArea + IoU * (xBnew - xA) * yA + yA * (xBnew - xA) - IoU * proposedy1 * (x2TL - proposedx1)
y2TL /= ((IoU + 1) * (xBnew - xA) - IoU * (x2TL - proposedx1))

Fig-5(e): Output and Equation of Quadrant 1

Since we have the bottom right boundary point feasible space (polygon2) can be defined. If the point lies in the feasible space it is a valid point (example point is shown in Fig-4).

Fig-6 red box is synthetic bounding box and blue box is ground truth box

Finally, as we have both top-left sample point and bottom-right sample point we can generate bounding as shown in Fig-6 with 0.7 IoU threshold.

Final Code Snippet

Numpy version of the original code

The code snippets given in the article is inspired and taken from the GitHub code of Generating Positive Bounding Boxes for Balanced Training of Object Detectors for reference.

I hope you found the content meaningful. If you want to stay updated with the latest research in AI, please follow us at VisionWizard

References

[1] Generating Positive Bounding Boxes for Balanced Training of Object Detectors

[2] Original GitHub Repository