Task: Text detection (can detect slanted text)

• contributions
  
  • Proposed an End-to-End full convolutional network to solve the text detection problem
  • Can generate geometric annotations in two formats: quardrangles or rotated boxes according to specific applications
  • Improved state-of-the-art method
• The core idea of ​​the algorithm: the main idea comes fromU-Net, U-shaped structure is used to obtain 1, pixel-level segmentation prediction results. 2. Pixel-level geometric prediction results. According to the results of 1 and 2, the coordinates of the four vertices of each bounding box can be calculated. Then use NMS to delete redundant and duplicate bounding boxes.

• Algorithm flow:

  • Training phase
    
  • Test phase
    
  • The U-Net network structure based on ResNet is shown in the figure below
    

• Algorithm details:

  • How to calculate ground truth?
    
    • The ground truth corresponding to the score: is to shrink the original bounding box by 0.3r inward according to the short side length r. In fact, I don’t know why this step is necessary to remove noise?
    • Ground truth corresponding to geometry: Let's take RBOX type data as an example, as shown in the figure below. For each point inside the bounding box, we calculate their distance to the top, bottom, left, and right sides, and calculate the angle. For the points outside the bounding box, we set the ground truth to 0.
      

  • Loss function?

    • For score: We are using balanced cross-entropy. This can balance the impact of the imbalance of positive and negative samples. Its definition is shown below. The implementation code is as follows:
      

      Ls=balanced−xent(Ŷ ,Y∗)=−βY∗log(Ŷ )−(1−β)(1−Y∗)log(1−Ŷ )β=1−∑y∗∈Y∗y∗|Y∗| $$\begin{gathered} L_s=balanced-xent(\hat{Y},Y^{\ast }) \\ =-\beta Y^{\ast }log(\hat{Y})-(1-\beta )(1-Y^{\ast })log(1-\hat{Y}) \\ \beta =1-\frac{\underset{y^{\ast }\in Y^{\ast }}{\sum }{y}^{\ast }}{|Y^{\ast }|} \end{gathered}$$

      def cross_entropy(y_true_cls, y_pred_cls, training_mask):
          '''
          :param y_true_cls: numpy array
          :param y_pred_cls: numpy array
          :param training_mask: numpy array
          :return:
          '''
          # eps = 1e-10
          # y_pred_cls = y_pred_cls * training_mask + eps
          # y_true_cls = y_true_cls * training_mask + eps
          # shape = list(np.shape(y_true_cls))
          # beta = 1 - (np.sum(np.reshape(y_true_cls, [shape[0], -1]), axis=1) / (1.0 * shape[1] * shape[2]))
          # cross_entropy_loss = -beta * y_true_cls * np.log(y_pred_cls) - (1 - beta) * (1 - y_true_cls) * np.log(
          #     1 - y_pred_cls)
          # return np.mean(cross_entropy_loss)
          eps = 1e-10
          y_pred_cls = y_pred_cls * training_mask + eps
          y_true_cls = y_true_cls * training_mask + eps
          each_y_true_sample = tf.split(y_true_cls, num_or_size_splits=FLAGS.batch_size_per_gpu, axis=0)
          each_y_pred_sample = tf.split(y_pred_cls, num_or_size_splits=FLAGS.batch_size_per_gpu, axis=0)
          loss = None
          for i in range(FLAGS.batch_size_per_gpu):
              cur_true = each_y_true_sample[i]
              cur_pred = each_y_pred_sample[i]
              beta = 1 - (tf.reduce_sum(cur_true) / (FLAGS.input_size * FLAGS.input_size))
              cur_loss = -beta * cur_true * tf.log(cur_pred) - (1-beta) * (1-cur_true) * tf.log((1-cur_pred))
              if loss is None:
                  loss = cur_loss
              else:
                  loss = loss + cur_loss
          return tf.reduce_mean(loss)

    • For geometry: we calculate its IoU loss, which is defined as follows:
      

      LAABB=−logIoU(R̂ ,R∗)=−log|R̂ ∩R∗||R̂ ∪R∗|wi=min(d2^,d∗2)+min(d4^,d∗4)hi=min(d3^,d∗3)+min(d1^,d∗1)R̂ =(d1^+d3^)∗(d2^+d4^)R∗=(d∗1+d∗3)∗(d∗2+d∗4)|R̂ ∩R∗|=wi∗hi|R̂ ∪R∗|=R̂ +R∗−|R̂ ∩R∗| $$\begin{gathered} L_{AABB}=-logIoU(\hat{R},R^{\ast })=-log\frac{|\hat{R}\cap R^{\ast }|}{|\hat{R}\cup R^{\ast }|} \\ {w}_i=min(\widehat{d_2},d_2^{\ast })+min(\widehat{d_4},d_4^{\ast }) \\ {h}_i=min(\widehat{d_3},d_3^{\ast })+min(\widehat{d_1},d_1^{\ast }) \\ \hat{R}=(\widehat{d_1}+\widehat{d_3})\ast (\widehat{d_2}+\widehat{d_4}) \\ {R}^{\ast }=(d_1^{\ast }+d_3^{\ast })\ast (d_2^{\ast }+d_4^{\ast }) \\ |\hat{R}\cap R^{\ast }|=w_i\ast h_i \\ |\hat{R}\cup R^{\ast }|=\hat{R}+R^{\ast }-|\hat{R}\cap R^{\ast }| \end{gathered}$$

• doubt?

  • In the specific implementation, the mask is not upscaled to the size of the original image, but only upscaled to 1/4 the size of the original image. It is said that this can better detect small text information. I don't know the principle?
  • In the specific implementation, regarding the loss of the score, the balanced cross-entropy loss is not used but the dice loss is used.