Hello all, I create the thread to discuss about the gated convolution and its input. Many questions in the repo. have been asked for the problem #405, #252.

First of all, the input of gated convolution are: uncompleted image img and its mask. In which, the mask defined as

for image inpainting, the input are composed of both regions with valid pixels/features outside holes and invalid pixels/features (in shallow layers) or synthesized pixels/features (in deep layers) in masked regions

in which M is the corresponding binary mask, 1 represents pixel in the location (y, x) is valid, 0 represents the pixel is invalid

However, the official code has some confusing, that is the reason why many questions has been asked. In the code,

From these lines, we can see that the mask defined as pixels 0 for outside the box and 1 for inside the box. Hence, the paper has a typo, it should be

in which M is the corresponding binary mask, 0 represents pixel in the location (y, x) is valid, 1 represents the pixel is invalid

So, the above lines answered the question #405, #252

Secondly, the incompleted image x defined as

It shows that the incompleted input is obtained by multiply the original image and (1-mask). It means we want to keep the original pixels of the image outside of the mask and set to zero pixels inside of the mask

Now, we will concatenate the incompleted input x and mask to obtain BxHxWx(channel_x + channel+mask). For example, the channel of x is 3 (RGB) and the channel of the mask is 1 (binary image). We have the input for gated convolution is BxHxWx4. Now, we will feed it to the gated convolution.

     x = tf.layers.conv2d( x, cnum, ksize, stride, dilation_rate=rate, activation=None, padding=padding, name=name)
    x, y = tf.split(x, 2, 3)
    x = activation(x)
    y = tf.nn.sigmoid(y)
    x = x * y

We can see that the concatenation of incompleted input and mask (defined as x) will go to a conv2d. The output then splits into two components: x is for convolution result of incompleted x, and y is for convolution result of mask. Then y goes to the sigmoid function to normalize it to [0,1] and multiply with activation(x) to perform gating.

Thanks for reading, please correct if I was wrong!