Thank you for your great work. I have been applying your research to my problem now.
Here is my problem:

DEM, stands for Digital Elevation Model, is a numerical matrix with each pixel represents its elevation correspond to a location.

SRTM is a globally DEM dataset. However, with a reservoir or a lake, SRTM showed the elevation of surface water at the time it collected (in 2000). Now I want to recover DEM below surface water of a lake or a reservoir.

Here is my step-by-step process:

1. My training set is 15k DEM images.

2. I generated masks correspond to input image. An input image has a specific mask. My mask is an connected area which has elevation lower than a random number. So it can describe a reservoir or a lake, I think so. Some of my masks with the input:

So I customized your code to read my input:

3. I also customized data_from_fnames in neuralgym toolkit to read '.tif' files. Then I used min-max normalize DEM image to 0-255 in order to pass it to your model.

4. Some first rows in my train.flist:

5. Because of limit memory of my GPU, I set image input shape (128, 128, 1) and batch size is 16 as your advice I saw in other issues.

#training
train_spe: 1000 
max_iters: 1000000
viz_max_out: 10
val_psteps: 500

static_view_size: 30
img_shapes: [128, 128, 1]
height: 128
width: 128
max_delta_height: 32
max_delta_width: 32
batch_size: 16
vertical_margin: 0
horizontal_margin: 0

6. I finally trained more than 70 epoches and get bad results:

My losses are not converging

Some of generated validation images:

Questions

1. Are all my customizations correct?
2. I saw your sample flist is shuffled and I didn't. Does it affect to the training result?
3. Could you give me some suggestion and your views on my problem?

Thanks a lot! <3