Add the Gluon Implementation of Deformable Convolution

[suyz526]

Description

The gluon implementation of deformable convolution

Checklist

Essentials

• Changes are complete (i.e. I finished coding on this PR)
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• For user-facing API changes, API doc string has been updated.

[suyz526]

@ChaiBapchya @sandeep-krishnamurthy @vandanavk @aaronmarkham @larroy

Hi,
things became complex after rebase...So I moved here with a new request.
I really have no idea about the Website error.

[suyz526]

http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html

and where can I find the BUILD_ID for a preview?

[aaronmarkham]

My guess about the Sphinx error is that it doesn't like this line.
Referring to the docs on citations, you can call it what you like, so maybe pick something more descriptive than "1" and maybe you won't have a collision. It actually says to use a non-numeric label.
Maybe to align with how this is categorized in the rst docs, call this section Citations, and then when anyone looks it up we know how it is supposed to work...

[suyz526]

Now I simplified the citations as it is in Gluon BN layer:
https://mxnet.incubator.apache.org/api/python/gluon/nn.html#mxnet.gluon.nn.BatchNorm

Besides, in the doc of Gluon.Contrib syncbatchnorm:
https://mxnet.incubator.apache.org/api/python/gluon/contrib.html?highlight=syncbatchnorm#id3

it seems to be able to create a hyperlink with an underline, e.g. [1]_

Now the website error is gone, thanks very much for your help.

[suyz526]

@mxnet-label-bot update [Gluon, pr-awaiting-review]

[aaronmarkham]

http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html

and where can I find the BUILD_ID for a preview?

If you click on "Show all checks" in the CI zone of this PR, you'll find one called "ci/jenkins/mxnet-validation/website". Click details. In the deploy stage (look for the green check marks), you'll see Julia selected. Select "Docs". Then click the last shell script. Then look all the way at the bottom of the output. There's your new link for a preview.

[aaronmarkham]

Latest preview: http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-14810/5/index.html

[suyz526]

http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
and where can I find the BUILD_ID for a preview?

If you click on "Show all checks" in the CI zone of this PR, you'll find one called "ci/jenkins/mxnet-validation/website". Click details. In the deploy stage (look for the green check marks), you'll see Julia selected. Select "Docs". Then click the last shell script. Then look all the way at the bottom of the output. There's your new link for a preview.

I see, thanks very much!

[wkcn]

LGTM. Thanks for your contribution!

[wkcn]

Merged. Thank you!

[lucinyaLi]

I met some error when I used the deformableconv layer presented above.
The error is :
MXNetError: [14:16:07] src/operator/contrib/././nn/deformable_im2col.h:94: only implemented in GPU
And I found some code you wrote
ctx = mx.gpu() net.initialize(force_reinit=True, ctx=ctx)
And in my code, only
if args.resume.strip(): net.load_parameters(args.resume.strip()) async_net.load_parameters(args.resume.strip()) else: with warnings.catch_warnings(record=True) as w: warnings.simplefilter("always") net.initialize() async_net.initialize()
I change my code to net.initialize(force_reinit=True, ctx=ctx), but there is another error :
RuntimeError: Parameter 'ssd0_vggatrousextractor0_init_scale' was not initialized on context cpu(0). It was only initialized on [gpu(1)].

Can I get your help ?

[suyz526]

Hi,

currently, the deformable convolution layer only supports the computation on GPU, so you need to make sure that all the parameter are stored on GPU. This can be done with

ctx = mx.gpu()  #ctx = mx.gpu(gpu_id)  
net.initialize(ctx=ctx) 

or in your case,
net.load_parameters(path, ctx=ctx)

Do you have some CustomOp in your network, which only supports CPU?

[lucinyaLi]

@ suyz526
This is a part of my net, I change some original conv layer in SSD into the deformable conv. There is nothing else I have changed.
`

with self.name_scope():
        # we use pre-trained weights from caffe, initial scale must change
        init_scale = mx.nd.array([0.229, 0.224, 0.225]).reshape((1, 3, 1, 1)) * 255
        self.init_scale = self.params.get_constant('init_scale', init_scale)
        self.stages = nn.HybridSequential()
        for l, f in zip(layers, filters):
            stage = nn.HybridSequential(prefix='')
            with stage.name_scope():
                    for _ in range(l):
                        stage.add(DeformableConvolution(f, kernel_size=3, padding=1, **self.init))
                        if batch_norm:
                            stage.add(nn.BatchNorm())
                        stage.add(nn.Activation('relu'))
            self.stages.add(stage)

and then
net.initialize(ctx=ctx) # where ctx=gpu(1)

net.load_parameters(path, ctx=ctx)`

I think all the parameter are stored on GPU. Is it right ? But the error :
RuntimeError: Parameter 'ssd0_vggatrousextractor0_init_scale' was not initialized on context cpu(0). It was only initialized on [gpu(1)].
What the error mean is that I should initialize the self.init_scale in CPU?

[suyz526]

CustomOp
This is a part of my net, I change some original conv layer in SSD into the deformable conv. There is nothing else I have changed.
with self.name_scope():
init_scale = mx.nd.array([0.229, 0.224, 0.225]).reshape((1, 3, 1, 1)) * 255
self.init_scale = self.params.get_constant('init_scale', init_scale)
self.stages = nn.HybridSequential()
for l, f in zip(layers, filters):
stage = nn.HybridSequential(prefix='')
with stage.name_scope():
for _ in range(l):
stage.add(DeformableConvolution(f, kernel_size=3, padding=1, **self.init))
if batch_norm:
stage.add(nn.BatchNorm())
stage.add(nn.Activation('relu'))
self.stages.add(stage)
and then
net.initialize(ctx=ctx) # where ctx=gpu(1)
net.load_parameters(path, ctx=ctx)
I think all the parameter are stored on GPU. Is it right ? But the error :
RuntimeError: Parameter 'ssd0_vggatrousextractor0_init_scale' was not initialized on context cpu(0). It was only initialized on [gpu(1)].
What the error mean is that I should initialize the self.init_scale in CPU?

Try
init_scale = mx.nd.array([0.229, 0.224, 0.225], ctx=ctx).reshape((1, 3, 1, 1)) * 255

[lucinyaLi]

@suyz526
init_scale = mx.nd.array([0.229, 0.224, 0.225], ctx=mx.gpu(1)).reshape((1, 3, 1, 1)) * 255
Or
init_scale = mx.nd.array([0.229, 0.224, 0.225], ctx=mx.cpu(0)).reshape((1, 3, 1, 1)) * 255
I tried the two ways.
It does no work. the error still exists.

[suyz526]

@suyz526
init_scale = mx.nd.array([0.229, 0.224, 0.225], ctx=mx.gpu(1)).reshape((1, 3, 1, 1)) * 255
Or
init_scale = mx.nd.array([0.229, 0.224, 0.225], ctx=mx.cpu(0)).reshape((1, 3, 1, 1)) * 255
I tried the two ways.
It does no work. the error still exists.

sry, this step is unnecessary. How about your input data?

[lucinyaLi]

@suyz526
init_scale = mx.nd.array([0.229, 0.224, 0.225], ctx=mx.gpu(1)).reshape((1, 3, 1, 1)) * 255
Or
init_scale = mx.nd.array([0.229, 0.224, 0.225], ctx=mx.cpu(0)).reshape((1, 3, 1, 1)) * 255
I tried the two ways.
It does no work. the error still exists.

sry, this step is unnecessary. How about your input data?

Nonono. I am very grateful for your help.
Just now, I found that this error is irrelevant to the deformable conv layers. It exists also in original SSD .
When I want to change the original code 'net = get_model(net_name, pretrained_base=True)' into 'net = get_model(net_name, pretrained_base=True, ctx=ctx)', the error exists.
And I ask this question in the SSD gluon forum, the URL is :
https://discuss.gluon.ai/t/topic/12903
I am looking forward for your further help if you want. And thanks again.