add pos_weight for SigmoidBinaryCrossEntropyLoss

[eureka7mt]

Description

Add pos_weight for SigmoidBinaryCrossEntropyLoss.
A value pos_weights > 1 decreases the false negative count, hence increasing the recall.
Conversely setting pos_weights < 1 decreases the false positive count and increases the precision.
This can be seen from the fact that pos_weight is introduced as a multiplicative coefficient for the positive targets term in the loss expression:
label * -log(sigmoid(pred)) * pos_weight + (1 - label) * -log(1 - sigmoid(pred))

It's adopted from tensorflow's implementation

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

[roywei]

@eureka7mt Thanks for the contribution, could you add a unit test for this case?

[roywei]

@mxnet-label-bot add[Gluon, pr-awaiting-review]

[anirudhacharya]

minor comment on the unit test and fix the CI failure.

Rest LGTM.

[stu1130]

@eureka7mt Could you fix the Trailing whitespace issue?

[eureka7mt]

Don't know why it failed in test_multinomial_generator() which is in the file '/work/mxnet/tests/python/gpu/../unittest/test_random.py' with unix-gpu.

[vandanavk]

@eureka7mt could you re-trigger the CI?

[ankkhedia]

@eureka7mt Could you please look into the CI failures?

[anirudhacharya]

@mxnet-label-bot update [pr-awaiting-merge]

[wkcn]

We will merge it after the CI passes.
Thanks for your contribution!

[eureka7mt]

Adding the if-else statement make an error.Though the default value of pos_weight is set to be 1,the pos_weight is usually an (1,N) NDArray.And it seems that an error happen in if-else statement when input is a symbol

[wkcn]

@eureka7mt
I see. Since pos_weight is a tensor, it is better to default pos_weight=None as you wrote before. I will update it. Thanks!

Edit: I think the pos_weight is a scalar, since it is a binary classification loss.

[eureka7mt]

@wkcn It could be a scalar for classifying a single class.But for multi-class and multi-label classifying,it should be a tensor.Because in this situation,the number of positive examples and negative examples isn't same for each class.
And in pytorch,it's also defined as a tensor.See the pytorch docs.

[wkcn]

I change the order of SigmoidBinaryCrossEntropyLoss inputs from (self, F, pred, label, pos_weight=None, sample_weight=None) to (self, F, pred, label, sample_weight=None, pos_weight=None), since we need the compatibilty for other projects

[wkcn]

Sorry that I trigger the sanity problem.
Could someone please help me solve it? Thanks!

[eureka7mt]

Maybe the broadcast_mul isn't necessary.I think that a NDArray * a NDArray will do broadcast_mul automatically.

[wkcn]

@eureka7mt I think we may pass Symbol into SigmoidBinaryCrossEntropyLoss. Symbol will not broadcast_mul automatically in my test.

import mxnet as mx
from mxnet.gluon import nn

class TestBlock(nn.HybridBlock):
    def __init__(self):
        super(TestBlock, self).__init__()
    def hybrid_forward(self, F, x, y):
        return x * y

block = TestBlock()
block.hybridize()
a = mx.nd.zeros((10, 1))
b = mx.nd.ones((1, 5))
c = block(a, b)
print (c.asnumpy())

[wkcn]

The PR has been merged.
Thanks for your contribution: )