Make to_tensor and normalize to accept 3D or 4D tensor inputs #13614

sandeep-krishnamurthy · 2018-12-11T08:59:56Z

Description

With this change:

ToTensor transformation operator can take 3D (h, w, c) or 4D (n, h, w, c) at once.
Normalize transformation operator can take 3D (c, h, w) or 4D (n, c, h, w) at once.
Also parallelized the operator with omp
This change will be required to fuse transformation pipeline with network graph during inference. Where inputs are usually 4D inputs (n, c, h, w).
This is backward compatible change, this do not change any existing behavior.

In another PR, I will work on to support list of 3D tensors (list of images) as input for different shape image tensors use case.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Support 3D or 4D input tensors to to_tensor and normalize transformation operator.

TODO

API documentation updates. (WIP)

@stu1130 @zhreshold @apeforest

…transforms

stu1130

Thanks @sandeep-krishnamurthy

stu1130 · 2018-12-11T21:13:38Z

src/operator/image/image_random-inl.h

+
+      for (int i = 0; i < channel; ++i) {
+        DType mean = param.mean[param.mean.ndim() > 1 ? i : 0];
+        DType std_dev = param.std[param.std.ndim() > 1 ? i : 0];


mean and std_dev should be float type defined by line 115, 116

stu1130 · 2018-12-11T21:15:24Z

src/operator/image/image_random-inl.h

+  MSHADOW_TYPE_SWITCH(inputs[0].type_flag_, DType, {
+      float* output = outputs[0].dptr<float>();
+      DType* input = inputs[0].dptr<DType>();
+


Suggested change

#pragma omp parallel for collapse(2)

would this make better performance?

Cannot be used inside Macro.

How about checking the DType by ourself without using the Macro

stu1130 · 2018-12-11T21:20:46Z

src/operator/image/image_random-inl.h

+        DType mean = param.mean[param.mean.ndim() > 1 ? i : 0];
+        DType std_dev = param.std[param.std.ndim() > 1 ? i : 0];
+        for (int j = 0; j < length; ++j) {
+          output[step + i*length + j] = (input[step + i*length + j] - mean) / std_dev;


if input is int, should it be int or float after nomarlization? I prefer float here

Good point, it should be float. Making the change.

src/operator/image/image_random-inl.h

tests/python/unittest/test_gluon_data_vision.py

roywei · 2018-12-12T00:09:36Z

@mxnet-label-bot add[Gluon, Data-loading, Operator]

apeforest · 2018-12-12T17:39:26Z

python/mxnet/gluon/data/vision/transforms.py

+    Examples
+    --------
+    >>> transformer = transforms.Normalize(mean=(0, 1, 2), std=(3, 2, 1))
+    >>> image = mx.nd.random.uniform(0, 1, (3, 4, 2))


Can you give an example of 4D (N x C x H x W) here?

apeforest · 2018-12-12T17:40:33Z

src/operator/image/image_random-inl.h

+      << "Input image must have shape (height, width, channels), or "
+      << "(N, height, width, channels) but got " << shp;
+  if (shp.ndim() == 3) {
+    SHAPE_ASSIGN_CHECK(*out_attrs, 0, TShape({shp[2], shp[0], shp[1]}));


It would be clearer to define enum constant N, W, C, H instead of using 0, 1, 2, 3

apeforest · 2018-12-12T17:41:58Z

src/operator/image/image_random-inl.h

+
+      for (int l = 0; l < length; ++l) {
+        for (int c = 0; c < channel; ++c) {
+          output[step + c*length + l] = static_cast<float>(input[step + l*channel + c]) / 255.0f;


255.0f is already a float, so this cast may not be needed here.

Another question is why 255.0f? Maybe using a constant variable with clear name is more readable.

Agreed. Making the change.

apeforest · 2018-12-12T17:45:20Z

src/operator/image/image_random-inl.h

+
+    #pragma omp parallel for
+    for (auto n = 0; n < batch_size; ++n) {
+      ToTensorImpl(inputs, outputs, length, channel, n*step);


Good change!

apeforest · 2018-12-12T17:46:12Z

src/operator/image/image_random-inl.h

+      << "Input tensor must have shape (channels, height, width), or "
+      << "(N, channels, height, width), but got " << dshape;
+
+  uint32_t nchannels;


Use int or int32_t. See: https://google.github.io/styleguide/cppguide.html#Integer_Types

Thanks. Good learning!

apeforest · 2018-12-12T17:50:36Z

tests/python/unittest/test_gluon_data_vision.py

    data_in = np.random.uniform(0, 255, (300, 300, 3)).astype(dtype=np.uint8)
-    out_nd = transforms.ToTensor()(nd.array(data_in, dtype='uint8'))
+    out_nd = transforms.ToTensor()(nd.array(data_in))


Why remove the dtype in original 3D input test?

apeforest · 2018-12-12T17:51:16Z

tests/python/unittest/test_gluon_data_vision.py

+    assert_almost_equal(data_expected_4d, out_nd_4d.asnumpy())
+
+    # Invalid Input - Neither 3D or 4D input
+    invalid_data_in = nd.random.uniform(0, 1, (5, 5, 3, 300, 300)).astype(dtype=np.float32)


apeforest · 2018-12-12T17:53:26Z

src/operator/image/image_random-inl.h

+    const int batch_size = inputs[0].shape_[0];
+    const int length = inputs[0].shape_[1] * inputs[0].shape_[2];
+    const int channel = inputs[0].shape_[3];
+    const int step = channel * length;


How large can this value be in practice?

Good point, IMO make step unsigned long long int would be enough?

apeforest · 2018-12-12T17:53:45Z

src/operator/image/image_random-inl.h

+    ToTensorImpl(inputs, outputs, length, channel);
+  } else if (inputs[0].ndim() == 4) {
+    // 4D input batch of images
+    const int batch_size = inputs[0].shape_[0];


How large can this value be in practice?

apeforest · 2018-12-12T17:54:18Z

src/operator/image/image_random-inl.h

+  if (shp.ndim() == 3) {
+    SHAPE_ASSIGN_CHECK(*out_attrs, 0, TShape({shp[2], shp[0], shp[1]}));
+  } else if (shp.ndim() == 4) {
+    SHAPE_ASSIGN_CHECK(*out_attrs, 0, TShape({shp[0], shp[3], shp[1], shp[2]}));


Can shp[0] be zero?

sandeep-krishnamurthy · 2019-01-08T21:24:52Z

@stu1130 @apeforest - I have addressed your comments. I have raised a separate PR - #13802 only for Normalize. It also includes GPU support.

Closing this PR. I request you to review other PR. Thanks.

Make to_tensor to accept 1 or batch of images

a6a43fd

sandeep-krishnamurthy requested a review from anirudh2290 as a code owner December 11, 2018 08:59

sandeep-krishnamurthy changed the title ~~Make to_tensor to accept 1 or batch of images~~ Make to_tensor to accept 3D or 4D tensor inputs Dec 11, 2018

make normalize transformation operator to take 3D or 4D input

b3c255d

sandeep-krishnamurthy changed the title ~~Make to_tensor to accept 3D or 4D tensor inputs~~ Make to_tensor and normalize to accept 3D or 4D tensor inputs Dec 11, 2018

Add API documentation about 3D or 4D input to ToTensor and Normalize …

1ce9a73

…transforms

sandeep-krishnamurthy requested a review from szha as a code owner December 11, 2018 20:41

stu1130 reviewed Dec 11, 2018

View reviewed changes

Address review comments

270ce27

marcoabreu added Data-loading Gluon Operator labels Dec 12, 2018

apeforest reviewed Dec 12, 2018

View reviewed changes

sandeep-krishnamurthy mentioned this pull request Jan 8, 2019

Image normalize operator - GPU support, 3D/4D inputs #13802

Merged

4 tasks

sandeep-krishnamurthy closed this Jan 8, 2019

sandeep-krishnamurthy mentioned this pull request Jan 10, 2019

Image ToTensor operator - GPU support, 3D/4D inputs #13837

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make to_tensor and normalize to accept 3D or 4D tensor inputs #13614

Make to_tensor and normalize to accept 3D or 4D tensor inputs #13614

sandeep-krishnamurthy commented Dec 11, 2018 •

edited

Loading

stu1130 left a comment

stu1130 Dec 11, 2018

stu1130 Dec 11, 2018

sandeep-krishnamurthy Dec 11, 2018

stu1130 Dec 11, 2018

stu1130 Dec 11, 2018

sandeep-krishnamurthy Dec 29, 2018

roywei commented Dec 12, 2018

apeforest Dec 12, 2018

apeforest Dec 12, 2018

apeforest Dec 12, 2018

apeforest Dec 12, 2018

sandeep-krishnamurthy Dec 29, 2018

apeforest Dec 12, 2018

apeforest Dec 12, 2018

sandeep-krishnamurthy Dec 29, 2018

apeforest Dec 12, 2018

apeforest Dec 12, 2018

apeforest Dec 12, 2018

stu1130 Dec 12, 2018

apeforest Dec 12, 2018

apeforest Dec 12, 2018

sandeep-krishnamurthy commented Jan 8, 2019

Make to_tensor and normalize to accept 3D or 4D tensor inputs #13614

Make to_tensor and normalize to accept 3D or 4D tensor inputs #13614

Conversation

sandeep-krishnamurthy commented Dec 11, 2018 • edited Loading

Description

Checklist

Essentials

Changes

TODO

stu1130 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roywei commented Dec 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sandeep-krishnamurthy commented Jan 8, 2019

sandeep-krishnamurthy commented Dec 11, 2018 •

edited

Loading