add new API: paddle.clone;Tensor.element_size;nn.utils.parameters_to_vector #38020

zhwesky2010 · 2021-12-09T13:09:29Z

PR types

New features

PR changes

APIs

Describe

1.New API: `paddle.clone`

Returns a copy of input Tensor. It will always have a Tensor copy.

Tn addition, the OP provides gradient propagation.

import paddle

x = paddle.ones([2], stop_gradient=False)
clone_x = paddle.clone(x)

y = clone_x**3
y.backward()
print(clone_x.grad)          # [3]
print(x.grad)                     # [3]

2.New API: `paddle.Tensor.element_size`

Returns the size in bytes of an element in the Tensor.

import paddle

x = paddle.static.data(name='x', shape=[3, 2], dtype='bool')
x.element_size() # 1

x = paddle.static.data(name='x', shape=[3, 2], dtype='int16')
x.element_size() # 2

x = paddle.static.data(name='x', shape=[3, 2], dtype='float16')
x.element_size() # 2

x = paddle.static.data(name='x', shape=[3, 2], dtype='float32')
x.element_size() # 4

x = paddle.static.data(name='x', shape=[3, 2], dtype='float64')
x.element_size() # 8

3.New API：`paddle.nn.utils.parameters_to_vector`

Flatten parameters to a 1-D Tensor.

import paddle
linear = paddle.nn.Linear(10, 15)

paddle.nn.utils.parameters_to_vector(linear.parameters())
# 1-D Tensor: [165]

4.New API：`paddle.nn.utils.vector_to_parameters`

Transform a Tensor with 1-D shape to the parameters.

import paddle
linear1 = paddle.nn.Linear(10, 15, paddle.ParamAttr(paddle.nn.initializer.Constant(3.)))

vec = paddle.nn.utils.parameters_to_vector(linear1.parameters())

linear2 = paddle.nn.Linear(10, 15)
# copy weight of linear1 to linear2
paddle.nn.utils.vector_to_parameters(vec, linear2.parameters())  
# weight: Tensor(shape=[10, 15], dtype=float32, place=CUDAPlace(0), stop_gradient=False,
#                 [[3. , ..., 3. ],
#                  [..., ..., ...],
#                  [3. , ..., 3. ]])

paddle-bot-old · 2021-12-09T13:09:46Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

TCChenlong · 2021-12-14T09:27:43Z

python/paddle/tensor/creation.py

+    """
+    Returns a copy of input Tensor. It will always have a Tensor copy. 
+
+    Tn addition, the OP provides gradient propagation.


这两句描述不够清楚，可以在改一下

TCChenlong · 2021-12-22T02:39:08Z

python/paddle/tensor/creation.py

+    """
+    Returns a copy of input Tensor. It will always have a Tensor copy. 
+
+    Tn addition, This function is derivable, so gradients will flow back from the output to input.


Tn addition -> In addition

…vector

TCChenlong

LGTM

Xreki · 2021-12-22T06:01:41Z

python/paddle/nn/utils/transform_parameters.py

+                shape = param.shape
+                numel = reduce(lambda x, y: x * y, shape)
+                end = start + numel
+                slice_data = _C_ops.slice(vec, None, None, 'axes', [0],


能否用1个split代替多个slice？

Done，已去掉多个slice

Xreki · 2021-12-22T06:02:25Z

python/paddle/nn/utils/transform_parameters.py

+                slice_data = _C_ops.slice(vec, None, None, 'axes', [0],
+                                          'infer_flags', [1], 'starts',
+                                          [start], 'ends', [end])
+                _C_ops.reshape2_(slice_data, None, 'shape', shape)


为什么这里需要显式使用inplace接口呢？

core.ops.reshape2_是inplace机制，core.ops.reshape2是view机制。使用view机制的OP是写死的，目前有4个：

inplace机制下：输入、输出是同一个var。
view机制下：需要创建一个输出var，但输出var会和输入var是share_buffer_with的，所以两者都没有成本。
这里指定用inplace版本的，写法会简单些，不需要创建一个临时输出var

Xreki · 2021-12-22T06:04:47Z

python/paddle/nn/utils/transform_parameters.py

+                helper.append_op(
+                    type='assign',
+                    inputs={'X': slice_data},
+                    outputs={'Out': param})


vector_to_parameters可能会比较慢

已去掉assign，改成最简化的写法：

parameters_to_vector：使用1个concat OP + 完全inplace的reshape_ OP
vector_to_parameters：使用1个split OP + 完全inplace的reshape_ OP

目前全部使用inplace版的reshape_ OP，每个API实质上只有1个concat或split OP的耗时，性能可以得到保证了

TCChenlong

LGTM

luotao1

LGTM for CMakeLists.txt

zhwesky2010 force-pushed the add_clone_elementsize branch from 89a8d71 to 4b3da4f Compare December 10, 2021 05:13

zhwesky2010 force-pushed the add_clone_elementsize branch from 4b3da4f to e4ce605 Compare December 10, 2021 08:22

zhwesky2010 force-pushed the add_clone_elementsize branch from b7109a2 to e813185 Compare December 13, 2021 08:21

zhwesky2010 force-pushed the add_clone_elementsize branch from 26dd7d9 to 0a9e5c5 Compare December 13, 2021 12:38

zhwesky2010 force-pushed the add_clone_elementsize branch from 0a9e5c5 to b0d8d43 Compare December 13, 2021 15:04

zhwesky2010 force-pushed the add_clone_elementsize branch from b0d8d43 to 0d99a96 Compare December 14, 2021 03:28

zhwesky2010 force-pushed the add_clone_elementsize branch from 0d99a96 to 572fcf6 Compare December 14, 2021 06:58

TCChenlong reviewed Dec 14, 2021

View reviewed changes

zhwesky2010 requested a review from Xreki December 15, 2021 03:00

zhwesky2010 force-pushed the add_clone_elementsize branch from 572fcf6 to cd894ae Compare December 20, 2021 06:45

zhwesky2010 force-pushed the add_clone_elementsize branch from cd894ae to 7582067 Compare December 20, 2021 15:23

zhwesky2010 force-pushed the add_clone_elementsize branch 2 times, most recently from 0d28fde to 92be1b7 Compare December 21, 2021 03:38

TCChenlong reviewed Dec 22, 2021

View reviewed changes

add new API: paddle.clone;Tensor.element_size;nn.utils.parameters_to_…

0d56823

…vector

zhwesky2010 force-pushed the add_clone_elementsize branch from 92be1b7 to 0d56823 Compare December 22, 2021 02:48

TCChenlong previously approved these changes Dec 22, 2021

View reviewed changes

Xreki reviewed Dec 22, 2021

View reviewed changes

zhwesky2010 dismissed TCChenlong’s stale review via d6c6d00 December 22, 2021 09:40

zhwesky2010 requested a review from XiaoguangHu01 December 22, 2021 10:10

fix comment

593f113

zhwesky2010 force-pushed the add_clone_elementsize branch from d6c6d00 to 593f113 Compare December 22, 2021 12:14

zhwesky2010 requested a review from lanxianghit December 22, 2021 12:18

lanxianghit approved these changes Dec 23, 2021

View reviewed changes

TCChenlong approved these changes Dec 23, 2021

View reviewed changes

zhwesky2010 closed this Dec 23, 2021

zhwesky2010 reopened this Dec 23, 2021

luotao1 approved these changes Dec 23, 2021

View reviewed changes

zhwesky2010 merged commit 0eb03ed into PaddlePaddle:develop Dec 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add new API: paddle.clone;Tensor.element_size;nn.utils.parameters_to_vector #38020

add new API: paddle.clone;Tensor.element_size;nn.utils.parameters_to_vector #38020

zhwesky2010 commented Dec 9, 2021 •

edited

Loading

paddle-bot-old bot commented Dec 9, 2021

TCChenlong Dec 14, 2021

zhwesky2010 Dec 21, 2021 •

edited

Loading

TCChenlong Dec 22, 2021

zhwesky2010 Dec 22, 2021

TCChenlong left a comment

Xreki Dec 22, 2021

zhwesky2010 Dec 22, 2021 •

edited

Loading

Xreki Dec 22, 2021

zhwesky2010 Dec 22, 2021 •

edited

Loading

Xreki Dec 22, 2021

zhwesky2010 Dec 22, 2021 •

edited

Loading

TCChenlong left a comment

luotao1 left a comment

add new API: paddle.clone;Tensor.element_size;nn.utils.parameters_to_vector #38020

add new API: paddle.clone;Tensor.element_size;nn.utils.parameters_to_vector #38020

Conversation

zhwesky2010 commented Dec 9, 2021 • edited Loading

PR types

PR changes

Describe

1.New API: paddle.clone

2.New API: paddle.Tensor.element_size

3.New API：paddle.nn.utils.parameters_to_vector

4.New API：paddle.nn.utils.vector_to_parameters

paddle-bot-old bot commented Dec 9, 2021

TCChenlong Dec 14, 2021

Choose a reason for hiding this comment

zhwesky2010 Dec 21, 2021 • edited Loading

Choose a reason for hiding this comment

TCChenlong Dec 22, 2021

Choose a reason for hiding this comment

zhwesky2010 Dec 22, 2021

Choose a reason for hiding this comment

TCChenlong left a comment

Choose a reason for hiding this comment

Xreki Dec 22, 2021

Choose a reason for hiding this comment

zhwesky2010 Dec 22, 2021 • edited Loading

Choose a reason for hiding this comment

Xreki Dec 22, 2021

Choose a reason for hiding this comment

zhwesky2010 Dec 22, 2021 • edited Loading

Choose a reason for hiding this comment

Xreki Dec 22, 2021

Choose a reason for hiding this comment

zhwesky2010 Dec 22, 2021 • edited Loading

Choose a reason for hiding this comment

TCChenlong left a comment

Choose a reason for hiding this comment

luotao1 left a comment

Choose a reason for hiding this comment

zhwesky2010 commented Dec 9, 2021 •

edited

Loading

1.New API: `paddle.clone`

2.New API: `paddle.Tensor.element_size`

3.New API：`paddle.nn.utils.parameters_to_vector`

4.New API：`paddle.nn.utils.vector_to_parameters`

zhwesky2010 Dec 21, 2021 •

edited

Loading

zhwesky2010 Dec 22, 2021 •

edited

Loading

zhwesky2010 Dec 22, 2021 •

edited

Loading

zhwesky2010 Dec 22, 2021 •

edited

Loading