Python packages layer and operator

[wangkuiyi]

In a DL system, we can compose one or more fine grained operators into a coarse grained one. For example, the FC layer can be composed of a multiplication operator and an add operator.

Here is a question -- what is the difference between layer and operator.

In general, operators are those very fine grained operations, e.g., mul and add. In the implementation, we can write them as C++ functions:

template <typename T> T add(T x, T y) { return x + y; }
template <typename T> T mul(T x, T y) { return x * y; }

Then we can wrap them into operators which are C++ classes and can be created from Python bindings by name. A C macro can do this. For example, the following macro invocation

#define MAKE_FUNCTION_OPERATOR(mul);

generates

class mulOp : public OperatorBase {...};
REGISTER_OP(mulOp, "mul");

so that in Python we can create operator mul by:

X1 = Var()
X2 = Var()
Y = Var()
paddle.cpp.create_operator("mul", input=[X1, X2], output=Y)

Also, at the same time, we can compose a coarse level C++ operator class by composing functions mul and add:

class FCOp : public OperatorBase {
 public:
  void Run(...) {
    add(mul(Input("X"), Input("W")), Input("b");
  }
};
REGISTER_OP(FCOp, "fc");

We need to support such composition in Python as well. To do so, we need a higher level Python wrapping of operator creation than paddle.cpp.create_operator. This higher level operator API should be compatible with the layer API.

Let's explain using an example. Suppose that we are going to compose the FC using mul and add in Python, we'd like to have Python functions mul and add defined in module operator:

def mul(X1, X2):
    O = Var
    paddle.cpp.create_operator("mul", input={X1, Y1], output=O)
    return O

def add(X1, X2):
    O = Var
    paddle.cpp.create_operator("add", input={X1, X2], output=O)
    return O

so that we can define

def layer.fc(X):
    W = Var()
    b = Var()
    return operator.add(operator.mul(X, W), b)

We'd like to have Python bindings to operators in package paddle.operator, and Python compositions of operators in package paddle.layer. This is how we differentiate layer and operators in PaddlePaddle.

[wangkuiyi]

To make this a PR of design doc.