Add data transform fn #6953
Add data transform fn #6953
Conversation
… add-data-transform-fn
… add-data-transform-fn
… add-data-transform-fn
paddle/framework/data_transform.cc
Outdated
| data_transform_map = new DataTransformFnMap(); | ||
| } | ||
| return *data_transform_map; | ||
| } |
There was a problem hiding this comment.
Following is a thread-safe singleton:
DataTransformFnMap& DataTransformFnMap::Instance() {
static DataTransformFnMap inst;
return inst;
}
| namespace paddle { | ||
| namespace framework { | ||
|
|
||
| using DataTransformFN = |
There was a problem hiding this comment.
DataTransform should not only use Tensor as its argument type.
How about SelectedRows? LoDTensor? vector<Scope>? It should take Variable as its argument.
paddle/framework/data_transform.h
Outdated
| size_t left_hasher = kernel_type_hasher(kernel_pair.first) << 1; | ||
| size_t right_hasher = kernel_type_hasher(kernel_pair.second); | ||
| std::hash<size_t> hasher; | ||
| return hasher(left_hasher + right_hasher); |
There was a problem hiding this comment.
A little concern on the correctness of hash calculation.
There was a problem hiding this comment.
ok, I will find a better way to do hash
There was a problem hiding this comment.
HashCombine fit here well.
There was a problem hiding this comment.
HashCombine combinate the different hash value without any conflict, as @QiJune shows in previous PR, https://stackoverflow.com/questions/2590677/how-do-i-combine-hash-values-in-c0x
| const Variable& in, Variable* out)>; | ||
| using KernelTypePair = std::pair<OpKernelType, OpKernelType>; | ||
|
|
||
| static void hash_combine(std::size_t& seed, const OpKernelType& t) { |
There was a problem hiding this comment.
Code style is:
template <class T>
inline void HashCombine(const T& v, std::size_t* seed) {
std::hash<T> hasher;
*seed ^= hasher(v) + 0x9e3779b9 + (*seed << 6) + (*seed >> 2);
}
We can fix it later.
fix: #6823