Define constants with specific type

[wangkuiyi]

As stated in #3186 (review), we have code snippets like

class Something {
  constexpr T kLOG_THRESHOLD() const { return static_cast<T>(1e-20); }
};

The snippet has two problems -- 1. the naming violates code style, 2. a constant should be define with specific type as it is obvious that 1e-20 cannot be casted into any integral type.

[jacquesqiao]

The code is

class Kernel {
    constexpr T LOG_THRESHOLD() const { return static_cast<T>(1e-20); }

    void compute() {
      for (int i = 0; i < batch_size; ++i) {
        Y_data[i] = -std::log(
            std::max(X_data[i * class_num + label_data[i]], LOG_THRESHOLD()));
      }
    }
}

1. The reason of add a LOG_THRESHOLD is that if X_data[i * class_num + label_data[i]] is two small, the result of log(x) will cause numeric problem.

2. The reason of add a Template parameter T is that std::max(x, y) requires that the type of x and y should be the same.

3, In our case ,T may be float, float16 and float32, and will not be int because crossentropy is the next layer of softmax, and the output of softmax is probability of each label, it should be floatxx.

4, According to google code style of const name, I will rename it to
kLogThreshold

[wangkuiyi]

@jacquesqiao Thanks for the plot and detailed explanation!

It looks to me it is like you are doing log to a small value. I cannot see the full context, but it sounds like the log-sum-exp trick might help: https://www.xarg.org/2016/06/the-log-sum-exp-trick-in-machine-learning/. Hopefully, we can reformulate the problem using this trick so that we don't take log to a small value at all.

Even if the trick doesn't help and we'd just have to compute log to a small value, a template function might not be what we want, because the small-enough-value differs with types -- double and float. Please see the following code snippet:

#include <iostream>
#include <cmath>

int main()
{
    const double kSmall = 1e-60;
    std::cout << "log(kSmall) = " << std::log(kSmall) << "\n";

    const float  kSmallFloat = 1e-60;
    std::cout << "log(kSmallFloat) = " << std::log(kSmallFloat) << "\n";
}

It runs and prints

$ g++ a.cc -o a && ./a
log(kSmall) = -138.155
log(kSmallFloat) = -inf

It looks to me that we can check-and-change the result value from log instead of the input. This would save us from defining a global const value:

#include <iostream>
#include <cmath>
#include <type_traits>

template <typename T>
T tolerable_log(T x) {
  static_assert(std::is_floating_point<T>::value,
                "tolerable_log works only on float, double and double double.");

  T ret = std::log(x);

  if (ret == - INFINITY) {
    const T kApproxNegativeInf = -300;
    return kApproxNegativeInf;
  }

  return ret;
}

int main()
{
    const double kSmall = 1e-60;
    std::cout << "log(kSmall) = " << std::log(kSmall) << "\n";

    const float  kSmallFloat = 1e-60;
    std::cout << "log(kSmall) = " << std::log(kSmallFloat) << "\n";

    std::cout << tolerable_log(kSmall) << "\n";
    std::cout << tolerable_log(kSmallFloat) << "\n";
}

Please be aware that the const value kApproxNegativeInf is defined in the function tolerable_log other than as a global value.

The outputs shows that tolerable_log corrects the -inf returned by std::log:

$ g++ a.cc -o a && ./a
log(kSmall) = -138.155
log(kSmallFloat) = -inf
tolerable_log(kSmall) = -138.155
tolerable_log(kSmallFloat) = -300

[jacquesqiao]

the purpose of check is to handle INFINITY, and there are another situation like:

T a = x/y; // y is very small

so I check the ret value and return a tolerable one with tolerable_value