Timeline for fast and stable x * tanh(log1pexp(x)) computation
Current License: CC BY-SA 4.0
3 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 2, 2020 at 11:05 | comment | added | Yashas | I didn't mention in the question but I was looking for a good approximation to accelerate neural network inference. I need to run trained models (which are trained with the exact form of mish) and I don't have control over the training. So I would need something that closely resembles and is really fast. | |
| May 2, 2020 at 11:01 | comment | added | Yashas | You are quite right about your impression. If you are interested, this function was introduced in this paper. A closely related paper is for swish. It was a search for rectifier like functions with a specific set of characteristics that tends to perform really well in deep learning tasks. However, it is required that any approximation approximates mish function relly well because the neural networks are trained to use mish and any arbitrary approximation that shows similar behaviour might not work as a replacement. | |
| May 2, 2020 at 10:27 | history | answered | gnasher729 | CC BY-SA 4.0 |