New course! Every coder should learn Generative AI!
Try a free lesson+ 4
Why does relu train faster than sigmoid?
2 Answers
+ 2
Efficiency:Ā ReLu is fasterĀ to computeĀ thanĀ theĀ sigmoidĀ function, and its derivativeĀ is fasterĀ to compute. This makes a significant difference toĀ trainingĀ and inference time for neural networks: only a constant factor, but constantsĀ canĀ matter.
0
SITHU Nyein does it also have to do with the fact that relu has less noise (deactivates neurons below zero completely, unlike sigmoid)?