On Activation Design for Improving Robustness and Effective Capacity in Neural Networks

On Activation Design for Improving Robustness and Effective Capacity in Neural Networks

Sentiurev M.A. (ISP RAS, Moscow, Russia)
Makarov I.A. (ISP RAS, Moscow, Russia)

Abstract

Activation functions play a critical role in shaping both the robustness and effective capacity of neural networks, yet their impact can vary across architectural settings. This work investigates how activation design and simple training strategies influence model behavior under fixed parameter budgets, implemented in a custom C++ experimental framework. We consider two complementary approaches. First, we study non-monotonic gradient-preserving activation functions combined with an Equivalent Exchange (EqEx) training heuristic for fully connected networks. Second, we introduce a non-monotonic gradient-preserving activation variant aimed at improving training dynamics in convolutional architectures. Empirical evaluation on MNIST shows that EqEx with non-monotonic activations reduces the success rate of the evaluated targeted gradient-based input synthesis procedures by up to 37% for Carlini-Wagner L2 and 29% for the Basic Iterative Method, while maintaining comparable or slightly improved accuracy (up to +0.6%-0.8% accuracy improvement depending on configuration) under equal parameter budgets. On CIFAR-10, the proposed activation demonstrates improved optimization behavior, achieving higher accuracy (70.8%) compared to Leaky-ReLU (68.7%) and symmetric/mirrored activation variants (68.0%), approaching the performance of models with twice the number of channels (71.8%). These results suggest that different activation design principles are effective in different regimes: non-monotonicity combined with structured training improves robustness against the input synthesis in fully connected networks, while preserving gradient flow is more beneficial for convolutional architectures. Overall, activation design provides a lightweight mechanism for reducing the success rate of targeted input synthesis and improving effective capacity without increasing inference cost.

Keywords

activation functions; non-monotonic activations; Equivalent Exchange (EqEx); Immortal ReLU; robustness; input-synthesis resistance; gradient flow; convolutional neural networks F

Edition

Proceedings of the Institute for System Programming, vol. 38, issue 6, part 2, 2026, pp. 147-162

ISSN 2220-6426 (Online), ISSN 2079-8156 (Print).

DOI: 10.15514/ISPRAS-2026-38(6)-30

For citation

Sentiurev M.A., Makarov I.A. On Activation Design for Improving Robustness and Effective Capacity in Neural Networks. Proceedings of the Institute for System Programming, vol. 38, issue 6, part 2, 2026, pp. 147-162 DOI: 10.15514/ISPRAS-2026-38(6)-30.

Full text of the paper in pdf Back to the contents of the volume