Treffer: Hassan1278/DynaLR: DynaLR: Adaptive Learning Rate Optimizers using PID Control

DynaLR — Advanced Learning Rate Optimizers for PyTorch Author: Hassan Al Subaidi Version: 1.0.3 Abstract 📌 DynaLR introduces a principle of adaptive learning rate optimizers using PID control theory. 2.6% accuracy gain over Adam on CNN architectures Faster convergence (3–5% speedup) Architecture-aware performance (excels on CNNs) Four specialized variants for different use cases Benchmark Summary (30 Epochs, 3 Seeds) Hardware: A100 GPU (ResNet18), ve6-1 TPU (SimpleCNN) SimpleCNN on CIFAR-10 DynaLRMemory: 77.35% ± 0.31% (Time: 153.9s, +1.01%) DynaLRenhanced: 77.08% ± 0.24% (154.4s, +0.74%) DynaLRnoMemory: 77.11% ± 0.89% (155.5s, +0.77%) Adam: 76.34% ± 0.33% (159.0s, Baseline) DynaLRAdaptivePID: 71.95% ± 0.76% (152.0s, -4.39%) ResNet18 on CIFAR-10 DynaLRMemory: 87.45% ± 0.85% (271.8s, -2.19%) DynaLRenhanced: 87.71% ± 0.69% (271.7s, -1.93%) DynaLRnoMemory: 87.76% ± 0.56% (273.8s, -1.88%) Adam: 89.64% ± 0.30% (276.6s, Baseline) DynaLRAdaptivePID: 88.71% ± 0.28% (271.5s, -0.93%) SimpleCNN on CIFAR-100 DynaLRMemory: 45.89% ± 0.88% (154.9s, +2.64%) DynaLRenhanced: 45.26% ± 0.40% (156.9s, +2.01%) DynaLRnoMemory: 45.03% ± 0.61% (156.5s, +1.78%) Adam: 43.25% ± 0.76% (164.8s, Baseline) DynaLRAdaptivePID: 35.08% ± 0.13% (156.7s, -8.17%) ResNet18 on CIFAR-100 DynaLRMemory: 64.05% ± 0.72% (272.1s, -0.84%) DynaLRenhanced: 61.42% ± 1.30% (276.4s, -3.47%) DynaLRnoMemory: 63.16% ± 0.51% (277.7s, -1.73%) Adam: 64.89% ± 0.35% (276.7s, Baseline) DynaLRAdaptivePID: 65.04% ± 0.62% (276.4s, +0.15%) Key Findings 🔍 CNN Dominance: Up to +2.64% accuracy over Adam ResNet Specialist: DynaLRAdaptivePID beats Adam on CIFAR-100 Architecture Matters: Memory variant best for CNNs AdaptivePID best for ResNets Speed Advantage: Average 2.5% speedup vs Adam License MIT License (c) 2025 Hassan Al Subaidi Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, ...