Implicit stochastic gradient descent (ISGD) is a variant to standard SGD, whereby we use implicit updates in the main update of the algorithm. This adds numerical stability and robustness to learning rate specification, which is a known problem with the standard method.
Specifically, standard SGD performs updates of the form:
where θ_n is the n-th iterate, γ is the learning rate and f_n' denotes the stochastic gradient at the n-th iteration. This specification is sensitive to γ, and could overshoot if γ is large enough. In Implicit SGD the same update would be of the form:
where the next iterate, θ_n, appears on both sides. This adds stability and robustness. The contrast SGD-ISGD has natural connections to :
The Proximal Robbins-Monro Method, (2019, pdf)
Asymptotic and finite-sample properties of estimators based on stochastic gradients, (Annals of Statistics, 2017, pdf, slides)
Statistical analysis of stochastic gradient methods for generalized linear models, (ICML' 14, pdf, slides)
Scalable estimation strategies based on stochastic approximations: Classical results and new insights, (Statistics and Computing, 2015, pdf)
Stochastic gradient methods for principled estimation with large datasets, (Handbook of Big Data, 2016, eds. Buhlmann et. al., www)
sgd R package: CRAN https://cran.r-project.org/web/packages/sgd/index.html
