Numerical aspects are increasingly central in machine learning. The size and diversity of data and corresponding machine learning problems require flexible and scalable optimization solutions. We aim at developing sound optimization algorithms tailored to tackle a variety of modern machine learning problems. Our approach puts emphasis on robustness and adaptivity to the underlying problem geometry, which is exploited to derive provably flexible and efficient solutions.