We formulate microscopic optical potentials for nucleon-nucleus scattering from chiral two- and three-nucleon forces. The real and imaginary central terms of the optical potentials are obtained from the nucleon self-energy in infinite nuclear matter at a given density and isospin asymmetry, calculated self-consistently to second order in many-body perturbation theory. The real spin-orbit term is extracted from the same chiral potential using an improved density matrix expansion. The density-dependent optical potential is then folded with the nuclear density distributions of Ca40,42,44,48 from which we study proton-nucleus elastic scattering and total reaction cross sections using the reaction code talys. We compare the results of the microscopic calculations to those of phenomenological models and experimental data up to projectile energies of E=180 MeV. While overall satisfactory agreement with the available experimental data is obtained, we find that the elastic scattering and total reaction cross sections can be significantly improved with a weaker imaginary optical potential, particularly for larger projectile energies.