Cosmic rays also don't get deflected by the magnetic field (either photons or very high energy particles). The atmosphere's density shields us from radiation much more than the magnetic field. Also a full half of the sky's radiation is shielded by the ground.