One reality may have magic and the next may not. Two realities that both have magic can still differ in how easy it is to reach.
In a reality with no magic, there is no casting at all. If you want a fireball there, build it the physical way.
But only magic stops. Cosmic power and psychic power are separate channels, so they can still work in a world where magic does not.