We see everything in terms of light and shade. If all colour were removed we would still see the scene because each item in the scene is a differing shade of grey.
Think about that - each item! Each leaf on each tree of a forest, each blade of grass in a field. As we pass in a vehicle glancing out the window our eye takes in all this detail and our brain processes it, imagine the info that gets filtered out!
We see the distance (depth of field) also because of light and shadow. Even in shadow we see the scene because of the differing amounts of light there.
Although our appreciation of a scene may be enhanced by colour the underlying structure is created by light and shade.
50 shades of grey? No our brain is I'm sure unlimited in its ability to recognise the shades.