Colored surfaces absorb light at certain wavelengths and reflect it at others. Other solar wavelengths are invisible to us.
Ultraviolet light (<400nm) is full of energy and is responsible for sunburn. Infrared (IR) light (>700nm) is less energetic but comprises a large percentage of the solar energy that actually reaches us. Both Ultraviolet and Infrared light are invisible, and have no effect on color. However, all light, whether visible or invisible, will heat an object that absorbs it.
The more solar energy the object absorbs, the greater the heat build-up. Conversely, the greater the reflectivity of an object, the less it will build up heat sitting under the sun. Two objects can be identical in visible color, yet have very different reflectance characteristics in the Infrared spectrum. The object that reflects IR-light will remain cooler than the object that absorbs it. And because IR-light comprises fully half of sunlight, the IR-reflectivity of an object is even more important than its color when it comes to heat build-up.
In other words, an object doesn’t have to be white to be cool.