Peter Chng

Molyneux’s problem

Molyneux’s problem is a thought experiment which poses the following question:

Suppose a person is born blind, but learns through touch to distinguish between shapes like cubes and spheres. If that person suddenly gains the ability to see, would they be able to distinguish those shapes by sight alone?

William Molyneux, who originally posed the question1, thought the answer was “no”, and so did many other philosophers who encountered this question. Essentially, their reasoning was:

  • Visual concepts are distinct from other sensory concepts such as touch.
  • It is only through experience (e.g. learning that an object is a sphere, and associating that with its feel and appearance) that we make the connection between the different sensory inputs of a concept or object.

We implicitly learn these connections as we experience our world, and often take them for granted. But this thought experiment helps us understand what is going on behind the scenes in our brain.

An example of this might be how to explain colors to a blind person: Most of them have to tie the concept of “color” to another sensation which is associated with that color. In a similar vein, understanding what a straight edge or a curved surface “looks” like having never experienced the sense of sight might be impossible, even if you understood what they felt like.

Our brain actually has to “learn” to do a lot of “pre-processing” on the visual stimuli it gets from the eyes. For example, the lens of the eye projects the image onto the retina upside-down, so that upside-down image needs to be translated into a form that is “rightside-up”, so that we can map physical movements properly into the visual space.

There was a famous experiment where George Stratton, a psychologist, created a set of glasses which inverted the image sent to his eyes, so that the image on his retina was now rightside-up. He took to wearing the glasses all day. Initially he was very clumsy moving about, but he reported that within a few days, he was able to move about (mostly) normally, though things didn’t feel 100% “normal”. He also reported that after wearing the “upside-down” goggles for a few days, after taking them off, it took several hours to feel “normal” again. If we take this experiment at face value, it seems to show that this part of the brain (the image “pre-processing”) seems to show some capability for rewiring (or relearning) itself, so that the rest of the brain can continue to work as normal.

Experimental answer to Molyneux?

A study managed to assemble five individuals (subjects) who were born essentially blind, but eligible for procedures that would restore their sight many years after birth. (R. Held et al. (2011). The newly sighted fail to match seen with felt) This was done for the express purpose of getting an experimental answer to Molyneux’s problem.

Each subject was given a series of tests (see the paper for the testing procedure). The results suggested that the answer to Molyneux’s problem was as the philosophers had conjectured. Quoting from the paper:

Newly sighted subjects exhibited excellent performance on the touch-to-touch (T-T) and vision-to-vision (V-V) tasks, but were near chance on the transfer (T-V) task… Our results suggest that the answer to Molyneux’s question is likely negative. The newly sighted subjects did not exhibit an immediate transfer of their tactile shape knowledge to the visual domain. This finding has important implications for bimodal perception. Whatever linkage between vision and touch may pre-exist concomitant expo-sure of both senses, it is insufficient for reconciling the identity of the separate sensory representations.

A curious phenomenon is the funnily-named Bouba/Kiki effect. It asks people whether the (made up) words bouba or kiki would be associated with either a rounded, blob-like shape, or a spiky, star-like shape. It seems that overwhelmingly, people associate bouba with the rounded shape, and kiki with the spiky shape, and the effect appears to be consistent across different cultures and languages. (Ćwiek et al. (2021). The bouba/kiki effect is robust across cultures and writing systems)

In that study, the authors mention limited evidence for this effect in congenitally blind individuals: Some studies showed an effect, while others did not, so the evidence seemed not strong.

A later study (Piller et al. (2023). Visual experience shapes the Bouba-Kiki effect and the size-weight illusion upon sight restoration from congenital blindness) found that in congenitally blind individuals after sight was restored the effect was initially absent, but then developed within a few months.

Molyneux’s problem and AI

If models like LLMs begin to move into the physical world (via robotics), it appears obvious that properly-integrated multimodal training data (whether hand-generated, or synthetically generated), will be important. This can be seen in the architectures of multimodal LLMs like LLaVA, which uses OpenAI’s CLIP visual encoder to translate images into tokens, so that image inputs and text inputs are in the same “space” and can be processed by the same network. CLIP was trained on many text-image pairs. Similar encoders would be needed to effectively process tactile and other sensory inputs.

  1. Other somewhat-related concepts from philosophy are the Blind man and an elephant and Plato’s cave, though those are slightly different and used to illustrate different ideas. ↩︎