QA with Beau Lotto On seeing yourself see


Neuroscientist and artist Beau Lotto joined the TED Blog for a short Q&A after his 2009 talk from TEDGlobal. He covered some of the fascinating, perception-bending projects he wasn’t able to cover in his talk — an iPhone game that substitutes sound for sight; a new way for composers to experience their music synesthetically — and detailed an ingenious education project that gives children the chance to participate in real science experiments.

I see your studio has created a game called Bing Bong. Tell me about that.

Bing Bong is an iPhone game. It’s a video game in sound. In the game, the player has to catch a ball, but they must do so without seeing it. They must position a paddle to catch the ball, but they can only hear the ball falling, relative to where the paddle is.

Hopefully, Bing Bong is fun in and of itself — all of my projects have to work at the immediate level, whether that be aesthetic or fun — but, more deeply, it’s a game that’s about getting people to experience the process of having an experience, of seeing themselves see. In doing that, people can better understand how intimately tied they are to their environment, and their interaction with their environment — in other words, their ecology.

How does Bing Bong fit in with your wider research?

Bing Bong is part of a larger program of research, which is sensory substitution or sensory augmentation.

When I say to people that the light that falls onto our eyes is meaningless, people find that really hard to believe. You open your eyes, you look around, you see nothing but meaning; you ask yourself, How could it be meaningless? But that’s of course because you’re not seeing the light that falls onto your eye, but your brain’s perception of that light. You’re seeing that light in the context of the millions of years of evolution that our brains have gone through.

But if I translate that light into sound, and you instead hear the visual information, you directly experience the meaninglessness of it.

Then, through interacting with the world through sensory substitution, you experience yourself literally making sense of it. You begin to hear patterns. Those patterns start having a meaning for you. You’re actually an observer of yourself as you do this.

Through sensory substitution, we can create prosthetics for the visually impaired — to make the world navigable. But we can also do things like make music from color, which is an element that I touched on during my TEDTalk.

The brain almost doesn’t seem to care what type of information it receives — it just starts decoding any information it is given. Do I have that right?

The most fundamental thing the brain does is it evolves to evolve. It adapts to adapt. The brain is wonderfully plastic. Now, it’s a plastic of different levels of flexibility — the brain couldn’t function without some stability as well.

The reason why we survive in the world is not because we evolved to see the world as it is; it’s because we evolved to be able to adapt to a changing world — and to continually redefine normality. And that process even exists at the level of evolution. Evolution itself has evolved to have something called evolvability.

In my lab’s research, we not only work on networks of the brain, but also networks of genes, and networks generally. In doing that, we evolve what we call artificial life systems, or artificial life agents.

If you give these artificial life agents an environment, and in the environment there is a problem, the agent that evolves the solution to the problem faster will out-compete the one that evolved it slower. Both evolved the solution, but the one that did it faster was better. The successful agent is actually more evolvable. Its evolvability helped it survive.

There is evidence that organisms have evolved to be evolvable. They have evolved to be adaptable. This quality is something that is relevant to any adaptive network — not merely the brain.

The aim of much of your work seems to be making science and art accessible to everyone. Talk a bit about that.

As far as I’m concerned, science is nothing more than playing games. And in that case, everyone does science. There’s nothing special about science or the people that do it. The only thing that makes scientists different is that they formalize the process; they have access to certain tools that other people don’t have access to.

In the education center we’re trying to build, we want to give people access — and the confidence — to do real scientific experiments. The education center will be fundamentally about discovery. It might be discoveries that no one else has made. Or it might be just discoveries that are relevant to each individual that participates.

What’s a specific example of the sort of discovery you mean?

The Mother program is about enabling musicians to use musical instruments as an interface to a computer. And what happens is the musician can play the instrument, and the music they play gets visualized — not in the way that, say, iTunes visualizes music; the Mother visualizations are far more complex than that.

The significance here is that the musician can develop an intuition about the visual images that will be created by the way they gesture, the way they move their hands, the way they play their instrument. And what can then happen is, the musician can now create music not just according to what it sounds like, but also according to what it looks like. And that itself can feed back and alter the kind of music that the musician plays. They discover new music by being able to experience their playing through senses other than hearing.

You mentioned an education center. Talk a bit about your current work on education.

I’m doing a project with a friend and collaborator, Dave Strudwick. His background is working with kids from disadvantaged backgrounds, whom he calls “excluded” kids.

I came to him with this concept of “seeing yourself see.” I told him we were exploring this concept in the context of architecture and art. I asked, “What might we be able to do with this, in education?” The premise is that education is fundamental to everything. It’s not just school: it’s everything that humans do. That question led to a series of conversations and meetings about creating a framework for education that we call My School. We call it My School because the people in the school have ownership of the school.

My School is all about education that is specific to the individual. The aim of the education program is to create a curriculum in architecture grounded in the idea of seeing yourself see, emphasizing everything that the education system does — what we call the five Cs: compassion, choice, community, creativity and confidence. Seeing yourself see leads to these five Cs.

The education system is about facilitating people to become aware of how they’re shaped by their experiences. The point is that when our kids are our age, they will be dealing with careers 90% of which do not exist now. They’re going into a world that is incredibly dynamic, unpredictable, uncertain. So, really, the program is about celebrating uncertainty, which was the point of my TEDTalk: giving people the agility of thought to adapt and to thrive in that kind of uncertainty, and to create.

Can you give a specific example of the sort of work you’re doing now with kids?

One particular project is taking the bumble bee arena — we’ve had it as an installation in the Hayward Art Gallery and various other places — and taking it to a school, and going through a process of experimentation with the kids where it’s very much them-led: the kids lead the process.

We started getting them to think about questions such as, “If you were asking someone a question, but they couldn’t communicate an answer except through their behavior, how would you find the answer?” We created games and puzzles to help them figure out how they might solve a problem like that.

We showed them that they can do this sort of puzzle not just with people, but also with other creatures, such as bumble bees. We got them thinking about what kinds of questions they might want to ask a bumble bee if they could. In doing so, they had to put themselves in the perspective of a bumble bee: What’s important, what might be interesting to a bumble bee?

The kids came up with a list of questions. They collectively decided on one of those questions. Then they designed a set of experiments to ask that question of a bumble bee using the bee arena.

We installed the arena in an old Norman church next to their school, and I carried out their experiments over the next two weeks. The process is still going on; in total, it’s been about three and a half months. The actual experiment took about two and a half weeks. The experiments took place on Sundays during church services. There was a funeral at one point. The whole community got involved. People from the community would come up. Parents would come by before and after school.

We got the data, and then the kids analyzed the data. Then they began writing up the paper. I would open up my laptop and say, “What do we write?” We’ve now just finished the paper. Everything is in kid-speak. For instance, they wanted to start the paper with “Once upon a time,” which we’ve done. In one of the methods sections, they had to describe the second test for the bumble bees, and they said [ominously] “Bum bum bummmmmmm!” So, that’s in there too.

All the figures are hand drawings in crayon.

We’ll be submitting it for publication by the end of this week. If it’s published, there will be 25 authors, all of whom are 8 years old or younger. If it’s published, it means they should all get master’s degrees, because by definition they’ve all made a unique contribution to science.

It will be interesting to find out whether, in fact, their paper will be published. Being someone who does bee research, I know the findings are unique. But a “real” science paper has references in it. You contextualize the study. You say, “This is what’s been done before, this is what we’ve done, and this is what it now means in the context of what other people have done.” Of course, as this is a paper by 8-year-olds, they don’t have that context. They don’t know the literature.

Will the community publish a paper with genuine results, but without the contextualization? It’s an interesting question. Their introduction to the paper is what led them to do the experiment. Their discussion is what the results of the experiment might mean to them, and what it might mean to the bumble bees. So, the paper might never get published because it’s in kid-speak and it’s not contextualized — but the data is strong.

I wanted to switch gears and circle back to your work on perception. Several TEDTalks feature illusions. How do you respond to artists and scientists who use illusions to show that you can “hack” the human mind?

There’s an artist — I won’t bother naming him — who was up for a prize, and he often used illusions in his work. One critic said his work demonstrated the “fragility” of the human senses. Artists and others often use illusions to demonstrate how our senses are susceptible. But, as I said in my TEDTalk, if our senses were fragile, we wouldn’t be here.

The whole concept of an illusion is predicated on a misconception. The misconception is that we evolved to see the world as it is, and that to see the world differently from “as it is” is an illusion. But the point of my TEDTalk is that we actually can’t see the world as it actually is. We have no direct access to the physical world. All we can ever do is see it the way it was once useful to see.

Illusion is more a state of the world than it is a state of mind. What’s being presented to you is an unusual situation. What you see is what would have been useful, given that situation in the past. That’s significant because artists often use context in order to manipulate what people see. But they often don’t go beyond that.

The far more interesting question is not that “context matters” — not that we see illusions — but why we see them. When you see illusions, you’re entertaining two realities at the same time. You’re seeing one reality (two gray squares look different) but you also know another reality (that the gray squares are, in fact, physically the same).

You’re in the position, at that moment, of actually experiencing yourself having an experience.

Do you find that your work is in conflict with philosophy on perception or consciousness? Are angry philosophy students knocking on your door?

Well, first of all, philosophy isn’t going to answer many of these questions about perception.

I have a very good friend, Tom Polger, who is a philosopher, and he and I have written a paper on why we see four colors. He was a great person to talk to — a very interesting person, generally. The debates were complimentary.

If anything, my research is about taking these concepts in philosophy and trying to ground them. One idea is as good as any other — but if you can ground them in true, real experiences, and then share these experiences with the public in a way that is intuitive.

So no, my work isn’t in conflict with philosophy in that way. I don’t get very many angry philosophy students.

What’s the takeaway from your work, as a whole — the big lesson you want your art and science to teach?

The lesson is that the brain evolved to continually re-define normality, and that understanding that creates the capacity for compassion and creativity. What’s true at the simplest level, seeing lightness — it doesn’t get any simpler than seeing lightness; even jellyfish see lightness — has got to be true all the way up.

Find more Q&As on the TED Blog:
+ Oliver Sacks on neurological curiosities
+ Bruce Bueno de Mesquita on Iran’s nuclear program
+ Garik Israelian on the secrets of spectroscopy