I recently had a discussion about creativity with a colleague. We were discussing music and how creative many bands and groups are. At the end of our conversation, my friend told me, half-sarcastic-half-serious, how much more creative the people in the music industry are than him and that he just cannot find good ideas in his area of research even though he tried so hard for such a long time. I was a bit surprised because I thought of him as someone very creative. However, it is not uncommon to hear scientists lament about their lack of creativity compared to academic superstars. I think about creativity in academia is a bit distorted and a straight view can help to feel less bad about one’s own creativity.
This blog post is part of a series of blog posts about scientific thinking in deep learning, natural language processing and science in general. I am currently on vacation in China, and I wanted to relax a bit by writing down some reflective blog posts which capture the thoughts that were lingering in my mind for weeks or months.
Are Theoretical Physicists Creative?
I think the paradox that very creative people think they are not creative is best demonstrated by looking at theoretical physicists and compare them to children. In psychological research, it is well known that children score much better on many tasks of divergent thinking than adults do: They do not think about the limitations of an object so that a brick which is used for building buildings is suddenly a tool for weight training, or a door stopper, or a paperweight, and so forth. If you ask people to build towers of spaghetti and marshmallows children do better than adults because they are not limited by what they think a structure of a tower should look like. But all of this is mere idea generation. Is this really creativity?
There is another famous case of similar creativity among physicists which might shine a light on what the boundary between idea generation and creativity in research is: The undergrad theory of everything. It is a common problem for academics in physics to be tortured by undergrads who just invented “a new theory of physics which can unify gravity and quantum mechanics”. The problem here is that the undergrads do not yet have the proper knowledge to understand the intricate relationships among equations to understand what is permissible and what is not. They see the brick as a door stopper, when in fact a brick is used for building buildings and paving walkways. An important part of creativity is to understand what are bad ideas — some physics undergrads think it is just about idea generation. Do not get me wrong, idea generation is important, but it is not the most important part of creativity in academia.
This can go to an extreme if you work in theoretical physics and other fields where ideas are severely constrained by proper thought. There are so many bad ideas and so few good ideas that nobody really is coming up with anything good anymore. However, it would be ludicrous to say that people like Edward Witten are not creative because he did not come up with any good ideas since string theory. Similarly, Albert Einstein labored for decades trying to unify gravity and quantum mechanics only to come up with nothing. Bertrand Russel would often take a sheet of paper in the morning and work on a logical problem and write down whenever he found a useful thought. Most often the paper was still blank in the evening. So if you see creativity as idea generation, Albeit Einstein and others should be seen as failures compared to the children that churn out ideas. This demonstrates that the view of creativity as idea generation is problematic.
One thing that has to be understood when thinking about creativity is that some fields of thought are highly constrained in terms of which ideas are valid. To come by a good idea is a very lengthy and labor-intensive process. Other fields, like music, are very free in their expression and you can take any two ideas which do not seem to be related at all, mash them together, and with a little bit of work you can make it sound nice. I am exaggerating, but you get the idea.
Some fields, like machine translations, are now more and more constrained and good ideas need a team of people equipped with large computational resources that collaborate effectively for a long time come up with, and verify an idea which will yield a tiny improvement. One can expect the constrains on ideas increase exponentially with time in any given sub-field — just like it did in experimental physics. However, while finding valid ideas is becoming exponentially more difficult these fields also spawn new sub-fields as offspring. In these new fields, it will be very easy to come up with new ideas since — similarly to the music industry — anything is valid. As the field progresses the idea space becomes more and more constraint and finding valid ideas is much more important than generating just any idea. If you work in an area which is very constrained, you should have more compassion with yourself. Creativity is not just about generating some imaginative ideas — it is more about finding strange ideas which are still valid.
“Not Coming Up with Good Ideas” is Essential for Creativity
Expertise is important and a requirement for creativity. You need to be able to understand what are valid ideas and which are not. The next step is to loosen up the boundaries between ideas that may seem unconnected at first glance. Psychological research says, that once one has one of these strange ideas it is important to hammer on it over and over to exhaustion. The idea will reshape itself from one form to the next and eventually, you will probably fail to come up with something reasonable that works. Science says, that this is normal and the further insights are made unconsciously. After you give up an idea, your unconscious mind is still in the process of piecing together the puzzle and you might arrive at something useful over time. With the next puzzle piece put into place by your unconscious mind, you might be able to make some progress on an idea which might lead to a working valid idea.
Many researchers fail in the creative process because they do not understand it well. They feel like failures if their ideas fail. But the process of hammering on ideas and not making any progress is the first part of creativity. Only if you know all the ways that do not work can you come up with the solutions that nobody else is seeing. The second step is often abandoning the idea for some time. Some researchers feel that if an idea did not work out and you abandon the idea you also failed and it is a sign of not having creativity. But this step can be a critical element of creativity. It is important to have phases in which you do not think about an idea so your unconscious mind can make the connections that your conscious mind cannot see. The next step is to pick up a failed idea and try again. The unconscious insights are revealed in this way and you might quickly have a way to get an idea to work.
Another problem with the creative process is that researchers often work on a single idea. Instead, it is much more effective to work on many ideas. One idea for you to work on actively, while the other ideas are in the back of your mind and provide enough material for your unconscious mind to churn on. These ideas do not need to be totally different from each other, just different enough to not bother your conscious mind while you work on another idea.
I think to have a sane creative process, it is essential to acknowledge and even embrace this long-winded exhausting struggle with multiple rounds of failure as an essential part of creativity.
Researchers are often very harsh critics of themselves in terms of creativity. They do not come up with good ideas or with too few ideas or their ideas do not work out. But this does not mean that you are not creative. Some fields of research are very constraint in what ideas are valid and it is expected that the raw quantity of ideas in these fields is low. Furthermore, making no progress and abandoning an idea to work on something else are essential parts of creativity and should be celebrated and embraced. The next time you fail to make progress and think about abandoning an idea you should give yourself a pat on the back — you just reached the first milestone to come up with a great idea!
Andy Yang says
Hey, Tim. This is Andy.
I’m an NLP research engineer in an NLP related start-up company in Beijing.
I have been read your posts for a long time.
This post is again a great post.
When I read it, it let me come up with the time during my master degree.
As you wrote, at first I tried several ideas and failed.
But later, when I tried on another direction, some failed ideas I tried before come to my mind. Thanks to that idea, I finished my thesis.
I am so glad to hear that you are on vacation in China.
If you pass Beijing or stay in Beijing now, I would like to invite you to have a drink or a meal if you can.
Tim Dettmers says
Thank you, Andy, for your kind comment. I am back in Seattle right now, but I might visit China again the next year. I will let you know if I do!
Look forward to your next visit!
David Morgan says
So, after seeing you rank seriousness of interest in deep learning by hardware possessed (“I want to try deep learning, but I am not serious about it”), I wanted to see if you had gone beyond using VGG ‘s architectures. So, I looked to articles you have published since to see what else you’ve done and ran across this article interestingly enough. I think that most innovation (creativity) in this field will be qualitative and not quantitative. You might find the creativity coming from researchers who are more playful than serious, and who are working with less than the greatest hardware.
Tim Dettmers says
I agree. It is true what you say. Quality of ideas will always more important than quantity. However, there is also strong evidence that the most creative people are those that have the most ideas. They try again and again until things work. When things work, these creative people are usually associated with the successful idea rather than the plentitude of unsuccessful ones. If you look at inventors this is very common. Scientists also fail very often but usually, their failed ideas are not known or published. All of this might make it seem that quality is more important than it really is.
I think one major reason for the innovation in deep learning is that you can test more ideas quickly if you have good hardware. So, if you are playful abd you have great hardware, you might be even more creative in that you can try more ideas in a shorter time. However, I would also say that “too much hardware” can stifle creativity. The application of lots of hardware might or might not indicate the degree of creativity of an idea, but if using more hardware stifles creativity probably is not a sign of lack of creativity but a sign of lacking other characteristics such as foresight and understanding what research is important, needed, and replicable by other researchers (that lack the hardware that oneself has). A stereotypical example of using too much hardware unwisely is when you use a lot of hardware for the sake of using a lot of hardware, for example, NVIDIA’s super-large transformer would fall into this category. However, onr could imagine that lots of compute is used for a creative project. For example, the early Google brain project to do unsupervised learning on a large collection of youtube videos could be seen as a creative way to gain deeper insights into how features relate to data.