My latest book, The Poetry and Music of Science (2019), starts with my experiences of visiting schools and working with sixth-form pupils in general-studies classes. These students, aged 17-18, would tell me that they just didn’t see in science any room for their own imagination or creativity. Not just on one occasion but repeatedly I heard this from young people bright enough to have succeeded at any subject to which they set their minds.

Yet it doesn’t take an Albert Einstein to observe that, without the essential first step, without a creative reimagining of nature, a conceiving of hypotheses for what might be going on behind the perceived surface of phenomena, there can be no science at all. Einstein did of course have something to say on the matter. As he told an interviewer in 1929:

I am enough of an artist to draw freely upon my imagination. Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world.

Every scientist knows this, but for two centuries they have fallen mute about it, preferring instead a safer narrative about the ‘empirical method’ or ‘the logic of scientific discovery’. Science education favors the presentation of results, and a focus on knowledge, rather than the human stories of wonder, imagination, failed ideas and those glorious and uninvited moments of illumination that thread through the lives of all who actually do science. Our media mouths the same message – I will never forget the BBC documentary on computer science in which the presenter assured viewers, face to camera, that there is no room for imagination in science. No wonder my young colleagues had become disillusioned.

If scientists are somewhat shy about their experiences of imagination, then the artists, writers and composers I spoke to needed the same patience (and similarly the occasional drink) to draw them out about their repeated need to experiment. Scraping the paint from the canvas, redrafting the novel for the 10th time, rescoring the thematic musical material is – as every artist knows – the consequence of the material constraints that creativity meets unanticipated. The artist, too, makes hypotheses about how her material, words or sounds will achieve the goal in mind, however indistinctly conceived. The historically contemporaneous birth of the English novel and of the experimental method in science turns out to be no coincidence. Without making the naive claim that art and science are in any sense ‘doing the same thing’, the narrative similarities in the experience of those who work with them are remarkable. They need digging out because they become obscured by scientists shy of talking about imagination and artists about experiment.

The project of listening to anyone who creates – be it with music or mathematics, oil paint or quantum theory, and the creative power of the constraints they encounter – itself became my book project. Yet in a strange obedience to the pattern of its material, the originally imagined plot of The Poetry and Music of Science refused to play out. Juxtaposed catalogues of creation-stories in science and art, followed by an extended ‘contrast and compare’ essay, increasingly failed to do justice to the material. Historical and contemporary sources were telling a very different story about creative imagination, one that didn’t divide across the worn-out lines of ‘The Two Cultures’. Instead, a pattern of three ‘modes’ of creative expression seemed more faithful.

The first mode of visual imagination is, of course, the chief source for the artist, but the same is true for many scientists, from molecular biologists to astrophysicists. Astronomy is the provider of the original projective perspective. If the observer of a painting is asked to recreate a three-dimensional world from a representation or impression on a two-dimensional canvas, then the task of ‘seeing’ the Universe from the picture that we call the sky bears clear structural resemblance.

A second mode is textual and linguistic. The entanglement between science and the written word in prose or poetry might possess a principle knot at the birth of the novel, as we have already noted, but its story is a much longer one. It also has an ‘alternative history’, envisioned by the poet William Wordsworth in his preface to the Lyrical Ballads (1798) – and surely Johann Wolfgang von Goethe and Alexander von Humboldt before him – in which:

The remotest discoveries of the Chemist, the Botanist, or Mineralogist, will be as proper objects of the Poet’s art as any upon which it can be employed, if the time should ever come when these things shall be familiar to us …

With notable exceptions (such as R S Thomas and occasionally W B Yeats in poetry, and the ever-present fluttering trespass of Vladimir Nabokov’s beloved butterflies from his scientific work into his novels), this early Romantic vision has sadly yet to be fulfilled, and is surely frustrated by the very desiccated presentation of science with which we began.

Imagination’s third mode appears as both pictures and words fade away. For there, when we might have expected a creative vacuum, we find instead the wonderful and mysterious abstractions of music and of mathematics. This shared space is surely why these two have something in common – it is surely not their superficial sharing in numerical structure that links melody and harmony with mathematical structure, but their representational forms in entire universes of our mental making.

When a journey has taken one to as reflective a place as this, it is but a short step to recognise the need for interdisciplinary thinking to make sense of it all. The anthropology and cognitive neuroscience of creativity is fascinating, the one taking us to the stone tools of our distant ancestors at the dawn of humanity, the other to the delicate balance between the analytic left hemisphere of our brains and the integrative right. The philosophical tradition is equally rich, discovering, for example Emmanuel Levinas’s suspicion of the visual mode for its implied distancing, preferring the vocal or auditory for its immersion of subject in object. The phenomenologist tradition from Martin Heidegger and Maurice Merleau-Ponty to Hannah Arendt speaks of a relational mode between the human and nonhuman that deploys both art and science to describe nature as if it were the product of human imagination. As the literary critic George Steiner wrote in his Real Presences (1989):

Only art can go some way towards making accessible, towards waking into some measure of communicability, the sheer inhuman otherness of matter …

I could say precisely the same of science, so how might a richer appreciation of the service provided by the creative imagination in science be developed in a practical way? There are consequences for both practising scientists themselves, and for the wider community.

Reflecting on my own formation as a professional physicist, I cannot recall a single hour spent during my doctoral or postdoctoral training on even as instrumental an aspect of creativity as the discussion of working practices or lifestyles that might enhance the vital creative flow of scientific ideas. Yet there is much to be said: the regular engagement with the visual and auditory, the alternation of sharp mental focus and integrative defocusing, the allowance for fallow periods when working on a problem ­– all these are worth talking through early in a scientific career.

More widely, the contemplative good of lay science, of engagement with high-quality scientific writing, including the poetic ‘notable exceptions’ – John Carey’s The Faber Book of Science is a good start (1995) – recognising that science holds as deep a structural place in human culture as art does, will only enrich and enable. By exploring other avenues into science than the formally educative – its history and philosophy, its deep ideas simply put, and a rediscovery of the joy brought by acute observations of nature – more people might discover that the notion that ‘science is not for me’, too often acquired early in life, is simply a cruel deception.

‘The Poetry and Music of Science’ by Tom McLeish is published via Oxford University Press.