Most of the people alive now are incredibly fortunate compared to almost all of their ancestors. According to the World Bank, less than 10 per cent of the world’s population is now living in absolute poverty, at the edge of bare subsistence. They estimate that in 1800 over 80 per cent of the world’s population were in that state.

Moreover, that was how it had been for the whole of human history since the advent of agriculture. Poverty was the norm, the default state of affairs and this did not change for centuries. The living conditions and everyday experience of an Italian peasant in 1800 were not significantly different from those of his Roman ancestors two thousand years earlier (his diet was different because of New World crops but his standard of living was much the same).

In addition, our world is not only vastly richer than that of our ancestors, it has changed profoundly in other ways as well. Slavery, a universal institution for all of history, has vanished. We now live in a world where more than half of all people live in cities: previously the figure had seldom if ever exceeded 10 per cent. The range of opportunities and possibilities for human beings is incomparably wider and more varied than it was for our ancestors, even the elite. The big question for all of us is why we are so fortunate. How and why did the world change so dramatically, so that we now live in a world so different from that of our ancestors?

The key to understanding almost all of human history is that our ancestors (who were just as smart as we are) lived in a world of limits. It was a Malthusian place in which output grew only through either windfall gains or by increasing inputs (time worked, number of people, area of land cultivated and so on). This meant there was no increase in the productivity of the combined factors of production and so living standards rose very slowly, if at all.

And because all productive activity was subject to diminishing marginal returns human population and consumption typically rose to the maximum that the productive system could support (hence most people living on the edge of subsistence). Periodically complex societies would collapse, typically over a hundred year period, and revert to a much simpler way of life with lower populations, less urbanisation, and less trade and exchange.

The only way to escape from this Malthusian cage was through sustained innovation. There were indeed times when innovations happened and you began to see the first signs of modern economic growth. However, these episodes did not last. The innovation was not sustained and any improvement proved to be only a step change – there was no sustained upward trend in productivity – and eventually life regressed to the norm.

That happened for two reasons. First, the elites or ruling classes had strong reasons to fear innovation, because it threatened their position, and so they typically checked it or discouraged it. Even more importantly, all societies had a network or system of institutions, rules and practices (the so-called ‘moral economy’) that protected people against misfortune and made society more robust but at the same time systematically discouraged innovation.

Then, about 250 years ago, in the last third of the 18th century in parts of North Western Europe, something changed.

Another episode of innovation began and this time it was sustained. After about 1850 the pace of innovation and economic growth both accelerated sharply and this has remained the case, despite world wars and a great depression. This obviously raises a more specific question that we must answer if we are to understand how the world we live in came about and what it is that it depends on. What changed? Why was this  time different from earlier episodes (such as, for example, the one in China in the 12th and 13th centuries)?

Over the last few decades many scholars from several disciplines have explored this question. The key discipline however is history because this is ultimately a historical question, even if historians make use of the insights provided by economists, anthropologists, and sociologists to name just three. Not surprisingly there are many answers.

One, which is popular on the left broadly defined, explains it in terms of the profits of slavery and exploitation by Europeans of the abundant resources of the Americas, after diseases brought by invading Europeans had wiped out most of the indigenous populations. However the empirical facts simply do not support this argument. Until well into the 19th century, trade with Asia was several times more economically valuable than that across the Atlantic with the Americas. No more than 10 per cent of the capital invested in early industry came from the slave trade (trade with India was more important as a source and that in turn was less than saved profits from domestic agriculture). Most of the silver that came from Mexico and Peru merely passed through Europe – the bulk of it ended up in China.

There is another set of explanations that are popular with the right, particularly those of a free market persuasion. According to these, the key was that European civilisation had qualities and features that set it apart from other civilisations such as the Chinese or Islamic and in particular that it had a number of institutions that made it much more dynamic and innovative.

Douglass North argues that these arose in Britain with the Glorious Revolution of 1688. The problem here is that most of the institutions he mentions had been around for hundreds of years and so most of the other scholars who take this line think that Europe became distinct during the middle ages. This set of theories also do not work however. If Europe became institutionally more innovative by the 12th century, why did it take 600 years to have an effect?

The real problem though is that the institutions and activities that people focus on were not found only in Europe but throughout the major civilisations of Eurasia so they cannot be the reason why the burst of innovation that happened in eighteenth century Europe went on to be sustained and to transform both European societies and the world.

So the real question is why the barriers that had suffocated sustained innovation gave way at that time and in that place. In my new book, The Wealth Explosion, I argue that the critical factor was a change in the incentives facing European ruling classes. Because of the way the military revolution of the 16th century had worked out in Europe, as compared to other parts of the world, they faced intense competition which gave them an incentive to encourage innovation and sweep away barriers to it.

Faced with a global Malthusian crunch at the end of the 18th century, they did just that. Simultaneously powerful social interests that gained from innovation were able to win the intense political conflicts that this caused and so the process was not aborted but sustained.

However all this leaves us with a greater challenge and difficult questions.

Will innovation continue or, as many argue, is it actually slowing down? Could there be a political and social movement to stop innovation or even reverse it (as happened in China in the 14th and 15th centuries)? Could it be that we have only temporarily escaped the bars of the Malthusian cage and they are about to close around us once more?

It could be that the transformative dynamism of modernity will continue with even more radical results but alternatively this could be only the longest episode and our descendants may live the way our ancestors did. Basically, it is up to us to determine which of these futures comes about.