James Watt is a former British Ambassador to Lebanon, Jordan and Egypt. He now works in London as a partner in the political risk consultancy the Ambassador Partnership LLP.

The territorial defeat of IS in Iraq and now Syria ought to be an inflection point in the fortunes of the Middle East. For decades, the scale and frequency of conflicts in the region has grown, devastating ancient communities and historic countries, and driving tens of millions from their homes. As if the strains of rapid urbanisation, joblessness, and environmental degradation were not acute enough without the added trauma of war.

No one in the region, or beyond it, would disagree with this stark truth. But to observe regional politics today is to see a depressing absence of optimism that its logic – the systematic, committed search for enduring regional peace – is going to be followed.

There is no doubt of the difficulty. There are tangled layers of mistrust, threats, false steps, and unfulfilled promises. There are active enmities, disparities of power, and far too many arms, all mixed with the exaggerated rhetoric of existential danger. External factors are changing, with the relative waning of US and European influence, the return of Russia as a power-broker in Syria, and China now intending to use its weight.

There are changes, too, in leadership within the region. The strong Iraq of Saddam Hussein has ceded first place to a newly assertive Saudi Arabia. But there are continuities as well: the Emirates have long played a quietly moderating role, both in foreign policy and in respecting religious freedom. Egypt, absent during the traumatic years of 2011-13, is back as a stabilising force. Syria seems set to survive as a united country. Libya, too, seems to be returning to stability.

Just as Europe rebuilt its security architecture after the Second World War, and renewed it at the end of the Cold War, so the Middle East needs to tackle its own need. Outsiders can offer advice, but the key changes will come from decisions taken by regional countries themselves. Those regional countries include Israel, Iran and Turkey, as well as the Arab States.

What essential decisions are required?

First, to curtail the role of ideology. Religious beliefs should be respected, but de-politicised. The region has suffered too much from the weaponisation of bigotry: the export of Iranian revolutionary doctrines in the name of Shi’a Islam, and the propagation of Wahhabi fundamentalism leading directly to radicalisation and Al Qaeda/IS. Leading the regional resistance to ideological extremism have been the smaller Gulf States such as the UAE. The new Saudi policy under Mohamed bin Salman has taken the same course. Egypt has reaffirmed its historic moderation after the Muslim Brotherhood interlude. But Turkey, until recently a role model, is now wracked by witch-hunts claiming an ideological justification. And Israel continues to struggle with the highly ideological agenda of the settler movement.

Second, to revisit entrenched positions and question them. Is Israel an existential threat to Iran? Is Iran an existential threat to Israel? Is confrontation between Iran and its Arab Gulf neighbours inevitable? I do not believe so. But it will call for collective statesmanship that has been sorely lacking so far.

To move to the practical, these are some of the steps that can be – and have to be – taken:

Challenge Iran’s hold over Syria, Iraq and Lebanon (through its proxy, Hizbullah). To Iran, this hold is claimed as a prudent defence in a hostile neighbourhood. To the Arab side it is an unacceptable power-grab and a threat. The classic formula for an avoidable major war. The challenge should be peaceful, backed by clear arguments and offering measures that would reassure Iran in return for its withdrawal. Patient diplomacy, in other words. Outside powers should help.

Refrain from excessive armaments. Every state has the right to self-defence: if you wish for peace, prepare for war. But excess of military power creates fear and is destabilising. From the well-armed position the Arab Gulf States now have, they should feel the confidence to build the regional architecture they need to bring true security. This means engaging with Iran, and with Iran’s perceived greatest threat, Israel. The discreet contacts which now exist between some of the Gulf States and Israel provide a good basis for a mediatory role.

Do nothing to strengthen the hard-liners in Iran, which would make all this impossible. President Rouhani has his own struggle with the militant elements who still claim the export of the Islamic revolution and Shi’a sectarianism as a justification for their intervention in Iraq, Syria and Lebanon. They are wrong, and the overwhelming mass of Iranian society rejects that tired and self-serving propaganda. Respect for national sovereignty and the protection of religious freedom under the law is the only basis for peace. President Trump’s mishandling of Iran and the nuclear agreement risks achieving precisely the return of the hard-liners: regional players should pre-empt this effect and open their own more constructive line of policy.

Finally, end all conflicts including the war in Yemen, and preserve the Kurds from further conflict, building on their justified pride in helping defeat IS and respecting their distinctive development.

All of these measures fit perfectly with the desire to modernise society and governance, and deprive extremism of its roots, a fact often obscured by the dire headlines of conflict in the region. At this inflection point, those governments should seize their region’s future.