Eighteen months ago, central Mogadishu was like an African Stalingrad. The heat may have been equatorial but everything else seemed strangely familiar: a dirty cat-and-mouse war, often fought hand to hand among the spectacularly bombed-out ruins of a once-thriving city centre.

On one side were the forces of the Western-backed government, supported by thousands of Ugandan and Burundian troops of AMISOM, the African Union Mission in Somalia. On the other was al Shabaab, a virulent militant Islamist organisation aligned with al Qaida. The two sides had been fighting for control of the capital for three years.

Between offensives it was possible to take a tour of the battlefield, courtesy of AMISOM, whose troops commuted there from their base by the ocean-front airport, shuttling back and forth in convoys of Casspirs, hulking armoured personnel carriers with bullet-cracked windows and V-shaped hulls designed to deflect mine blast.

The front line was an imposing wall of sandbags that snaked through miles of roofless residential districts, a post-apocalyptic ghost town where the danger of random mortar or sniper fire was constant. The soldiers on the fire steps manned their gun-slits from the comfort of smashed-up sofas and armchairs rescued from abandoned sitting rooms. Al Shabaab had developed an extensive network of tunnels and trenches, and in some places they were dug in less than 50m away. They had learned to crawl even further forward, under cover of night and the sound made by the shredded tin roofs flapping and clanging in the hot sea breeze, and to lob grenades over AMISOM’s parapets.

AMISOM had been advancing recently, although progress was costly and desperately slow. A Ugandan commander told me that it could take three days just to clear one small house. At this rate, he calculated, his men would still be fighting through Mogadishu in 2015.

Today, though, there are no trench lines in Mogadishu. On 6 August 2011, to the astonishment of just about everyone, al Shabaab pulled back overnight from all city centre positions. Their propagandists called it a tactical retreat, but it turned out not to be temporary. The insurgency was collapsing across central Somalia and falling back on its heartlands to the south.

Sensing the opportunity, Somalia’s neighbours Ethiopia and Kenya quickly joined the AMISOM effort and invaded from the west and south. In September 2012 the Kenyans captured al Shabaab’s last remaining stronghold, the southern port of Kismayo, effectively ending the insurgents’ long ambition to take over Somalia.

This is an astonishing moment for a country long dubbed the “world’s most failed state”: the first chance in a generation for genuine change, and what the UN Special Envoy Augustine Mahiga called “an unprecedented opportunity for peace.” As turning points go it is comparable, perhaps, to the US ejection of the Taliban in Afghanistan in 2001.

Somalia has, famously, had no properly functioning central government for over 20 years. Its leaders have long been riven by internal clan rivalries, and hamstrung by outrageous institutional graft. For the last six years, Somalia has consistently beaten Afghanistan to the bottom spot on Transparency International’s annual ‘Corruptions Perceptions index.’ Yet in 2012, Somalis held their first democratic elections in decades, ousting their former Islamist president, Sheikh Sharif Sheikh Ahmed, and replacing him with Hassan Sheikh Mohamud, a little-known university professor who used to work as a consultant for the UN.

There are other reasons for cautious optimism. So many of Mogadishu’s long-abandoned seafront villas are being rebuilt, in many cases by owners returning from twenty years of refugee exile, that the city is experiencing a minor property boom.

Meanwhile, piracy in the Indian Ocean, although far from eradicated, appears to have peaked thanks to land-based efforts by the regional Puntland government and cleverer counter-piracy measures at sea. There were 70 Somali-related attacks on shipping in the first nine months of 2012, compared to 199 over the same period in 2011, according to the International Maritime Bureau. It was reported this month that the Gulf of Aden has now been surpassed by West Africa’s Gulf of Guinea as the world’s piracy hotspot.

On the face of it Somalia represents that rarest of things, a good news story from the Muslim world. Even the threat of further famine, which followed the region’s worst drought for 60 years and that killed tens of thousands of children in 2011, suddenly receded thanks to unusually kind winter rains.

There is, though, no room for complacency. The new government is still unproven, and al Shabaab are far from defeated. Indeed, the militants had already begun a switch to a deadly, Taliban-style hit-and-run strategy before their withdrawal from Mogadishu. Terrorist attacks are also rising alarmingly in neighbouring Kenya, including in the once-safe Somali enclave of Eastleigh in Nairobi, and the Muslim-dominated tourist areas in and around Mombasa.

As the British ambassador Matt Baugh points out, fixing Somalia is not just in Somali interests but affects the security of us all. “Somalia represents a kind of threat we haven’t seen before,” he said. “There are massive numbers of Somalis living in all the neighbouring states as well as around the world. It is not a traditional, geographical country, but a diffuse, global entity – and that is not physically containable.”

Some 2 million Somalis fled abroad after the civil war of the 90s, and now form one of the largest diasporas in the world. There are perhaps 300,000 of them in Britain alone. The Islamists have already shown a willingness to export their ideology abroad, as well as an ability to recruit in the West. The danger of ‘home-grown’ Somali terrorism was amply demonstrated by the failed suicide bomb attacks against London Transport in 2005: two of the 21/7 conspirators, Ramzi Mohamed and Yassin Omar, were born in Somalia.

If a security threat cannot be contained, the only alternative is to neutralize it by tackling the main driver of terrorism: the discontent of the Somali young, whether here in the West or in their homeland. As Afghanistan has shown, Somalia’s problems will not be solved by military means alone. The Somali state needs rebuilding from scratch, through sustained Western commitment to political, social and economic reform. The question is whether the West truly has the appetite for this mammoth task.

A major international peace conference in London in early 2012 – the 20th on Somalia since 1991 – was trumpeted as an outstanding success by the Cameron government, yet many Somalis complain that there has been no real follow-up on the pledges and promises then made. They know that their country remains a very sick patient that will need the best aftercare available if the disease of state failure is not to go into remission.

Somalis want what young people want everywhere: education, jobs, security, a home. Without the hope of these things, young people, and particularly young men, may turn in desperation to violent rebellion; young Muslim men may also turn to extreme forms of Islam. The clue, perhaps, was always in the insurgents’ name for themselves: al Shabaab in Arabic means ‘the Youth’. It is no doubt significant that Somalia has a particularly low median age of 17.8, and that this is about the same as in Afghanistan.

As in all those countries affected by the so-called Arab Spring, the challenge for the West is primarily a demographic one. ‘The US does not have a robust and comprehensive strategy for targeting the connection between youth and conflict,’ Professor Jennifer Sciubba, a demographer and adviser to the US Department of Defence, said recently. (She was talking about Afghanistan, but might just as easily have been referring to Somalia). ‘Victory, in whatever form, will remain elusive as long as this segment of the population is marginalized.’

In the course of my research I was constantly struck by the similarity of al-Shabaab foot soldiers, pirates and the members of Somali street gangs I interviewed in Britain and the US. They were all young men, and in some cases – such as Abdi-Osman, a 23-year-old ex-pirate, ex-al-Shabaab fighter whom I met in Mogadishu – literally interchangeable. ‘Every man who has nothing will try something to get money,’ Abdi-Osman explained.

Salvation will likely come from two directions. The first may be the oil and gas sector. It has long been known that Somalia possesses important reserves, both in the north of the country and to the south off the shore of Kismayo. With al Shabaab in retreat, the more adventurous prospecting companies are already circling, bringing the promise of massive foreign investment and, eventually, Gulf-style oil wealth to this impoverished nation.

The second, paradoxically, is the diaspora itself. Of course, not all young Somali exiles are potential terrorists. A whole generation have grown up in the West who are out of patience with the old ways of doing things, above all the traditional system of quabyalad, tribalism, which played such a key role in the destruction of their country in the first place.

The best of them have taken advantage of the opportunity to better themselves through work and education, absorbing Western values and ideas along the way. They represent a rich sump of reform-minded talent, and an extraordinary number of them are movingly determined to export their ideas back to Somalia in order to help rebuild their troubled homeland.

The young should be the West’s partners of choice in any African nation-building project. The future of Somalia may depend on our ability to listen to them.

The opinions contained in this article are solely those of the writer, and it does not represent the editorial opinions of kacadeey online