For many years now, the Labour Party and the wider Left has enthusiastically capitalised on identity politics as a strategy. By taking full advantage of any opportunity to pitch itself as the political vehicle for a wide range of divided and subdivided identity groups, Labour has managed to attach its own brand to various group identities, sometimes with remarkable electoral results among narrow demographics.

That was a dubious, if profitable, strategy to pursue. It required them to disregard reasonable concerns – about prioritising identity over beliefs, principles or outcomes, and the downsides for wider society of stoking division along the lines of race, faith, gender and so on – or to view such knock-on effects as a price worth paying in return for electoral advantage.

Labour might have been willing to strike such a bargain, but it seems that they underestimated the degree to which they were taking a risk by trying to harness very powerful forces which they did not fully understand.

The thing about identity politics is that it forces its practitioners to create a hierarchy of identities. Those who believe class is paramount argue that the workers of the world have a shared interest which trumps national identity. Those who see religion as the essence of their identity might identify more strongly with a fellow believer far richer or poorer than themselves ahead of someone of the same class, or nation. And so on, and so on, ad infinitum – the capacity to cut and cross-cut each group ever more finely has proved limitless.

Such beliefs can be powerful and dangerous, but identity-driven groups have the relatively simple task of arguing their own identity is most important, regardless of whether they are right or wrong. The complexity arises for those who try to align one political party to a wide range of such groups at the same time, and who do so not by emphasising shared issued which run across them but by pitching that party as each separate group’s natural representative.

Before long, some kind of conflict becomes inevitable; it’s impossible for one party to be the simultaneous and equal representative of the conflicting beliefs of several different groups. A clash occurs, and the party in question then has to start prioritising one group above another. Suddenly, the forces they thought they had harnessed become very difficult to contain.

For example, how can a Labour Party which pitches itself to some as radically feminist simultaneously present itself as the voice of traditionalist and paternalist parts of the Muslim population? It was the parallel pursuit of these two avenues of identity politics, two applications of the same strategy, that led to some Labour meetings being gender-segregated, even while others were discussing the further liberation of women.

You can phrase the question that arose several different ways: Which group was right? Which group did Labour agree with? Which was more important? Which came higher on the hierarchy of identities? It fell to Harriet Harman of all people to try to reconcile the two wildly different types of Labour Party – she half-heartedly rowed back on her life-long insistence on gender equality when she argued that at least gender-segregated events were “better than a men-only meeting”.

There are plenty of other instances of such tensions between identity groups that Labour tries to cultivate. The latest to flare up threatens to be particularly hard-fought: the battle going on over the place of trans women within Labour’s wider promotion of female candidates.

In its broad position on trans identity, Labour’s policy is to fully support self-identification – the principle that someone who identifies as a woman simply is a woman, regardless of biology at birth or indeed biology at the time of or after self-identification. That’s an increasingly widespread policy (including being backed by Maria Miller last summer), but it is not uncontroversial – and now it has brought the Party into conflict with some feminists within its own ranks.

Their objections take two forms. Some argue that identification alone does not imbue someone with the structural disadvantages that affect those born biologically female – from period poverty to discrimination when pregnant to lower incomes due to career-breaks while bearing children – and, they suggest, someone without that experience cannot represent such concerns, and so should not be eligible to stand as Women’s Officers in the Labour Party. Others feel that where there are schemes such as All-Women Shortlists, they ought to be reserved to provide some restorative advantage to those who have never enjoyed the ‘privilege’ of being a man, and so should only be open to life-long biological women, not to trans women.

Neither of these critiques is confined to Labour – each is a localised expression of wider argument made by some (but far from all) feminists. The now-famous protests to deny Germaine Greer a platform at Cardiff University, for example, were spurred by her criticism of treating as a woman “a man who has lived for 40 years as a man and had children with a woman and enjoyed the… unpaid services of a wife, which most women will never know…[who] then decides that the whole time he’s been a woman.” Julie Burchill notoriously expressed a similar view in far more blunt terms, as have various other feminists – often of an older generation. Miller noted after publishing her report that “the only negative reaction that I’ve seen has been by individuals purporting to be feminists”, and Helen Lewis’s considered piece for the New Statesman further illuminates the complexities of the debate.

In short, what’s happening within Labour is a skirmish in a wider clash between the fundamentally conflicting beliefs of two identity groups – both of which the Party has simultaneously sought to appeal to. While the Labour NEC has ruled that All-Women Shortlists are open to trans women on the grounds of self-identification, a pressure group calling itself ‘Keep All-Women Shortlists Female’ is campaigning against the decision on the grounds that “We believe that the election of self-identifying transwomen as women’s officers and their inclusion on all-women shortlists is reducing and undermining female representation in the Labour party.” At least one activist involved in that campaign has been suspended from the Labour Party for their behaviour.

It’s impossible to see how the two positions can be reconciled without one prevailing. Each side demands fundamentally incompatible things – the idea that there should be separate officers and initiatives to aid trans women, put forward by ‘Keep All-Women Shortlists Female’, obviously flies in the face of the competing idea that trans women are simply women like any other and should be treated as such under Labour’s rules.

It might all seem theoretical – and indeed at times it is heavy on both theory and jargon – but it would be a mistake to underestimate the strength of feeling about the issue on both sides, or the capacity for conflict that it involves. That strength of feeling is exaggerated by Labour’s routine emphasis that anyone’s group identity is sacrosanct, to be honoured rather than disrespected or challenged in any way. Yet now the Party finds itself forced to tell one group or another, effectively, that their concerns come in second place.

Really, this is a cautionary tale. Identity politics is a tempting electoral tool in the short term. What party wouldn’t pause for at least a moment at the offer of a strategy that holds the potential to bind whole chunks of society to its banner through fundamental and powerful social forces? And yet there are dangers in doing so. In the case of Labour, it’s clear that those trying to harness these forces can end up falling victim to them – having started out posing as an “ally” to a range of groups, they now have to choose which to mortally offend and which to stand by, after years of treating offence as an unforgivable sin.

Wider society, too, suffers when people are encouraged to believe that their collective group should be privileged above others, or when they are told that the inherent privilege of their race or gender disqualifies them from equal consideration. At best, such an approach encourages us to neglect the things we have in common. At worst, it opens up a whole political arena in which anyone – not just self-defined progressives – can prosper. In the United States, white supremacists now speak the language of identity politics and community liberation, playing left-wing collectivists at their own game.

There’s a conservative (and Conservative) alternative to this quagmire of relativism and social hazards, which doesn’t require us to start picking preferred groups, trampling individual liberty, or insulting personal beliefs. It involves treating people as individuals, judged on their merits and their actions, not primarily as members of one collective identity or another. As we all know, this brings its own challenges, not least the inevitable criticism that without such positive discrimination, change is too slow. It isn’t always a comfortable or easy principle to stick to, but the alternative is far more difficult to manage.