phyg

originally posted at https://canmom.tumblr.com/post/616645...

(This post was written when a former member of the Less Wrong cult talked to me about how my writing had helped her break from the cult, and it prompted me to look back from some distance at my own experiences. The title refers to the fact that during early discussions of whether LessWrong constituted a cult, they ‘tabooed’ the word cult by replacing it with rot13′d phyg. However, it’s not just about LessWrong, but about cults in general.)

I’ve kind of backed off from writing about the lesswrongers for a few reasons, mainly that I’d moved on in my own life. Though another strong reason is that the revelation of all the heinous sex abuse shit going on there (resulting in the suicide of members) meant that it was less ‘internet rabbit hole’ and more ‘some of these people are actively abusing people and many others are in the process of being victimised by them, and it feels very inappropriate to stand at the sidelines poking fun at Roko’s basilisk’

There’s a post going around about Jehovah’s Witnesses, and the way their ‘missionary work’ functions less to bring new members into the cult and more to give the existing members a perception of outsiders as being rude and hostile, thus drawing them back into the fold. So I worry a bit that taking a stance of making fun of lesswrongers helps fulfil a perception that non-members of the cult are a ‘sneer club’, and kindness can only be found inside.

It’s a fine line to walk because part of helping people escape must involve helping them see the flaws in ideas used to control and abuse them. Roko’s Basilisk was a rather crude example, but there’s many variants; certain LessWrong members seem very adept at manipulating feelings of guilt and obligation, and part of that often seems to involve trying to make people feel personally, individually responsible for very large-scale dynamics to which the person (and LessWrong in general) is the only remedy.

So you need to kind of make clear that no, what’s at stake isn’t the future of humanity, that all the stories they tell about AIs and so forth are science fiction.

My own history on the periphery of cults

I should also note that I kind of feel that the difference between ‘cults’, ‘religions’, ‘ideologies’, ‘movements’, arguably even ‘fandoms’ and ‘subcultures’ is often more a matter of degree (along various spectra) than kind. Different dynamics prevail at different scales. I’m going to outline the features that make me call something a ‘cult’ below.

I grew up in Glastonbury, a town that’s home to a lot of the remnants of new-ageism and hippie-subculture in the UK. Undoubtedly there were a few cults around me. My parents were at first neopagans, and later just became the kind of generically nonreligious but maybe a little ‘spiritual’ people which form the majority of the UK.

At some point in my late teens, I found the ‘sceptical’/‘atheist’ movement online, which at the time prioritised deflating alternative medicine and criticising creationism (before its hard right wing turn). It was through them I found my way to LessWrong, which presented itself as a kind of the next level of scepticism: through ‘Bayesianism’ they would systematise the kind of thought prevailing in sceptical movements and turn it into a machine for ‘overcoming bias’ and believing the right thing.

And as it happened, Yudkowsky-the-expert [prophet] wrote, properly applied Bayesianism would lead you to perceive the vast threat of an ‘unfriendly AI’. Wouldn’t it be lucky if someone was fighting the good fight against that before it happened?

After maybe a year, I managed to bounce off getting too deep into LessWrong because of those early posts about Roko’s Basilisk and its culty behaviour, but its traces - a perfectionist belief in ‘utilitarian’ ethics, a kind of po-faced affect where everything must be evaluated ethically at all times, a belief in the great scientific mission of humanity, a weird obsession with testing my beliefs against the things I found most unlikely (in this case, fundamentalist Christians at university), interests in epistemology and aging research and (then very nascent) ‘effective altruism’… I was never really a member but it kind of did a number on me!

At much the same time, I got hit hard by the rise of what I might call ‘online social justice’ (proponents might say ‘intersectional feminism’, but that refers to a lot of things). In this case, I think many of the fundamental beliefs - that our society is structured by deep injustices, such as white supremacy, heterosexual patriarchy, disablism to name a few that were salient to this movement - are absolutely correct. But this doesn’t mean that some of the same dynamics weren’t prevalent.

Again this belief system functioned heavily on guilt: starting with the itemised privilege checklist, the only way to address your complicity in oppression was to obsess over it at all times, and in particular scrutinise your language for inadvertent double meanings. To encourage this, a mechanism for punishing people through public shaming developed; it took a while for people to recognise that the dynamics of who got targeted for dogpiles, and what happened to those targeted, were largely orthogonal to what particularly terrible thing someone may have done or not.

Because this movement heavily overlapped with fandom (as a product of LiveJournal at first - this was where racefail ‘09, the incident that drew me in, played out), a great focus of this movement was media criticism. If the corporate entertainment products we consume could be made to portray The Gays in the right light, then surely social change would follow? I think a lot of this was driven by a need to be doing something within the social spaces we were moving in, which were focused on consumption of fan media.

Unlike LessWrong, ‘online social justice’ had its celebrities (and public sacrifices) but did not have any central charismatic figure. Still, this belief system provided a lot of fertile ground for people to build themselves up as progressive, ‘indie’ alternatives to the corporate media order. Most were sincere in pursuing this, but the ‘winners’ ultimately cast down anyone as it suited them as they scrambled for positions in that same order.

Also unlike LessWrong, ‘online social justice’ enjoyed a certain degree of mainstream success, seeing its language taken up by a few larger outlets; it also ended up provoking a big and very nasty right-wing backlash equally obsessed with the ‘social justice warriors’ who might threaten their power in whatever way. This backlash, though just as nasty and cultish in itself, picked up many of the criticisms of cultish ‘social justice’ dynamics, and so denying these dynamics were significant became itself a moral imperative and made it very difficult to actually assess what is happening.

So to be very clear: I am grateful that my participation in ‘online social justice’, however shallow my concerns seemed in retrospect, revealed a lot of places I was dangerously ignorant and I’m pretty sure in some ways made me a better, more caring person. However, it also gave me some very unhelpful self-destructive thought patterns, which made me pretty insufferable and sometimes quite nasty about things which really didn’t better, and I hope I’m growing out of the worst of it.

Ironically, the SJ side of things helped me avoid getting sucked into LessWrong too bad, because it was obvious that those guys didn’t really give a shit even before I learned about their friendship with neoreactionaries. I never made a decisive break with ‘SJ’, but hopefully I’ve since developed some more robust and less easily manipulated thought forms - that can’t be taken up by someone’s personal campaign to dispose of their victim quite so easily.

Then, in more recent years, my cult flirtation of choice was a strain of Leninism/Maoism. I never got anywhere near joining an actual Leninist party (thank fuck), but I did spend a lot of time challenging myself to read MIM (prisons) and the like, despite it obviously being kinda off.

Much like LessWrongism in relation to ‘skepticism’, the Leninism was able to present itself as a kind of refinement of the belief system I already had, and my rudimentary understanding of capitalism, colonialism etc. They could say that obviously racism is real, but unlike those nerds obsessed with cartoons, we have the right way to analyse it and the right program to destroy it. (That is of course elaborately written out nationalist fantasies and cheerleading for whoever America is fighting at the moment. It’s working great, guys.)

I had one friend who was particularly deep into this worldview, and eventually denounced me and cut all ties because I wouldn’t join in a harassment campaign calling a certain then-popular trans musician (who I have never spoken to before or since) a pedophile. Outside of that, you can see can see this worldview’s traces in posts from that period: I started dropping words ‘imperialism’ more for example.

What really prevented me from getting sucked into this cult was, perhaps, the same obsessive scrupulousness that I’d developed while dealing with the lesswrongers. I spent a lot of time digging into the history literature regarding things like the gulags and the famines in the USSR and China during the periods that the Maoists celebrated, and concluded that the historical evidence they dismissed was pretty strong, and they were full of shit. I didn’t really want to be friends with people who were such huge fans of gigantic incarceration programs and that kind of deflated the whole thing.

I also was lucky enough to find things like the Neue-Marx-Lekture and communisation theory and other more anarchist-aligned approaches to Marx. However accurate they may be about Marx (ultimately irrelevant, ideas are useful or not regardless of who came up with them), they made it very clear how many ways there were to approach this history and the appealing parts of Marx’s work, so the ‘package deal’ presented by the Maoists and Leninists (Marx’s criticism of capitalism is insightful, so you must cheerlead Stalin with us) became obviously nonsensical.

Of course, I’m now much more deeper into the leftist subcultural sphere than the average person. Most people do not have opinions on “self-abolition of the proletariat”, nor do they do any of the dubiously effective offline stuff. I like to think I have a healthily cautious approach to the various prevailing ideologies around me, and an actual sense of humour about all this nonsense (hard to emphasise how important that is!), but who knows what I’ll think in a year…

so what’s a cult anyhow

I’m not familiar in detail with the research literature on cults or ‘new religious movements’, but here are some salient features that seem to me to create the dangerous kind of pattern:

a claim to urgency

there is a great problem which nobody is taking seriously enough. this can be a real problem (gender and racism exist and people are suffering under them every day) or a made up problem (an AI might turn us all into paperclips); it will need to appeal to a specific milieu to be effective (programmers who read science fiction novels, people who have experienced homophobic abuse in their lives, followers of a ‘mainstream’ religion)

this is often presented as a world-ending catastrophe, but it can also be presented as merely a widespread unaddressed social problem. even very minor issues can bring out cult-like behaviour, such as inadequate investigation of some scientific phenomenon. not all cults claim a motivating threat.

a claim to legitimacy

the cult are uniquely equipped to face this problem, or possess a unique wisdom. perhaps they alone have the right theoretical tools, or perhaps they have a claim to a lineage. a very hardline distinction between ‘correct’ and ‘incorrect’ helps here (Mao’s idea of the ‘two-line struggle’ was a gift to cult-builders).

for religious cults (the most stereotypical kind), the cult leader is the recipient of a unique vision, or simultaneously descended from Jesus, Mohammed and the Buddha.
for a certain kind of leftist, the Party (all five members!) is the sole inheritor of the ‘red thread’ of history that begins with Lenin and the Bolsheviks, uniquely upholding the correct ideological line in the face of revisionism.
for (pseudo)scientists like LessWrong, the cult’s methods are truth-preserving in a way mainstream academia isn’t, or hidebound academics are too blinkered to investigate the phenomenon in question whereas we have the sense to see something there.
I’m sure you can think of more… a scrappy underdog speaking truth to power is always a handy one I guess?

a means to isolate members

this can be physical isolation (the cult lives in a remote location), social isolation (members encouraged to cut ties with non-members), or it can be something like opaque jargon or outlandish, difficult-to-explain beliefs so the only people who can discuss the cult’s worldview are other members.

I should emphasise that many people have unusual beliefs and come to no harm for it; it’s their role in a system of power and control that makes the beliefs dangerous, which can work almost regardless of the content of these beliefs.

a threat of punishment

particularly isolation. If leaving the cult means alienating your entire social circle, then it’s an almost insurmountable obstacle. But cults can also instil complexes of guilt (if you leave or disobey, you will be responsible for poverty or the failure of the revolution), or practice regular public shaming. I haven’t personally dealt with physical punishment but I’m sure it’s an element.

the LessWrong probabilistic mindset is ingenious here: you can make something ever so slightly more or less likely according to a subjective probability model, but on the level of intuition that still feels like you are responsible for the horrifying terrible thing!
the ritual of a public apology is another extremely powerful mechanism (and why I’m very wary of leftist notions of ‘self-crit’). even if you’re just doing it to get people off your back, making a definite declaration has an effect on your worldview (are all these people wrong?); and watching someone take the stance of apologetic failure/sinner has a big effect on observers as well in terms of illustrating the lines of power.
a feeling of constant scrutiny and unpredictable punishment is very effective. no matter how hard you try, your words might betray your secret error/sin, and you can never be sure of the underlying principles or reliably apply them, so you must obsessively self-scrutinise and research the ‘right ways’ to act, perhaps even pre-emptively apologise if you catch something you did wrong before. but the prevailing narrative is of course that, all of this is (or should be) simple and obvious! or else the effort of having scrupulously correct language helps demonstrate your personal virtue.

abuse

particularly, sexual. Many of these dynamics are abusive in themselves, but once you’ve got a cult running, it seems all but inevitable that someone will abuse the power they have over members in a more personal, direct way. All the cult or cult-like movements I’ve described, and many other cults I’ve brushed up against like just about every Leninist or Trotskyist party in the UK left, have their history of sex abuse scandals that could call the organisation into question, and cover-ups within the ranks. There’s probably a lot more that doesn’t get revealed.

disposability:

given the power of ostracism outlined above, members must be prepared to learn that a friend has become suspect and that it’s dangerous (in a moral sense, or for their own personal social future) to continue to associate with them. This may happen after a long period of bullying, when it is no longer useful or convenient to keep that person around for the individuals wielding power, or it may happen seemingly at random as a constant threat to the remaining members.

The common feature running through all of these is that they ensure the ongoing reproduction (and perhaps even growth) of the cult. This doesn’t have to mean the same people: some cults keep a small core members for a long time, others (like your prototypical Trotskyist party) have a hard core who wield the power, and a continuous churn of peripheral members who are exploited.

Of course, many if not all of these traits can be recognised in ‘mainstream’ society, under the power of a state or company or in academia:

claim to legitimacy

democratic mandate, ‘rule of law’, the contrast with the Hobbesian ‘state of nature’ or rival ‘authoritarian’ state, a history of success in business, science vs. superstition, possession of scientific and other academic expertise, fame in itself…

isolation within a worldview bubble

this one may seem like a stretch since it’s not really ‘isolation’ if most of society shares it, but mechanisms like language differences, preferential media coverage and the routine incarceration of people deemed ‘insane’ can all help to keep people within a particular ‘Overton window’ within a particular society.

in smaller scales, e.g. a company or academic institution can provide a more specific ‘bubble’ effect, focusing attention and effort on the organisation’s concerns.

claim to urgency

this one’s more complicated since most societies are not organised on the basis of a single overarching threat. probably the closest thing we have is ‘reproductive futurism’, the requirement to reproduce the next generation and the threat of failure to do so. on smaller scales, a project may fail, a company might go bust, the economy might go into a recession and so we must work… and then there’s exceptional events like ‘terrorism’ and the present virus pandemic. (claims to urgency may be at least somewhat legitimate!)

threat of punishment

the police and threat of incarceration and other official punishments are the most obvious mechanisms, but there’s also a lot of punishment done under the guise of healthcare, such as holding people deemed ‘mentally ill’ in wards. medicine is practised to the primary end of ensuring the reproduction of society, and ‘mental health care’ is an indistinct blend of heavy coercion and things that might be genuinely useful in other circumstances.

Apart from that, you’ve got the wage mechanism: if you don’t work for someone with money, you don’t eat. Almost nobody has the option of producing their own food.
You also have people bound together in reproductive units, notably the family. You cannot leave if doing so would deprive you of your means of subsistence.

abuse

is blatantly prevalent anywhere there’s power. the heterosexual nuclear family and prisons deserve special acknowledgement here.

disposability

any ‘social safety net’ is designed to push people back towards work. If you end up homeless on the street, odds are pretty high you’ll die of pneumonia rather than find your way back into some form of stability. We are trained to walk past people who need help, knowing or deluding ourselves that we can do nothing for them, every day we go outside. And that’s not to go into dynamics within just about all specific ‘communities’ to guard the walls and expel problematic cases

Does this mean the label ‘cult’ is useless, since it can capture almost all groups at a stretch? As mentioned earlier, it’s a matter of degree, such as the particular intensity of the cult mechanisms. Thus I still think it’s helpful to talk about how these dynamics can manifest (often all the more intensely) in more marginal spaces.

On LessWrong

LessWrong seems like an unusual cult in a few ways. A lot of its internal discussions are not made private; rather, what keeps them closed is the opaque forest of jargon which can only be parsed given extensive familiarity with its writing.

It’s very conscious of itself in relation to academia in the hard sciences, which is both its aspirational model (providing much of its language and its obsessions) and its bugbear (they won’t take us seriously, we’re too many inferential steps away). They may appeal to a historical lineage to a degree (the Enlightenment!!), but they are also quite proud of making displays of novelty (we’ll propagandise through a Harry Potter fanfiction, look how modern and switched on we are), and enjoy the sense of being challenging and disruptive.

One quite nasty trick is that LessWrong sells itself as improving rationality, and illustrates this by drawing on genuine ‘cognitive bias’ research and pulling out a battery of common epistemological errors which they claim to inoculate against; thus as one gets drawn in they can easily believe that they’re becoming more cautious and sceptical, not more credulous. It also potentially gives a quick way to dismiss people who haven’t gotten in: they are too ridden with bias to be worth consideration.

Of course, LessWrong members do not reason like a theoretical Bayesian agent any more than any other human does. Performing a Bayesian update on all your beliefs in light of new evidence is impossibly computationally expensive, and as they well acknowledge, the majority of our reasoning and perception is better understood on the basis of heuristics and habits which work ‘well enough’ to get by. So far, I doubt they’d be disagreeing with me.

What goes wrong is when the ‘Bayesianism’ starts becoming a rhetorical performance: speaking in terms of ‘probabilities’ which have not been calculated and could not be, claiming to be ‘updating’ when one learns something new, appealing rhetorically to some mathematical property of Bayes’ theorem without actually ever doing a Bayesian calculation.

With these devices, LessWrong members can paint a picture of a careful, considered mathematical reasoner sharing their results in detail, while actually the appeal works on that performance: it uses the right jargon, it affects the right rhetorical style. This performance probably works on the speaker as much as anyone; it feels right to use.

That said, much of LessWrong on this website has moved on from the dramatic performance of Bayesianism per se, but they still have a tendency to write in a particularly insular style drawing more on the rhetoric of the blog Slate Star Codex (which seems to have almost eclipsed Less Wrong itself in the milieu). Despite a few rationalists habitually picking fights with members of other cliques, they tend to fairly effectively repel non-LessWrongers.

Of course, there are many cliques in this website (no doubt I can be said to be in a few!); use of jargon and speaking to a specific group with shared concerns is not in itself automatically a problem. I would be remiss if I didn’t acknowledge that many of the concerns of LessWrong are quite interesting subjects to explore, and aren’t often explored elsewhere without a lot of money to go to university (and be subject to its own forms of brutal hazing). The problem is not that a group of people exist who share unusual beliefs, but the function of the cult-mechanisms outlined above, and the way they lead on to actions like SWATting and the sex abuse linked at the top.

So far, so familiar; and this may be the last thing I write about LessWrong, at least for a long while. I hope my words can be useful for people in that milieu to start considering, if not immediately getting out, at least making sure they are anchored in personal relationships outside the cult context, and treating it with the same degree of scrutiny it encourages you to apply to everything else.

For me, the instructive thing about LessWrong is to try and keep track of these dynamics, so I might be able to tell whether I’m falling into them in some other context.

When do ‘groups of people with unusual beliefs’ turn into cults?

This is really the crux of it, yet also the hardest part. Simply naming cults I’ve orbited is not very useful if I can’t recognise a pattern, or work out a way to intervene (at least inside my own head) which could have helped me before.

In many cases, (pseudo)cults may be a defensive formation. Certain actions - gender transition for example - automatically invite a great deal of hostility, which can really only be survived by banding together with other trans women. That doesn’t make us a cult, but it lays a seed.

By contrast, trans woman exterminationist feminists form a particularly blatant cult, which portrays our existence as a threat in the ways described above. They level the accusation of being a cult against us in ways that are for the most part quite hollow, since they’re predicated on gender transition being unthinkable as something one might actually desire for any reason beyond ‘brainwashing’. Yet I think it would be dangerous to us to completely deny the existence of some cult-like dynamics in the way we treat each other (if not in the illusory ‘trans community’ at large, then in specific cliques and subcommunities around you, transfem reader). Cult dynamics do not require that the threat be fake.

I can observe a few things that help resist this: we have incredibly varied means of analysing gender and naming ourselves, and many attempts to establish an orthodoxy ultimately fall rather flat. As much as I’d like my own understanding to be better known, I think this is a valuable trait. There are also groups of us who have seen these dynamics play out time and time again, and consciously attempt to defend each other.

But I don’t know if there’s any general recipe for anti-cultishness. Even a principle like ‘anti-disposability’ can be used to enforce cult-like behaviour: if you can be convinced that personally cutting ties with someone who’s treating you horribly amounts to disposing of them, as has happened to so many trans women, then you can be forced to put up with a whole lot of shit when actually the person would be just fine if you cut ties, or at least the effect they have on you is not worth bearing.

I often come back to something my friend @porpentine wrote (before I knew her), a followup snippet to her essay Hot Allostatic Load which described her experience of abuse and trashing by a particular cult-like group of indie game developers. Among what she wrote is this:

Rules

HAL can be used to help people or justify abuse, just like anything. If I carve a walking stick because I'm injured and need help walking, and then I throw it away when I'm done, sure, someone can pick it up and hit someone else with it, or maybe they genuinely need it, who fucking knows. But it's not the thing itself; it's just intended to point at other things, to help you walk somewhere else.

Any rule will be abused, that's why leftist spaces with their naive, dogmatic injunctions are a rapist's wet dream. You can't invent a better set of rules, only super-contextual guidelines. Any safety in our lives will come from a deliberate, sustained application of will against entropy. All things decay towards greed and abuse. If I want my life to be less like it was in the cult, I have to daily commit myself to managing a thousand different variables - to make sure my speech about others is not gossip, but equally to make sure I do not ignore harm - to set my boundaries and life my life while also making sure that invisible field of energy is not pushing too far into someone else's - to constantly and forever evaluate the balance of all those variables until I die. This is part of what it means to love.

Like she said, this ‘sustained application of will against entropy’, facing up to the social systems around us in all their harrowing complexity, is not something that can be reduced to an authoritative text, containing a list of rules or fundamental principles from which everything else neatly follows. That’s something which, unfortunately, can be internalised only through experience and practice, and you will make the wrong call sometimes. All I can hope is that the story I tell, and the models I’ve built, can end up being helpful to a few who read my writing.

So what I will say to finish is: there certainly are real catastrophic threats in the world (like climate change), and there are all sorts of daily miseries which endlessly claim more people (far too many to list). This does (and should!) inspire a sense of terrible urgency as you become aware of it. But, this world also has many systemic dynamics and groups which will (consciously or not) eagerly inculcate and exploit your feeling of urgency, guilt and desperation, and use it to control and abuse you, or use you to do that to others. Somehow, we have to act responsibly to find a worthwhile path in all this, to take it seriously but not obsess in a way that’s futile and harms ourselves and others.

It’s not easy, and I certainly haven’t solved it. But that’s ‘the work’…

Comments

goblin

2023-09-19 00:21

honestly, its such a fucking shame seeing people take advantage of the vulnerable in this manner. ive seen far too many friends of my own - genuinely fragile people, marginalized and socially reclused in diferent, unique and altogether very generally endangering ways - get dragged into some weird shit backwards ass fucking prison camp denying “racial cleansing” sympatizing cult-esque subcultures, and just losing these people because theyre a bit “too deep in”; even after warning after warning after 6 hour long debate abouts fucking ‘viability of a large scope transition to a maoism-based society in europe’- just goes to show the systemic targeting of stuff of the sort (eg right wing internet extremism and the general archetype of an ‘incel’) and this is a really helpful summary that may be a catalyst for someone one day break out and cut off from a cult! hell, even just to help single out a movement or something of sorts.

tl;dr ty for making this, might really help some poor shmuck someday