What we know about “What We Know”
The “What We Know” project of Cornell University has produced a number of collections of studies regarding various politically relevant topics centered around homosexual and transgender issues. Its stated purpose is to make the findings of the academic world accessible to the public at large, and journalists, researchers and policymakers specifically.
The project aims to provide simple answers to complex questions and since the expected demographic is not expected to dig deeper, it is important that conclusions made are clearly stated, transparent and sturdy. In line with this, the methodology section for the “What does the scholarly research say about the effect of gender transition on transgender well-being?” collection states the following:
“the peer-review process is imperfect but we operate on the principle that it represents the best method we have for holding research accountable to both good faith and sound methodologies”
The collection contains 72 studies of which 55 are said to be relevant to the question, analysed and divided into the two categories “gender transition improves the well-being of transgender people” and “mixed or null findings”. It was not always this number, which I will cover later.
The first problem is that these terms are very unclear. Transgender alone is very broad, but just to illustrate the wide range of things that fit into the term gender transition, we can start listing them.
First are the social factors, being closeted, partially out or fully out, name change, legal sex change, appearance, being sexually active, your community and the countless different factors that go into how people treat you, and many more.
For the medical side there is hormone blockers, cross sex hormone treatments, phalloplasty, mastectomy, facial feminization surgery, breast augmentation, various forms of vocal surgery, hysterectomy, metoidioplasty, and once again, many more. 55 studies appears a very small number in light of the vastness of the topic, especially since the type and quality of the material varies significantly.
One very interesting thing stands out about their stated methodology.
“Our research protocol was developed in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) Guidelines.”
PRISMA is a perfectly acceptable set of guidelines for systematic reviews and the “What We Know” team explicitly state their review should be able to stand up to them. Yet when we start running down the handy 27 item checklist we find that they have missed almost every single one, with the possible exception of point 8 (search parameters) and if we are very charitable we can say that an attempt was made at point 17 (study selection).
With that said, let’s talk about why the number is different on the main page from the methodology page. If we read their methodology, they call this a systematic literature review, and with two PhDs and an MPH leading the team, one would expect a correction to be clearly explained somewhere.
What happened was this “systematic literature review” contained the same study twice with the same title and authors but taken from two different publications. How many PhDs does it take to count to two? That this could happen in the first place beggars belief, but their treatment of it is telling. A year after publication they simply updated the front page quietly while leaving the old, erroneous, numbers on the methodology page.
When discussing the problems with their process specifically, I should add that there are a number of studies in the collection that do not match their stated search methodology, the most strange being a cost-effectiveness analysis for health insurance coverage¹. The existence of this study in the collection strongly suggests that the stated methodology is a post-hoc construction, and an inaccurate one at that.
There are also a number of studies that match their search methodology but do not appear in the collection. This is not technically untruthful as they later specifically exclude “physical outcomes”, such as death, from their definition of “well-being”. I consider this extremely dishonest, as they still include some studies on suicidal ideation² but appear to have excluded others.
This is all shoddy work, of course, but it gets quite a lot worse. The real problems occur when you start reading the studies and contemplate which category you would put them in, “improved well-being” or “mixed or null”.
To start with, a quote from da Silva et al.³, a small scale cohort study which is also the one that they failed to notice they counted twice.
“When analyzing the facets separately before and after SRS, sexual activity, freedom, physical safety and security, financial resources, and health and social care were improved. Accessibility and quality also were improved after SRS, whereas energy and fatigue, sleep and rest, negative feelings, mobility, activities of daily living, and physical environment worsened.”
The findings are varied and do not fit a simplistic statement such as “improved well-being”. The positives clearly come bundled with negatives and by any honest evaluation, this must be a mixed result. The “What We Know” project has put this one under “improved well-being”.
We move on to Megeri & Khoosal⁴, another small scale cohort study with a very narrow focus, and its findings are accurately summarized in a single sentence.
“There was no significant change in anxiety and depression scores in people with gender dysphoria (male to female) pre- and post-operatively.”
This result unquestionably does not belong in the “improved well-being” category, but that is where we find it. We are now getting close to outright deception, and this is not somewhere we should have been lead by the work of multiple PhDs working for a prestigious university.
A number of the studies say little or nothing at all about the well-being of transgender people, like the above mentioned cost-effectiveness analysis but also literature that merely describe a cohort⁵ or provide mixed data but also hint at how little we know about the topic⁶. These are also found in the “improved well-being” category.
In the findings the key claim is about gender transition as a treatment.
“The scholarly literature makes clear that gender transition is effective in treating gender dysphoria and can significantly improve the well-being of transgender individuals.”
The first objection is to note that most of these studies do not deal with gender dysphoria, they deal with transsexualism. This is a small subset that can be roughly described as the most severe and intrinsic form, and applying findings for that group to the much wider transgender community should be done with great caution.
The second objection is that other reviews of largely overlapping material have described the quality of the evidence as low or very low. The contemporary literary review by Nobili et al.⁷ considered the situation as follows.
“Caution is still needed when generalising the findings due to the studies’ moderate risk of bias as well as their methodological limitations.”
Here caution is urged, but it is completely missing from the Cornell review. Nearly half of the sample size comes from internet-recruited surveys. A fifth of the studies are of less than 30 people. Sampling bias runs rampant, and a third of the studies have extreme loss-to-follow up (between 30% and 75%). These should be major concerns, but they are nowhere to be found in this material intended to inform public policy.
Another claim is made in the findings about the range of rates of regret found in these studies.
“Regrets following gender transition are extremely rare and have become even rarer as both surgical techniques and social support have improved. Pooling data from numerous studies demonstrates a regret rate ranging from .3 percent to 3.8 percent. Regrets are most likely to result from a lack of social support after transition or poor surgical outcomes using older techniques.”
The highest overall transition regret rate (a topic rarely discussed in these studies) to be found is in Bodlund and Kullgren⁸, where the rate is 5.2%. The sample size may be prohibitively small, but that should be explicitly stated as the reason that its findings are ignored. Further, the highest reported procedure regret rate is in Imbimbo et al.⁹, at 6% with a - for this field - relatively low 15% loss to follow-up (stated as “15% refused to partake in the study for various reasons”).
I would also like to point out the use of “extremely rare” and “even rarer”. When we talk about side effects of medical treatment, the word “rare” represents fewer than one in a thousand. “Very rare” would be fewer than one in ten thousand and “extremely rare” would presumably be fewer than that.
If we take their upper bound (3.8%) and apply it to various things, we can see the problems with the phrasing. For example the USA represents about 4.3% of the worlds population, which presumably makes Americans at least very rare. Their lower bound (0.3%) is more akin to being Australian, which is less common but far from extremely rare. In general, using loaded language in a scientific setting should be highly discouraged.
The specific claim that regret rates are decreasing is not consistent with what happens when you order the regret rates provided by studies chronologically, and appears to be based on a single study. That the findings of the review differs from what is actually in the studies it cites is very concerning.
To harken back to their own statement, what I found when I explored this review was the opposite of “good faith and sound methodologies”. That a number of PhDs could produce something that would get rejected if an 8th-grader wrote it is pitiful. The “What We Know” review is deceptive, apparently intentionally, and my personal opinion is that Cornell should be embarassed to host such material on their servers.
[1] Padula WV, Heru S, Campbell JD. “Societal Implications of Health Insurance Coverage for Medically Necessary Services in the U.S. Transgender Population: A Cost-Effectiveness Analysis”, J Gen Intern Med. 2016;31(4):394–401. doi:10.1007/s11606–015–3529–6
[2] Bailey, L., J. Ellis, S. and McNeil, J. “Suicide risk in the UK trans population and the role of gender transition in decreasing suicidal ideation and suicide attempt”, Mental Health Review Journal, Vol. 19 №4, pp. 209–220. doi:10.1108/MHRJ-05–2014–0015
[3] Cardoso da Silva D, Schwarz K, Fontanari AM, et al. “WHOQOL-100 Before and After Sex Reassignment Surgery in Brazilian Male-to-Female Transsexual Individuals”, J Sex Med. 2016;13(6):988–993. doi:10.1016/j.jsxm.2016.03.370
[4] Deepak Megeri, Deenesh Khoosal “Anxiety and depression in males experiencing gender dysphoria”, Sexual and Relationship Therapy, 22:1, 77–81, doi:10.1080/02699200600565905
[5] Vujovic S, Popovic S, Sbutega-Milosevic G, Djordjevic M, Gooren L. “Transsexualism in Serbia: a twenty-year follow-up study”, J Sex Med. 2009;6(4):1018–1023. doi:10.1111/j.1743–6109.2008.00799.x
[6] Pimenoff, Veronica & Pfäfflin, Friedemann. “Transsexualism: Treatment Outcome of Compliant and Noncompliant Patients”, International Journal of Transgenderism. 13. 37–44. doi:10.1080/15532739.2011.618399.
[7] Nobili A, Glazebrook C, Arcelus J. “Quality of life of treatment-seeking transgender adults: A systematic review and meta-analysis”, Rev Endocr Metab Disord. 2018;19(3):199–220. doi:10.1007/s11154–018–9459-y
[8] Bodlund O, Kullgren G. “Transsexualism — general outcome and prognostic factors: a five-year follow-up study of nineteen transsexuals in the process of changing sex.”, Arch Sex Behav. 1996;25(3):303–316. doi:10.1007/BF02438167
[9] Imbimbo C, Verze P, Palmieri A, et al. “A report from a single institute’s 14-year experience in treatment of male-to-female transsexuals.”, J Sex Med. 2009;6(10):2736–2745. doi:10.1111/j.1743–6109.2009.01379.x
