March 6, 2012

The background behind the 5% effectiveness / 95% ineffectiveness claim (LONG dissertation)

I've long read statistics about the success or failure rates of 12-step treatment "rehab" facilities and, separately, A.A.

I've never seen a 97% failure rate figure for 12-step rehab facilities before, or anything near that high. I'll leave it to others to delve into that.

On the "AA has a 95% dropout rate in the first year", the source of this controversy is an (allegedly) internal A.A. document, "Comments On A.A.'s Triennial Surveys (5M/12-90/TC)" available at (http://www.scribd.com/doc/3264243/Comments-on-AAs-Triennial-Surveys) And in particular, "Figure C-1" on page 11, which is a graph that also helpfully includes the data being graphed. The data is as follows (I've also included Figure C-1's heading in the below):

/============================================================
"% of those coming to AA within the first year that have remained the indicated number of months.

1mo 2mo 3mo 4mo 5mo 6mo 7mo 8mo 9mo 10mo 11mo 12mo
19% 13% 10% 9%   8% 7%   7%   6%   6%   6%    6%   5%
\=============================================================

I agree with Agent Green http://www.green-papers.org (or at least I think his interpretation is much more plausible). Here is the relevant excerpt from his web page --

Does AA's retention rate indicate 95% failure?

Orange quotes ( http://www.orange-papers.org/orange-effectiveness.html#AA_dropouts ) an internal AA survey as evidence that 95% of people who begin going to AA meetings will have left after a year. In reality, the graph shows that 74% will leave within their first year not 95% - Orange either doesn't know how to read a frequency distribution graph, or is willfully presenting it dishonestly.

The graph (http://www.scribd.com/doc/3264243/Comments-on-AAs-Triennial-Surveys) is very simple.

The researchers went into different AA meetings and asked the people there how long they had been attending, they plotted the results for those within their first year on this graph by monthly averages. So it shows, 19% of people were in their first month, 13% their second, 10% in their third and so on up to 5% in their twelfth month. Orange claims the 5% of people in their twelfth month indicates that 95% had left after a year (oblivious to the fact that the other 95% in the survey was comprised of those sitting in the same room and with less than 11 months time attending meetings). Agent Orange is in need of a math lesson.

This survey is the other source for Orange's purported 5% success rate. The graph actually shows that 26% of people who try an AA meeting for the first time are still attending AA after the first year, the attrition is from 19% (those in their first month) to 5% (those in their twelfth), and therefore around 74%.

Orange also claims that this 74% attrition is an AA failure rate. I will now use Orange's spectacularly warped logic to prove that exercise is unhealthy. Watch carefully!

•After one year, 74% of people who began work-out routines at a gym are no longer using the gym.
•Therefore gyms have a 74% failure rate.
•Therefore exercise is unhealthy!

It is indeed true that only 26% of visitors to AA stay more than a year, and AA has shown some concern about this statistic. But:

•Some people visit AA and decide it's not for them.
•Some people get sober and decide to leave AA.

Orange's 'failure' statistics turn out to be at best ignorance and at worst flat out lies.

What Agent Green left off, unfortunately (for those trying to defend his interpretation from his critics), is that there is a couple of statements in A.A.'s study which would lead one to believe the Agent Orange interpretation that only 5% remain after 12 months, rather than Green's interpretation that 26% of those in month 1 are still around in month 12. Frankly, from the Green (and my and AAHistoryLovers' viewpoint <1> ), Figure C-1 is mislabeled; and also the page 11 "It is possible to calculate" statement in a2 below is also incorrect:

a1). The heading of Figure C-1 (p. 12): "% of those coming to AA within the first year THAT HAVE REMAINED THE INDICATED NUMBER OF MONTHS." ((emphasis Progree's. Note that Tom E., a writer of a number of postings on this subject at AAHistoryLovers, also indicates the wording is incorrect: "The title of C-1 doesn't match the data" <1> -Progree))

a2). p. 11 - "It is possible to calculate from completed questionnaires, by month, the number of members that have "been around" a given number of months. This relies on the question that determines the month and year that the respondent first came to A.A. The calculation has been performed for the twelve months of the first year for the five surveys, and the results are plotted in Figure C-1. Such results can be interpreted to show the probability that a member will remain in the Fellowship a given number of months" ((immediately following this statement is the "to be more explicit" paragraph in b2 that makes clear the 26% interpretation - Progree))

And here is why we think the Green 26% interpretation is the one intended (and repeating the data again for convenience of proximity):

1mo 2mo 3mo 4mo 5mo 6mo 7mo 8mo 9mo 10mo 11mo 12mo
19% 13% 10% 9%   8% 7%   7%   6%   6%   6%    6%   5%

b1). p. 2 - "approximately 50% of those coming to A.A. leave within 3 months" ((only the "Green 26%" interpretation of the numbers comes anywhere near close to matching this statement -- for every 19 people in month 1 there are 9 in month 4 -- 9/19 = 47% = about half. Perhaps the intended comparison is 19 in month 1 and 10 in month 3 -- 10/19 = 53% also equals about half. --Progree))

b2). p. 11 - "To be more explicit: if all the members who report they have been in the Fellowship for less than a month were still present a month later, then the number who report being in A.A. between one and two months should be equal the number that report being in less than a month, subject, of course, to month-to-month fluctuations and to any possible seasonal effects. The same should apply to succeeding months. However, it is observed that there is a steady decline, (subject to inevitable fluctuations)"

b3.) The percentages add up to 102%, supporting the frequency distribution interpretation ((a frequency distribution table with exact numbers will add up to 100%, but since all 12 numbers are rounded to the nearest whole number, a sum of 102% is quite consistent with a frequency distribution table. It just means that 2 more numbers got rounded up than got rounded down. -Progree))

b4.) It is highly unlikely that given the Orange interpretation of drastic attrition, that only 1% leave in the 5 month period between 6 months and 11 months. Data: (6mo, 7%), (11mo, 6%).

There are a couple of other reasons for thinking the Green 26% interpretation is correct, but they are difficult to explain succinctly.

Anyway, now you know where statements like "according to A.A.'s own statistics, 81% of newcomers leave in the first month (and only 19% remain after the first month); and 95% leave in the first year (and only 5% remain after the first year)" come from. And why that interpretation persists and is so widespread. For example, just Google (without the quotes)

"A.A. 5% retention rate"
"A.A. 95% dropout rate"

and similarly, in the above replace "retention" with "effectiveness" and "dropout" with "attrition" and similar terms to get some more hits.

Another piece of datum cited supporting A.A.'s alleged 5% effectiveness rate is the Vaillant study, which Agent Green also debunks at green-papers.org (and having looked at the Vaillant study extensively myself, I agree with what Green says about it).

Just thought some of you might be interested in knowing where that 5% effectiveness rate stuff comes from. And about the Orange v. Green fight. If you wonder why the A.A.'s don't get together and present the case for A.A.'s effectiveness (such as it is, but almost certainly better than 5%), I don't know either. If you have wondered why A.A. has (apparently) never confirmed which interpretation of the Triennial Survey data is correct, a lot of people are wondering that too.

I agree with the authors of the A.A. study that we could and should be doing a better job at retaining people...

====== Footnotes ==========================
<1> AAHistoryLovers group - see: http://health.groups.yahoo.com/group/AAHistoryLovers/message/2044
See also messages 2379, 3374, 3385