Kazim to Chuck Colson: Prison Ministry statistics revisited

References:

• Kazim's review of The Faith
  
• Chuck Colson's post #1
  
• Chuck Colson's post #2
  
• Chuck Colson's post #3
• Kazim's compiled responses (including this post)

Now that I’ve discussed the central issue, I’d like to return to the information about your prison ministry that you discussed in your first letter.

In my first message to you I contested the success of your prison ministry, pointing out that your study considered someone to be "a graduate" only if they successfully got a job after release. This slants the data. It allows you to take credit for an achievement that which, whether or not someone took your program, would have a strong tendency to lower their recidivism rate. I argued that it is the job that predicts their success, not their participation in your program. I also pointed out that when you take the recidivism statistics altogether, the people who took your program (including the ones who did not "graduate") had an overall worse recidivism rate than the general prison population.

You replied:

"The grounds for the compilation of empirical data, established by Prison Fellowship in cooperation with the Texas Department of Corrections was that we would measure graduates, that is, those people who completed our program, as opposed to simply people who signed up for it. The reason we did that was obvious: we could not select the people coming in—we had no control over that. The state made that choice, as it did with the people in their control group. If both sides had made their own choices, then you would consider including drop-outs. But we knew the state couldn’t choose the kind of people we knew were motivated to do this."

No, of course you can't. That’s kind of the point of a randomized study. If you could cherry-pick people who were already "motivated," then you would be working with a subset of prisoners who were already inclined to get their lives straightened out, and then your sample would be even more biased.

I mean, suppose you knew how to identify the people who were motivated to turn their lives around. Once you've picked them out of the general population, no further action is needed at that point. I'll bet you could just study them and find out that this group naturally has a lower recidivism rate than those who are not so motivated.

"The researchers from the University of Pennsylvania believed that this was sound methodology. We were very clear about this from the beginning, because we use a process of self-selection. In other words, the initial curriculum was geared to a pretty intense biblical grounding, so only people who really wanted this would take part in it. Anyone who has worked in this area, like Alcoholics Anonymous, will tell you that it’s the motivation of the participant that is crucial. If the person doesn’t want to change, you can’t change him. As Christians, we believe everyone has a free will to choose or not choose to follow Christ. So we’re obviously looking for people in our programs who are at least open to that."

It’s not the soundness of the methodology that I’ve questioned here; it's your spin on the results. The study looked fine to me, and I certainly can't go back and try to reproduce the results myself. But I don't need to. On page 18, the study stated explicitly: "Simply stated, participation in the program is not related to recidivism reduction." This is actually generous, since the final results on page 19 show that the program was in fact counter-productive, once you stop filtering out "non-graduates" and just compare participants to non-participants.

Although I understand why you don't wish to count the program's effects on the people who didn't "graduate," the fact that the presence of the program had a net negative effect on recidivism is a very significant result. As I said before, you can't just define "graduation" to be a set of preconditions which, if met by prisoners who didn't take your program, would ALSO result in lower recidivism. Or you can, but if you do that then you don't get credit for prisoners who meet these definitions.

"The acid test here, however, is what has happened since that data was compiled and released in 2003. We have continued to monitor these programs across the country, and they have continued to produce between 8 and 10 percent recidivism."

Of course they did. Those are the people who already got a job. Again, I'm not questioning the conduct of the study, but your interpretation of the results.

"As for the case in Iowa, we did indeed lose it at the trial level. But the case was appealed to the Eighth Circuit Court of Appeals, which held in our favor on all of the substantive questions except accepting state funds. ... on the more important question of whether the program is effective and whether it is constitutional without federal funds, the Eighth Circuit, in an opinion joined in by former Supreme Court Justice Sandra Day O’Conner, did not declare anything we’re doing to be unconstitutional. I consider that a very significant victory."

It sounds pretty hollow to me, I'm afraid. I thought we were discussing whether the program actually helps people by incorporating your teachings, not whether a court allowed you get away with continuing the program. The only reason I brought up the case at all was because you used it in your book as a way to criticize Barry Lynn for hating truth.

"As for the question you raised about whether someone could be paroled, not get a job, and therefore not be counted in our statistics, that’s the first time I’ve ever heard that. I’ll ask one of our staff to look at that, but I can’t believe that’s the case. I don’t think the University of Pennsylvania would have accepted that. The researchers who did this study were enthusiastic about the results."

I'll be interested to know whether you have found anything out by now. It seems fairly clear to me from the study:

"A program graduate is someone who completes not only the in-prison phases of IFI dealing with biblical education, work, and community service (usually lasting 16 months), but also includes an aftercare phase (usually lasting 6 months) in which the participant must hold a job and have been an active church member for 3 consecutive months following release from prison." (page 5)

Sounds pretty cut and dried: if you don't have a job, then you are not a graduate.

Assuming that my interpretation is correct, another way I could think of to create a control group for your experiment is to keep track of all those prisoners who were rejected from your program, but went on to be paroled and then got a job. If you could compare that information, then you might weed out the bias. As it is, I certainly don't think you have sufficient grounds to criticize Barry Lynn for hating your program because it helps people.