Wednesday, October 03, 2007

More voucher fun: Our rebuttal to the Sutherland Institute's rebuttal

We like it when people stick up for their views, so we appreciate the detailed comment left this morning by Derek of the Sutherland Institute regarding our recent post about the lack of detail in the Institute's latest pro-voucher announcement. (See also Bob Aagard's post responding to Institute president Paul Mero's comment on his blog.)

We feel we owe it to Derek to relay his concerns in full, along with our responses, since they otherwise would be buried in the comments of an old post. But first this: Voice of Utah is not independent on this subject. We have never purported to be. We think vouchers are a thinly veiled excuse for mostly wealthy people to get tax money for sending their kids to religious schools. We think the scheme lacks accountability, sets bad precedent, and is just a bad idea in its present form.

Now, suppose Voice of Utah issued a press release announcing that we had conducted "independent" research on a voucher issue and (yea!) reached a result favorable to our view, with no details about how terms were defined, who we contacted, who responded, what figures each provided, and admitting that we had to "adjust" our results a bit to reach the dollar figure we featured in our headline. Would anyone buy that? No; and no one buys it when the Sutherland Institute does it, either.

And now, Derek's objections:

I did the research that the Sutherland Institute summarized in its press release. It seems clear to me after reading this blog post that your intent had nothing to do with a rigorous investigation into truth or fact, so I will attempt to answer some of your criticisms and do the work you were unwilling to do.
The work we were unwilling to do? Did you read our post? It did not purport to be a "rigorous investigation into truth or fact"; rather, it questioned whether the Institute had conducted a rigorous investigation into truth or fact as it claimed. You guys are the ones who issued the press release, remember? You should not be surprised when skeptical readers ask for the detail behind conclusory headlines.

First a little background, the topic we were investigating at Sutherland with this research was how vouchers affect affordability of private schooling for low-income families. This leads to the research question we wanted to answer: will vouchers bring a significant number of private schools within financial reach of low-income families? For high end schools (annual tuition over $10,000), the answer is obviously no. By excluding these schools then, we are now looking at whether vouchers will bring low to medium end schools within reach.
That might fly if you didn't then use the "adjusted" number to reach figures -- percentages and actual dollar numbers -- that you featured in your headline and press release. Your headline says: "Average tuition at Utah's private schools is $4,250." No, it isn't, even under your own survey. I guess we're lucky that you didn't eliminate all unaffordable private schools and then announce that 100% of "Utah's private schools" are affordable.
If a signficant number of these schools are not within reach for low-income families with a $3,000 voucher, we can conclude that vouchers will not help low-income families. As our results show however, vouchers do indeed bring low and medium end private schools within financial reach of low-income families: 64% of low and medium end private schools charge annual tuition under $4,500.
OK, it's 64% of low and medium-end private schools, taking into account that more than a quarter of the schools surveyed did not respond. Doesn't have quite the same ring to it as the headline. Incidentally, we are legitimately interested in which schools responded to your surveys and which didn't. Any info on the 24 no-shows?
If you don't believe that this tuition level is "affordable" look at Children First Utah, a privately funded voucher program for low-income families only. Their scholarships are capped at 1/2 tuition, no matter what private school recipients attend (which is much less than would be covered by a $3,000 voucher at 64% of the low and medium end private schools). Children First Utah received over 1700 applications last year alone, giving away 375 scholarships. This shows that many low-income families expressed a willingness to pay a HIGHER proportion of private school tuition than a voucher would cover because their children don't fit the public school mold. Your "the poorest families who would qualify for the $3,000-per-child max would hardly notice an extra $127 per month out of their paychecks" criticism is a straw man argument. We never make that claim (I'm talking to you as well jennifer). Of course they'll notice it, BUT THEY ARE WILLING TO PAY IT! That is our claim; that is what the facts show. Bottom line: $4,500 is an affordable tuition rate for many low-income families.
Straw man? Here are Mr. Mero's own words: “Affordability is a subjective term,” said Sutherland Institute President, Paul T. Mero. “But consider a low-income family that receives the maximum school voucher amount of $3,000 per child. The difference between the average tuition rate and the maximum school voucher is $1,520, or $127 per month. That is less than the cost of a car payment.”

Mero did not literally say that "a family wouldn't notice" the money (we thought that was fairly obviously our characterization of his words), but do you seriously deny that he is downplaying the impact of $127 per month -- $254 per month for two kids, $381 for three kids, etc. -- on a low-income family's budget? If by "affordable," the Institute means that some low-income families would be willing to pay it even if they could not truly afford it economically, then that is a unique definition of "affordability."
As a side note, these results show that your assertion relative to Catholic schools is false. There are only 14 Catholic schools in the state, and yet 37 schools in our survey charged annual tuition less than $4,500. You do the math: the majority of "affordable" schools are NOT Catholic. I noticed by the way that you failed to do any fact checking on the tribune article's assertions (or at least you didn't mention doing any). Why is that exactly?
Because the Trib article actually had facts in it. You know, names, individual figures, etc. We looked for that stuff in the Institute's press release and its website. We actually assumed we would find a detailed study, that the release was sort of an "executive summary" and that the nitty gritty would be available elsewhere. It seemed odd to us -- really odd -- that it was not. However, in recognition of your argument, we are changing the wording of our update in which we referred people to the Trib article. See what happens when you give people actual info?
The tuition question we asked was: "How much is your annual tuition?" The counties that had private schools when we published our results were: Box Elder, Cache, Davis, Duschesne, Garfield, Grand, Iron, Kane, Salt Lake, San Juan, Sanpete, Sevier, Summit, Tooele, Utah, Washington, and Weber. This information is easily accessible on the Utah State Office of Education's website. A newly updated private school list adds Piute county, making 18 of Utah's 29 counties with a private school in them. The Wasatch corridor includes the folling areas: Cache County, a slim portion of Box Elder County, Weber County, Davis County, Salt Lake County, and Utah County. Why do we define it this way? Look at a topographical map and maybe you'll realize why (if you still have a hard time, give us a call). Yes there are private schools in St. George, four of them in fact. You could have found this out for yourself with another quick search on the Utah State Office of Education website.
Oh, this is how it works: The Sutherland Institute issues a press release, and it's up to readers to figure out for themselves how the research was conducted, what definitions were used, etc. (assuming they have sufficient intelligence, of course, which you rightly question with us). What a novel approach!
If you really wanted to know the answers to any of the questions you raise in your post, why didn't you just pick up a phone? We aren't hiding the information; we couldn't if we wanted to (much of it came from publicly available sources like the Office of Education). It's obvious that your desire to "actually...know this information" only extends as far as it doesn't risk discovering facts that contradict your view of the world. Your accusation of our results as being "intentionally misleading" is truly ironic.
We'll take that as an invitation for Voice of Utah to call and get answers to these kinds of questions in the future. However, you still don't seem to grasp a principal point of our post: If a non-profit organization (1) claims to have conducted "independent" research; and (2) issues a press release; then (3) it should provide the minutia to back it up, or prepare to be called on it.

Remember, you issued the press release. You announced that you had conducted a study. We didn't. Yet the Sutherland Institute wants us to go do research (make that "independent" research) to figure out the detail behind its study. Talk about ironic.

P.S. We still believe your headline and results were intentionally misleading. Just thought we ought to mention that.

5 comments:

Misty Fowler said...

Do you feel blue in the face, yet?

Derek said...

Voice of Utah--I appreciate your gracious and thoughtful response to my post. In the future I'll check back sooner when I post to your blog to look for responses (many blogs I've come across don't respond at all to posts). For that matter, is there any way to be notified when you post responses to something I've posted, so I can get back to you in a more timely manner?

Unfortunately due to prior commitments this evening I don't have the time to post the response I would like. But just to let you know, I plan on responding on Monday. Stay tuned.

Voice of Utah said...

You know, that's a good question. I don't know if there are notification options or such. (I'm not the most computer-literate blogger.) I'll surf around Blogger to see if I find something. I usually just check the Bloghive --Utahbloghive.org -- to see what people have posted.

I look forward to hearing from you again (and yes, I mean it.)

Derek said...

Voice of Utah--I apologize for the delay in posting this response (long story short, my car broke down on Monday and I am actually posting this response from a conference in Maine). Thanks as well for looking into notification possibilities. Now, on to the response.

You write about your original post:

"It did not purport to be a "rigorous investigation into truth or fact"; rather, it questioned whether the Institute had conducted a rigorous investigation into truth or fact as it claimed."

I don't mean to pick at small issues, but by questioning our research aren't you really trying to investigate the truth and facts behind our research? You're trying to get at what we did and why we did it, right? My point then is that your method of doing so, as recorded in your original blog post, had no hope of accomplishing its objective because it was limited to one search of our website--i.e. it had no rigor.

You continue:

"You should not be surprised when skeptical readers ask for the detail behind conclusory headlines."

My surprise with your post was not the skepticism it contained, but rather the cynicism. I expect (and indeed hope for) public skepticism when any research or policy analysis is published; that is what weeds out sound analyses from poor analyses. What does surprise me however is when groups or individuals dismiss our research or analysis without much investigative effort, believing that since it is done by a group with an opinion that it must be the opinion that is driving the analysis. First of all, no research or analysis is done in an opinion vacuum, even impartial analyses done by government entities. If this is your definition of "independent," then independence does not exist and is a useless term. Independent research is simply research done honestly. The key to discovering independent research is to rigorously investigate the methods and analysis of the research, which I point out that you did not do. Your questioning of our methods and results seems just as applicable to your own blog post as to our research.

You write:

"Mero did not literally say that "a family wouldn't notice" the money (we thought that was fairly obviously our characterization of his words), but do you seriously deny that he is downplaying the impact of $127 per month -- $254 per month for two kids, $381 for three kids, etc. -- on a low-income family's budget? If by "affordable," the Institute means that some low-income families would be willing to pay it even if they could not truly afford it economically, then that is a unique definition of "affordability."

The Children First Utah data shows there are low-income families out there willing to pay the cost of private school--the real definition of "affordable" (i.e. that people are willing to pay for it). I think we should let families' actions define what is affordable, not create that definition for them. Paul's example simply provides the context for people who do not truly understand poverty how private school might be affordable for low-income families, as they show to be the case.

You write:

"Because the Trib article actually had facts in it."

There is little difference in the trib article and our research. the trib article had numbers and analysis of those numbers...so did our research. What is interesting (and what I point out) is that you choose to completely accept and validate the trib's analysis while being wholly skeptical and questioning of our own. This is inconsistent and upon first glance seems to stem simply from your disagreement with vouchers.

You conclude:

"We'll take that as an invitation for Voice of Utah to call and get answers to these kinds of questions in the future. However, you still don't seem to grasp a principal point of our post: If a non-profit organization (1) claims to have conducted "independent" research; and (2) issues a press release; then (3) it should provide the minutia to back it up, or prepare to be called on it."

You should take it as an invitation to call us in the future. Contrary to your belief we aren't trying to hide anything. If I failed to grasp the point of your post, it was because it was laced with cynicism (which you yourself infer in your post:"hate to sound cynical..."). Your post seemed not to simply be "calling us" about not providing our methodology--ironically enough you never actually did call us ;) --but rather used the fact that you didn't find our methodology as an excuse to dismiss our research as a ploy to do whatever was needed to win the voucher debate (i.e. implying we have no intellectual integrity or honesty). Yes we advocate for voucher policy, but that does not mean our research is not independent or sound. If you would have put a little more effort into your investigation of our research, you would have discovered this.

I respect your willingness to discuss these issues on your blog. Your post makes some valid points about our publishing process on our voucher research that has been informative for me. I look forward to future blog posts and discussions.

Derek

Per your request:
Info on the 24 no-shows:
-12 of the phone numbers we had were disconnected
-2 we found out after publishing were residential treatment facilities (would not qualify for vouchers)
-3 schools responded after our publication date, their average annual tuition rates were: $2740, $3300, and one school reported having no tuition, which we are following up on
-7 schools still have not gotten back to us

Anonymous said...

How about means-tested vouchers? I pay for private school for one of my children, and agree that I should continue to do so since I can afford it. Means-tested vouchers would allow poor families with gifted children to give their kids the same great education I give to my gifted child.