Measuring Programs Vs. Measuring Results

November 16, 2013

Tommie Martin got down to brass tacks when visitors from First Things First presented their story to the Gila County Board of Supervisors in 2010.

“How do we measure outcome versus output?” Martin wanted to know.

Then really warming to the topic, she spelled out concerns that many might share:  “I am interested in what we are doing that looks at real outcomes, not just that we are promoting more and more programs.  To me that’s not a real measure of success.  Lots of programs are not a measure of success; children succeeding are.”

Martin is just one supervisor in one county.  She has no authority whatsoever over First Things First.  First Things First is not really answerable to anyone but itself – and the public.

But her plain talk represents a major dilemma for First Things First, particularly if advocates of early childhood development want to attract more money to the cause.

First Things First administrators are eager to cite the number of programs, the dollars spent, and all kinds of program measures such as the number of information kits distributed to parents, the number of home visits made, the number of children who had their teeth checked.  They can tell you the number of stars for each early-care center participating in Quality First, and how many of those rankings have improved.

But even three years after making the statements quoted here, Supervisor Martin is still right on the money.  Those involved can quantify the effort made, but they can’t quantify the result.  No statistics reveal how many more children are how much more ready for school.

‘In God We Trust, All Others Bring Data*’

Some regard this of little consequence.  They know deep down in their hearts that these programs, all of which they explain are “evidence-based,” are by their very existence helping children.  But the ultimate question is how many children are progressing and to what degree.  Is it $132 million worth?

There are at least three good reasons why First Things First would want to answer this question:

  • Most importantly, to make itself more effective in investing in activities that are shown to help achieve the goal and cutting out the ones that don’t.
  • To answer not just the “show me” crowd such as Tommie Martin but more importantly to answer critics.
  • To start to build the case for more investment in the effort.

All this is not lost on First Things First.  It has commissioned several reports, including one by the state universities and a more recent one by a national advisory panel, and it has staff that is gathering lots of program data.

If pressed, advocates fall back on studies done elsewhere.  For instance, the website for Build Arizona – an organization seeking to create a wider network of support for early childhood programs – cites research showing that “every dollar invested in quality early childhood education saves taxpayers more than $16 in costs associated with crime, welfare, remedial education and other costs.”

From that we might extrapolate that the $132 million being invested each year in Arizona will one day bring the state annual savings of $2 billion.  That’s a mighty good deal.

But before taking the money to the bank, one might want to probe a little further.  The reference is pretty clearly to a study done of 123 high-risk children who with their mothers participated long ago in a carefully managed program at the Perry Pre-School in Ypsilanti, MI.  Whether the laboratory experience at one pre-school can be scaled up across an entire state is open to question.

For those who look for Arizona-specific results, the voter initiative that created First Things First is itself of little help.

While the measure describes in extensive detail how the entire endeavor is to be structured and is to make spending decisions, there are precious few words on the expected outcomes.  “Success,” it says, will “be measured by outcomes for children and families.”

What Does ‘Success’ Look Like?

The First Things First board has stepped into this void by establishing a set of “school readiness indicators.”  These targets tab well-baby visits, healthy weight, dental health, family support, enrollment in early-care programs, and identification of developmental delays.

The indicators assess progress on factors that are assumed to affect school readiness.  As such, they get a step closer to measuring outcomes.  Even so, these particular indicators won’t indicate whether children are actually more ready for school, or by how much, or the source of the improvement.

That type of assessment likely will have to emanate from one of three directions, although each presents difficulties:

Longitudinal tracking. As recommended by the national advisory panel, the systematic tracking of key data on children, families and the services they are provided might be the most effective method of ascertaining which programs contributed most to school readiness.
Privacy issues.
No corresponding system for tracking children once they enter the school system.

Kindergarten readiness.  The logical moment for evaluating readiness for school is the moment when children enter school.
What constitutes “kindergarten readiness” has not been established.
Aversion to adding still another assessment of children.
No baseline against which to measure progress.
Note: Arizona is one of nine states participating in a $6.1 million study to develop, pilot and evaluate a new “kindergarten entry assessment.”

School performance.   The ultimate test of how well children are prepared is whether they end up performing better in school.   Conceptually at least, the test scores of Arizona’s third-graders should start to reflect that in a few years.
Introduces another variable into the evaluation of readiness.  Progress in early childhood could be subsequently washed out by poor schooling before the testing occurs.
The existing baseline for third-grade performance is about to be lost with the switchover from the AIMS assessment to the Common Core Standards assessment.

If that isn’t enough of a problem, evaluating these early outcomes is really only the first part of the question.  The economic justification for making additional investments would need to be predicated on what will happen to these children down the road.

To what degree will better preparation lead not just to more success in school but to higher graduation rates, less social services, less crime, more high-paying jobs?   That’s the kind of data that would substantiate a $16 payback for every $1 invested.

But we’re getting ahead of ourselves.  In the short term, there’s clearly a challenge just in figuring out how to answer Tommie Martin.  Her question is not likely to go away.

– Richard Gilman

* With thanks to the late W. Edwards Deming, renowned statistician and industrial consultant.

Back to main story

Link to Services Vary Widely By Region

← Home page 23 thoughts so far. Contribute Yours Below.

Contribute your Thoughts to the Community

©2018 ThinkingArizona. Material from any ThinkingArizona pages may not be published, broadcast, rewritten or redistributed without written permission. All rights reserved. Privacy Terms Contact RSS

Please enable JavaScript in your browser
for a media-rich experience.