Skip to main content

The KIPP model has been held up in many circles as a silver bullet solution for urban education, and the "KIPP question" that I’ve heard many times is, "If KIPP can achieve dramatically better results with urban kids, why can’t everyone else?"

KIPP has been held up as a savior by its supporters and used as a perennial punching bag by its critics. In my view, these exchanges have not been particularly illuminating. And in most battles, both sides have neglected some important questions. So each day this week, I'll write about some of the issues raised by the KIPP case.

The Knowledge is Power Program (KIPP) is a network of 57 schools in 17 states. KIPP has received a significant amount of media attention for the results their schools have posted with urban students. A quick tour through the KIPP Annual Report reveals some impressive gains. What do we need to know before confirming that KIPP has a positive effect on their students’ academic achievement?

One way of outlining the challenge with evaluating KIPP is to think about how we would design a study to estimate its effects on the students KIPP serves. Recall that I am not asking whether KIPP provides a solution for students beyond those currently attending KIPP schools – this is a question I will address on Thursday. But the first question we need to ask is whether a student attending KIPP is better off than she would have been had she attended a non-KIPP school.

Many journalists have attempted to answer this question by comparing the performance of students in a KIPP school with those at the closest neighborhood schools. For example, Paul Tough wrote in last year’s New York Times magazine:

When the scores [at KIPP Bronx Academy] are compared with the scores of the specific high-poverty cities or neighborhoods where they are located...it isn’t even close: 86 percent of eight-grade students at KIPP Academy scored at grade level in math this year, compared with 16 percent of students in the South Bronx.

Tough observes that KIPP students are doing much better, and concludes that KIPP is effective. What’s wrong with this argument?

First, that students selected into a KIPP lottery makes them different from than those who did not. It may be that their parents are more involved in their education, that they are having a particularly bad experience at their neighborhood school, or that their parents can no longer pay for private school. Whatever the reason, families selecting in, even if they are all poor and minority kids, are different by virtue of choosing a non-neighborhood school.

Lots of choice advocates will spar on this point, and argue that everyone wants a better choice for their children, so there is no selection problem. While rhetorically effective, anyone arguing that families that choose into a charter school are the same as those who don’t is simply wrong. Random assignment is the gold standard of causal inference in the natural and social sciences, and kids are not randomly assigned to KIPP lotteries. Saying that 80% of the kids are poor and 90% are African-American and Hispanic doesn't solve this problem. Even if KIPP kids had test scores identical to their neighborhood school peers, we still couldn't compare KIPP and neighborhood school kids who didn't opt in with any confidence because there is selection on "unobservables" - things like motivation and aspirations that are not measured by administrative datasets used to make these comparisons.

To get around this problem, KIPP has compared students' baseline performance with their performance after three years in a KIPP school. Jay Mathews summarizes the results of these analyses in an article earlier this year:

A KIPP analysis of the scores of about 1,400 students in 22 cities who have completed three years at KIPP show they went from the 34th percentile at the beginning of fifth grade to the 58th percentile at the end of the seventh grade in reading and from the 44th percentile to the 83rd percentile in math....Gains like that for that many disadvantaged children in one program have never happened before.

The argument is that students’ growth demonstrates the effect of the KIPP school. The problem with this approach is we have no way of knowing that these students wouldn’t have made similar gains anyway.

The best approach, and one that Mathematica has been contracted to carry out, is to compare students who entered the lottery and won with those who entered the lottery and lost. To keep this example as straightforward as possible, let’s assume that all lottery winners enroll and stay, and all lottery losers go to their neighborhood school, and there is no attrition in either case. We can now compare the achievement of these two groups and call the average difference the "treatment effect on the treated" – the effect of receiving a KIPP education on the students who received it. If the KIPP students are better off, we can say that KIPP "worked" for them.

The effectiveness of the research design I just proposed rests on two assumptions. First, we need to be sure that the lottery is actually a lottery, i.e. that we are not dealing with a "broken experiment." (For a good explanation of this problem, see the Cullen and Jacob paper I wrote about last week.) A second way in which the design I described falls apart is when there is "selective attrition." Selective attrition is the idea that people who choose to leave an experiment are different in one way or another. I’ll discuss these two issues later in the week.

For now, I'll conclude that we know that 1) KIPP kids perform better, on average, than their neighborhood school peers, 2) KIPP kids exhibit very large value-added gains on standardized tests. But we actually don’t know if KIPP kids are better off academically by virtue of attending KIPP than they would have been if KIPP didn't exist. There are certainly good reasons to believe that they are - i.e. they are in school substantially more - but the size of the "KIPP effect" is probably much smaller than we currently believe it to be.

It perplexes me that journalists continue to downplay these concerns. For example, in an article from a few years back, Jay Mathews wrote:

Whatever the academic or family characteristics of incoming KIPP students, they are clearly disadvantaged -- 82 percent of all KIPP students qualify for federal lunch subsidies -- and at KIPP have achieved gains in reading and mathematics far above those of other programs trying to help such children.

And Paul Tough wrote:

In some ways, the debate seems a trivial one — KIPP is clearly doing a great job of educating its students; do the incoming scores at a single school really matter?

I hope this post convinces you that these evaluation concerns are non-trivial. If we really want to know if these schools are working and how large their effects are (and for whom), we need to take these issues seriously. Perhaps Jay Mathews himself said it best when he wrote:

I understand why we education reporters try to make KIPP sound like more than it is. We are starved for good news about low-income schools. KIPP is an encouraging story, so we are tempted to gush rather than report. We don't ask all the questions we should.

Originally posted to eduwonkette on Tue Oct 16, 2007 at 07:23 PM PDT.

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  Well (0+ / 0-)

    Kids who go there seem to do better. And frankly they have to be doing something right. However, I also believe that it's not the "end all and the be all" of education for poor and disadvantaged minority students.

    •  charter schools (1+ / 0-)
      Recommended by:
      JanL

      extract from parents a commitment to the school or else.  They do a great sales job on things that public schools do too only without the support of these same parents.

      Parents gush about having to mark in a folder evidence of their child's work in school are the same as those who couldn't care less about the same tool from a public school.  "Wow, my kid will get to take a trip to Washington if his grades are good enough", but in a public school was, "Wow they won't include my kid on the trip to Washington because his grades aren't good enough, not fair!"  

      "Wow, my kid gets to call the teacher any time in the evening for help with home work" but don't repsond to "Wow, Mom my teacher is gone and got replaced with a new teacher."
      Cheating in Dallas
      http://www.dallasnews.com/...

      http://www.charterschoolpolicy.org/...
      http://www.sfgate.com/...

      Jay Mathews
      http://www.washingtonpost.com/...

      MATHEWS (1/17/06): Among the most important things to say, for instance, about that astounding jump in math and reading achievement from the 28th to the 74th percentile is that it is based not on an independent study but on standardized tests that KIPP teachers gave to their own students. KIPP schools are run by their mostly young principals, not by the KIPP Foundation, and the principals decide which tests to use and conduct the testing. The schools in the report gave the Stanford 9 or Stanford 10 tests or the Iowa Test of Basic Skills.
      These are popular tests used by many school districts to see how their students compare to a national sample of students, and are as good as you are going to get in the world of relatively inexpensive off-the-shelf tests. But this is still KIPP testing itself, and that should be kept in mind. There are several cases of public school principals changing standardized test papers to enhance their results when they have had an opportunity to do so.

      whoo, I bought a house in Texas!

      by TexMex on Tue Oct 16, 2007 at 08:59:22 PM PDT

      [ Parent ]

      •  found it on the Daily Howler (0+ / 0-)

        ****

        whoo, I bought a house in Texas!

        by TexMex on Tue Oct 16, 2007 at 09:02:50 PM PDT

        [ Parent ]

      •  There are lots of cases of principals in ordinary (1+ / 0-)
        Recommended by:
        TexMex

        public schools with low test scores faking them because while the are low stakes tests for the kids, they are high stakes tests of the principal that can lead to them losing their jobs or having their schools shut down.

        "Those who can make you believe absurdities can make you commit atrocities" -- Voltaire

        by ohwilleke on Tue Oct 16, 2007 at 09:05:37 PM PDT

        [ Parent ]

        •  part is shell game (0+ / 0-)

          because they appoint loser principals to poor schools and then send the troublemakers from other schools there just before the tests. That way the poor kids dont' get any better, the principal doesn't feel responsible for the scores cause he knows that the superintendent is sending him troubled students and shrugs his shoulders and (excuse me).........go off to fuck the coordinator who poses as a Christian woman to try to throw off the scent all us regular teachers have had in our nostrils.  Later on after holding all the aids in a closed room to "check the tests"  is suprised to be awarded educator of the year for the increased scores.  But but but but  the principal had a head injury, plays golf with all the big guys and is a "really funny guy with the jokes"  Irish and will be retiring any year now.  Been seen to emerge from his office with the same shade of lipstick his coordinator has on all over his face.  But these folk were Catholic and would leave on together on Ash Wednesday and return in the middle of the day with ashes on their foreheads.  
          This I have seen.

          whoo, I bought a house in Texas!

          by TexMex on Wed Oct 17, 2007 at 11:45:58 AM PDT

          [ Parent ]

  •  for sake of argument, let's suppose KIPP works (4+ / 0-)
    Recommended by:
    Mi Corazon, JanL, vickie feminist, slksfca

    and there is ample evidence available to question that supposition.

    What characteristics are different about KIPP that would warrant our attempting to replicate it?  Are they replicabl?

    What measures besides test scores, even value added scores, indicate the improvement?  For example, if we give student a task which requires analysis and application of previously learned knowledge and skills and does not provide a choice of one out of a multiplicity of possible answers, are the students able to apply the knowldge/skill in a real world context?

    KIPP has been around long enough that students should have cycled into higher levels of education which are not KIPP based.   What evidence if any is there of persistence of the gains supposedly made under the KIPP approach?

    All of these could be the subject of the kind of serious evaluative efforts to which as far as I know KIPP as not yet been subjected.

    There is some evidence from a number of KIPP locations of manipulation of the student body in order to be able to claim the great gains which get them all the publicity.  It is not a subject that I have followed with great detail, although since I am on good terms with Jay Mathews we have had a number of electronic discussion about KIPP in recent years - let's say we continue to agree to disagree about its value, and I'm glad to see that he acknowledges the possible overreaction of education writers.

    I look forward to your continued posting, and hope they get more visibility.  At least they will probably get one more chance in the Teacher Lounge on Saturday, courtesy of Rserven.

    peace.

    Those who can, do. Those who can do more, TEACH! If impeachment is off the table, so is democracy

    by teacherken on Tue Oct 16, 2007 at 07:41:16 PM PDT

  •  Not convinced by this post. (1+ / 0-)
    Recommended by:
    JanL

    There are quite a few school of choice options that aren't KIPP, and those students are likely to be a pretty good proxy for the parental involvement and motivation factor.

    For example, where I live in Denver, we have a KIPP charter school, Cole, that will be going out of business at the end of this year, despite exceptional academic improvement for the kids in that school (long story).  We also have a lot of kids from high poverty areas with similar demographics who go to schools of choice that involve even greater burdens of transportation and also require significant parent buy-in to a non-traditional program, like Knight Fundamental Academy (which is also a rigorous program that drives less committed families out).

    The latter schools do marginally better than par for the course.  The KIPP school does much better than par for the course.  This suggest that while parental involvement/motivation factor may have an impact that it is considerable smaller in magnitude than this post suggests, without offering any real evidence that the possible selection effect is indeed all that great.

    The magnitude of the impact is too great, and the anecdotal evidence of published descriptions of what classrooms of kids entering the program are like and what kind of relationship there is with the parents also don't bear out the notion that the families coming in and the kids coming in are that dramatically different from their peers.

    KIPP isn't the only program that shows success with poor kids.  In Colorado, there is one parochial school in metropolitan Denver that shows similar results, and there are a cluster of ordinary public schools in a district near Pueblo that achieves far in excess of what would be expects from a high poverty group of kids.  But, you are talking about maybe 6-7 schools in a state with 4.6 million people in it and more than 170 school districts, each with many schools, in an entire state that are achieving exceptionally good academic performance with very poor children.

    In contrast, there are probably 70 or more charter or private schools in the state targeted at low income urban children that are producing academic performance significantly above that of ordinary public schools, and quite a few, PS 1 in Denver, for example, have actually performed worse that ordinary public schools with comparable socio-economic profiles.

    Moreover, the fact that KIPP is achieving these results across in multi-school program, rather than just on a hit or miss basis rules out the possibility that one or two KIPP schools are just statistical flukes in the general pool of charter and private schools serving predominantly low income kids.

    The evidence that KIPP is an exceptionally good curriculum, and not merely an exceptionally good screening system, is overwhelming.  The magnitude of the gains and performance compared to past student performance and to comparable socio-economic status schools of choice/private schools is simply to great to deny that it works.

    The trouble is that this doesn't mean that it is a reproducible model.  KIPP calls for maniacal effort on the part of some very bright teachers who are willing to be extraordinary flexible in some ways and extraordinary doctrinaire in others.  It calls for many more hours of student and teacher work per school year than is customary in either public or private schools.  It calls for community buy in to the painful notion that academic success comes at the cost of cultural re-education of students (something shared by the parochial school I reference).

    KIPP shows, in short, that academic success for schools in poor urban neighborhoods is possible, but that achieving it in that context takes extraordinary effort in time and money compared to what is required to achieve similar results in middle class neighborhood where mediocre teaching efforts and only a modest commitment on the part of parents and teachers is sufficient to produce reasonably academically successful kids.

    In hard dollars and cents, funding that kind of education for every child requires funding on the order of two to three times as much per poor urban child, as per middle class suburban child.

    "Those who can make you believe absurdities can make you commit atrocities" -- Voltaire

    by ohwilleke on Tue Oct 16, 2007 at 09:01:59 PM PDT

    •  Key typo (0+ / 0-)

      "In contrast, there are probably 70 or more charter or private schools in the state targeted at low income urban children that are not producing academic performance significantly above that of ordinary public schools, and quite a few, PS 1 in Denver, for example, have actually performed worse that ordinary public schools with comparable socio-economic profiles."

      "Those who can make you believe absurdities can make you commit atrocities" -- Voltaire

      by ohwilleke on Tue Oct 16, 2007 at 09:10:53 PM PDT

      [ Parent ]

    •  KIPP teachers & students (1+ / 0-)
      Recommended by:
      ohwilleke

      I took several seminars last summer, many of which looked at a model of KIPP for younger students.  Turns out that what we (classroom teachers of primary students) have been carping about is essentially what KIPP does - the kids are in school for about 9 hours per day and teachers are on call 24/7/365.  The parents must "buy in" to the longer hours and the stringent discipline, and the class sizes are limited.  
      When we had Reading First in my district (big urban, 100% qualify for free lunch and breakfast) we had after-school tutoring that focused on the skills previously taught in class that day with time for homework and one-to-one tutoring.  Yes, intense, but it worked for those students whose parents supported the after-school tutoring. Those students who could not stay for it did not improve as much, but did show smaller gains on state tests and other progess monitoring we did every 6 weeks.  In addition, all the parents of my students had my home phone number so if they got stuck on a homework question they could call, which was surprisingly a good idea and one which I have continued since.
      Ergo - it costs way more, and more adult one-to-one time, with urban kids than it does for suburban students.  Considering many states still have property taxes as their funding method for schools, it's far too expensive without considerable help from the federal gov't., and we know how popular it is to ask for money for big urban schools.  

      Think what you are doing today. -Fred Rogers

      by JanL on Tue Oct 16, 2007 at 09:27:11 PM PDT

      [ Parent ]

      •  The only qibble I have with your analysis is the (0+ / 0-)

        notion that the source of funding would have to be the federal government.

        Utah is conservative as hell, and yet it found the money to insist on among the best student-to-teacher ratio in the nation for their elementary school kids (and was also the first state to raise a big stink about No Child Left Behind).  

        Sure, it helps politically to be in a state where large families are the norm, where support for large families is a community mantra, and where the state is homogenous enough that us v. them politics are muted.

        But, this is also a state where Big Government is not loved, is the most Republican state in the nation, and is the only state in the nation where President Bush has a positive approval rating.

        Likewise, even Republicans back major funding for Head Start, because it has such a good track record.

        If the public were convinced that the money spent on big urban schools actually produced results, and there are certainly all sort of expensive programs for big urban schools that don't, then I think that the will could be found to fund them.

        And, if a big additional commitment isn't for everyone, who cares?  If you can help a decent share of the kids improve a great deal, it is still better than helping no one.

        "Those who can make you believe absurdities can make you commit atrocities" -- Voltaire

        by ohwilleke on Tue Oct 16, 2007 at 09:43:10 PM PDT

        [ Parent ]

  •  KIPP (2+ / 0-)
    Recommended by:
    ohwilleke, JanL

    Eduwokette, I hope you are going to describe the KIPP program in detail, as it is worth pointing out how they achieve these gains.

    In essence, they lengthen the school day, even the school week to include Saturday morning, so they are working their students long and hard.  But, for the public system, many parts of this cannot be replicated without significant changes in the structure of the school day and the amount of resources to pay for extra staff time.  

    KIPP also does significant things in terms of behavior, like uniforms, active listening, attitudinal responses, posture in class, slogans, etc.  It's a whole program and needs to be described, because the truth is it does work for some students, but will definitely not work for all students:  it's too rigid.  Some kids respond well to a high intensity, high structure, high socialization, high behaviorist program.  Others do not.  How many kids leave KIPP in an average year?

    Plus, the kinds of gains they are making are the low hanging fruit of test score improvement.  Low order mostly, with all the extra drilling time.   Does the program improve creativity, critical thinking, compassion, collaborative group skills?  

    If we take inner city kids and drill and grill them and basically take the place of their non-existent family, then I am sure we can materially change their lives, and that is a central feature of KIPP.  But, where they end up after this kind of, for lack of a better word, "indoctrination" into learning, is a whole other question.

    Help new teachers to grow and love their work at www.newteachernetwork.net

    by Mi Corazon on Tue Oct 16, 2007 at 09:08:01 PM PDT

    •  You have to run to establish the pass. (0+ / 0-)

      I'm not a believer in the notion that drilling to establish the basics necessarily undermines creativity or critical thinking.  You can't do those things usefully until you have established basic skills.  I'm also inclined to say that the rather elaborate behavior engineering involved in those programs improves compassion and collaborative group skills.

      Indeed, a core premise of the KIPP program is that social skills and attitudes are central to academic success.

      But, I do fully agree that the extra time commitment and role of the school and teacher of going beyond the role of a traditional school and into the role of a traditional family, are both key elements and are elements that have impact precisely because they fill gaps present in high poverty urban environments.  Indeed, rather than making the committed parent argument, I think that there is quite a bit to be said for the notion that a program like KIPP can only work in situations where parents are unable to model standard English and "middle class" attitudes about how to behave in a classroom.

      Even if KIPP is an approach that works only for 10% or 20% or 35% or 50% of urban poor kids, it is successful enough that a program of this type should be available for almost all of them, because for those kids, at least, the impact is huge.  There are 45,000 or so kids in the Denver Public Schools out of sixty some thousand who fit the urban poor profile.  A program that could produce dramatic gains for even 4,500 of them would be well worth doing.

      My own children are in a "British primary" format school with almost the opposite characteristics of KIPP (highly unstructured with little homework) and are doing just fine.  But, they have learned a lot of what KIPP has to offer by osmosis, because they've lived all their lives in a the same home, with the same two adults who happen to get along reasonable well, with their material needs met, with a stay at home mom through their preschool years, and they have been lucky enough to have every single one of their parents and grandparents be people who happen to have graduate degrees.  They don't need to have active listening, attitudinal responses, posture, standard English grammar and vocabulary, or the like, forced upon them, because those things come naturally to them through childhood imitation.

      For most urban poor kids success at school means learning what amounts to a foreign language in terms of both grammar and vocubulary, even if the dialect they grew up with is still called English, and it also requires them to learn and choose to adopt what amounts to a foreign culture.  I'm not convinced that either standard academic English or the culture associated with it are inherently superior or necessary to advanced learning (any more than knowledge of Confucian philosophy was inherently necessary to run the bureacracies that Confucian philosophy exams were used as screening devices for in ancient China), but if you want to succeed on standardized tests and in the learning environment present in 99% of American academic settings you need to learn this language and adopt this culture.

      "Those who can make you believe absurdities can make you commit atrocities" -- Voltaire

      by ohwilleke on Tue Oct 16, 2007 at 09:33:32 PM PDT

      [ Parent ]

      •  And you know me well enough (0+ / 0-)

        that I will argue that marginal or even substantial increases on a standardized test does not constitute a quality education, by any means.

        That, in fact, as you point out, while the KIPP kids are fighting for their lives to learn what amounts to a foreign language, the middle and upper class kids waltz home to music lessons, educational camps, vacations to exotic places, internet access, the works.  

        As it turns out, in terms of brain development, while some elements of the KIPP program help to grow the brain, other parts of it are inimical to brain development--- whereas the middle class has full access to activities that provide the kind of enrichment response the brain so craves in order to grow.  KIPP kids get one level of skills and development;  the suburban kids get a substantially enriched and more desirable one.

        So KIPP kids scramble to catch up to where the suburban kids were, but when they get there, the train  for college and upwardly mobile professions is already out of the station.

        And while KIPP kids could now effectively do some of the lower to moderately skilled work required by corporate America, guess what?  Those jobs can be done for pennies on the dollar in India and China.

        So while KIPP may represent a success in comparison to typical urban situations, I'm just not sure that getting on that merry-go-round really constitutes giving them what they need in order to truly have a shot at the American dream.

        And again, all this is premised around the fact that only a certain percentage of kids will have the requisite buy-in, extra time and discipline to stick it out and make it work in the first place.

        Help new teachers to grow and love their work at www.newteachernetwork.net

        by Mi Corazon on Tue Oct 16, 2007 at 10:07:40 PM PDT

        [ Parent ]

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site