Arne Duncan has been listening to us, I guess, about the limitations of the fill-in-the-bubble test, and he has an answer:
New York Times: U.S. Asks Educators to Reinvent Student Tests, and How They Are Given
Over the next four years, two groups of states, 44 in all, will get $330 million to work with hundreds of university professors and testing experts to design a series of new assessments that officials say will look very different from those in use today.
The new tests, which Secretary of Education Arne Duncan described in a speech in Virginia on Thursday, are to be ready for the 2014-15 school year.
They will be computer-based, Mr. Duncan said, and will measure higher-order skills ignored by the multiple-choice exams used in nearly every state, including students’ ability to read complex texts, synthesize information and do research projects.
“The use of smarter technology in assessments,” Mr. Duncan said, “makes it possible to assess students by asking them to design products of experiments, to manipulate parameters, run tests and record data.”
Because the new tests will be computerized and will be administered several times throughout the school year, they are expected to provide faster feedback to teachers than the current tests about what students are learning and what might need to be retaught.
These tests do sound more promising. In fact, I've said many times that one of the most valuable things anyone could do in education would be to design better assessments, ones that better reflected what we actually want the kids to be able to do when school is done. But, the press releases leave me very skeptical that this effort will be it. The timeframe is so aggressive - I assume artificially so for political purposes - that I am very concerned that it can meet its goals.
$330 million is a lot to spend on two grants for a totally speculative software venture to be launched in classrooms across the nation in just three years. A project that size - assuming the money is spent well - can't be managed by a group small enough to fit in a room and talk. What they've done is split it into two groups of consortiums with scores of states in each, and set them to compete against each other. My skepticism stands.
If I'd set this up, I'd divide it into smaller phases. A grant for the design of the functions, to write the spec. Then perhaps multiple competing grants to implement software to that spec. And, I wouldn't plan to deploy it to every child in the nation at the same time, but rather start with a handful of indicator schools, and try it out for a year or two on a small scale. The idea that 26 states can get together and build something like this, so outside of their core expertise is somewhat hard to swallow. It'd be like issuing a $170 million grant to 26 states to build a Mars probe in three years. (That's about the right budget, point of note.)
I dug around to find the budget for the grant to the 26-state consortium including Florida, Massachusetts, Illinois, New York, and California.
The Partnership is requesting a total of $149,994,386 in Level 1 budget modules over the four-year grant period to support this bold new assessment system. The Partnership also is requesting a total of $19,995,886 in Level 2 budget modules for a total budget request of $169,990,272 for the entire project.
If you click through, you'll find a helpful summary of the budget... fitting on a single page. Line items like
Assessment Design and Development ... $80,487,463
I looked, and I was unable to find any more detailed breakdown of the one page budget.
The full proposal is here:
http://www.fldoe.org/...
At first I was alarmed: 40 pages of children riding ponies and unicorns playing together in a glorious field of rainbow lilies thanks to the head education honcho in each of the 26 states, with full contact information. (The unicorns ensure the lily field is accessible to all students.) Flowery prose about how all our children will be ready for college because of these grand new (unspecified so far) assessments that will give teachers, parents, and students real-time feedback on their mastery of complex problem solving.
If you can get as far as page 45 or so, there start to be some concrete examples of what their assessment would include. It's ambitious. They want kids to give speeches which the system will record. They want the kids to write essays into the system multiple times a year, which will be part of the permanent portfolio, as well as scored at that time. They envision asking students to find themes and other literary elements not just from written texts, but also from audio or video renditions. They want kids to highlight the relevant sections, not just pick from a multiple choice list. They are projecting a system that involves quite a bit of free response answers - which I applaud, but I wonder if they have a plan to score.
[Reading further, I get a chart showing me that 100% of questions for grades 3-5 will be human scored, with the percentage scored by computer increasing as students get older. Now I'm wondering how much money this will cost to score these tests, who will produce that money, and how they're going to turn them around in two weeks as they promise.]
They even suggest it "might require students to conduct a research project to gather and synthesize information from multiple sources." Coool! But, how the heck are they going to do that? I want them to be able to test that - and for kids to be able to do it! - ... but if they're starting from scratch, how the heck are they going to build a system that can do that and deploy it across the nation in three years? You could spend three years on that component alone.
(We're going to have to spend a lot of class time just teaching the kids how to use the software. Will they pay for an extra week or two of school each year for that?)
Some interesting ideas about math, including an example of having the student respond to a question by drawing a graph. Sweet. Demonstrates true understanding, easy to build and for a computer to score. Good technology!
They plan to train teachers in the scoring, even teachers who aren't official scorers, and teachers will be able to access their students' responses so that they can assess them themselves, without waiting for turnaround. That would be valuable. If teachers can also access previous work on the system by that student, you can see that this could be extremely useful.
It's envisioned that these tests will be used in part at least to assess teachers. I'm left wondering how specific growth in a high school student's writing, for example, can be assigned to just one teacher. Does the history teacher or the biology teacher get any credit for sharpening up essays? They promise to issue a grant to study it.
from the RFP:
(A)(7) Technology Approach (up to 10 points)
(Um, Education Department dudes? Seems odd to only award 10 points for their prose discussing the foundation of the system, without which the whole rest of the response is just unicorn dust... Meanwhile, "project management" is worth 30 points.)
They plan to require open source... this is promising. But if they have a specific vendor in mind or believe that the technology they need is already available, they don't share that. Still, it's likely they have someone in their pocket.
The project management will be by Achieve Inc. I couldn't find much about them, but based on their staff, they appear to be more of a think-tank and policy incubator for educators and politicians without a lot of technology expertise specifically on staff. They have been working on the national standards project, itself a huge effort and a large part of making an project like this one useful in multiple states.
Most of the proposal is high-level goals. Now, goals are good. And largely I agree with the goals they're suggesting. I just have serious concerns that they can actually produce software that meets those goals and is fully tested and meets all the accessibility goals and has all the teachers trained to work with it by summer of 2014.
And then there's infrastructure. Our school district is far from having enough computers to test even one grade level at a time, and, as a rural school, we'll be lucky if we manage to get broadband by 2014. (Our district does have a grant application pending with the federal e-rate program; if it is approved, and all goes well, and we can shoehorn our matching moneys out of our budget, we might have it by late 2012. But not all schools are even as far along as we are.)
(DH jokes in the background that this is an enormous windfall for Microsoft, just to buy enough copies of Windows.)
If you're still reading by page 232, they address it.
To help Partnership states support schools and local districts in the transition to computer-based testing, the Technology Design Committee will work with Partnership states to develop online system specifications and minimum school and district technology requirements by the 2011–12 school year, subject to the specific assessment framework that the Partnership and the chosen assessment vendor(s) will develop during the 2010–11 school year.
That's pretty much all they say - they acknowledge it's a problem and promise to send out a plan of unspecified cost. They didn't even WAG it.
The 2010-2011 school year has already started. 2011-2012 will be tentatively budgeted by March 2011, roughly 6 months from now. But, if the feds are prepared to send money and specs, buying computers is something schools can do quickly. They also need to include money for techs and support to build the networks and do the installs. Older schools will either need wireless (if they can get good reception through the walls) or they may need some substantial rewiring for both electrical and data. It's a good use for ARRA type money, and the computers have utility beyond testing. But, is Congress prepared to let this money flow to schools next year?
I applaud the intent of the project. That the American public wants some sort of measuring stick is clear, and this wants to be a very sophisticated one that provides a lot of useful feedback besides a cut score. There are some powerful ideas in here. But, I worry that the timeline will crush the ideals, and the size of the two enormous state consortiums will make it impossible for either of them to be nimble enough to innovate. I don't want this to be a giveaway to big corporations and I don't want it to end up as another impossible underfunded burden at the school level.
Who is watching this grant at the Department of Education, and do they have the expertise and clout to keep it on goal rather than on timeline? Is Achieve Inc one of the good guys, and can they deliver what they promise? And, if it is successful, how can we ensure Congress will follow through and provide the money to support this ambitious program if it is deployed?