The Department of Education in April of this year, created two technical reviews to assess test development in the PARCC and Smart Balance Assessment Programs. (Delaware uses the Smart Balanced). The report was made public July 3rd. Essentially this is an internal review mid-process in its development. It's goal was to steer the second half of development towards its proper direction.
Here is what it found.
- Smart Balanced developed a high number of items for field testing this Spring: over 5000! it will used lessons learned in future test development. (Could also explain this year's drop in test scores, since answering these took time away better spent answering the questions used for the assessment).
- It was recommended that Smart Balance run its test past content experts to insure the answers were correct.
- They were questioned whether their list of 500 items was sufficient enough to train the field test question writers who would be the ones who would create appropriate questions for the assessment.
- They recommended improved training on knowing the Common Core standards be held for those writing field questions for the test.
On the English Language tests, the reviewers for the most part commended the texts being used for 2013, however a consortium within the consortium felt that there should be less emphasis on commissioned texts, and more on permissioned texts. Permissioned texts are published works for which a fee has been paid to obtain permission for their use. Commissioned texts are those contracted and written for a specified purpose, such as an assessment. Public domain texts are published works that are freely available to anyone. Apparently the commissioned texts were a little whacky.
Next year's field test will include 50 percent permissioned, 20 percent public domain, and 30 percent commissioned texts. (The commissioned tests are where they make their profit.)
In Math, reviewers recommended the assessment measures student performance at the “cluster” level of the mathematics Common Core Standards, so that the assessment system measures key concepts in the content standards at each grade. In the Common Core State Standards, the standards are organized by “clusters,” which are groups of related standards (for example, in grade 4, there are five clusters: operations and algebraic thinking; number and operations in base ten; number and operations, fractions; measurement and data; and geometry).
As a next step, the Technical Review found that Smarter Balanced should focus greater attention on the clarity of developing its items to ensure the alignment of items to college- and career-ready standards.
The Technical Reviewers highlighted the need for high-quality, authentic texts. In response, the plans for item development for the next field test increases the use of public domain or permissioned texts.
(Next step should be the elimination of commissioned questions entirely)...
The review panel noted that accessibilities for the disabled and those who native tongue was not English, were implemented from the ground up. However they recommended that the tests be analyzed by experts of both disabilities, to further enhance the approachability to the tests. They suggested that more lab testing on those with disabilities be done to allow the tests to accurately determine what students know, but have trouble expressing.
Upon research and planning, reviewers questioned whether having only just one informational text and one literary text, was a sufficient and adequate opportunity to determine if the entire scope of the Common Core Standards was being met. They recommend this question receive debate among the member states, educational experts, and professional teachers to determine if this could be so.
Reviewers also questioned whether the algorithm of the assessment (known by the letters CAT) was really accurate in determining if a student truly grasped the broad scope of Common Core materials. They questioned the validity of "being told it was so", by the entity that created, maintained, and sold the item.. They recommended independent research by appropriate educational academics to determine whether or not the algorithm was indeed legitimate.
The researchers recommended that Smart Balance also get independent confirmation on their plan to combine the algorithm and performance tasks into one score, and whether that one score properly would indeed reflect the claim of student achievement at the level which Smart Balance said one was purported to be.
The researchers recommended that Smarter Balance quickly "set in stone" its currently amorphous process and structure for setting standards and conduct pilots of its process on a small scale. It felt the "standards" were too loose and fluctuated from assessment to assessment.
Overall, essentially a passing grade was given for "effort" but with the small future problem noted:
- it was observed that no one yet knows if this is an adequate measure of whether a year's worth of Common Core Standards were absorbed by the student.