A report by a Tennessee state investigative office says that testing vendor Questar Assessment Inc. and the state’s department of education both share blame for a series of delays and disruptions that dogged the administration of an online education assessment last year.

The performance audit by the Tennessee state comptroller’s office found that the testing company made unapproved changes to its software that contributed to the woes and lacked customer support when problems emerged. It said the state was not sufficiently clear in its expectations and oversight.

Both Questar and the education department took issue with some of the audit’s conclusions, and the state agency said many of the proposed fixes to testing procedures have already been made.

Tennessee state legislative leaders had requested that the comptroller’s office take on the review.

The comptroller also made a broader point about a series of sweeping changes Tennessee has made to its online testing system in relatively short succession, some of them pushed by elected officials. Those steps included the passage of state legislation that delayed that state’s participation in common core-aligned PARCC tests and led to the development of a new test; the state signing a deal and later ending it with another vendor, Measurement Inc.; and the approval of a deal with Questar.

“All stakeholders recognize the critical need for the department and its vendor to sustain continuity of student testing from year to year,” the report says. “However, department management should avoid being forced to implement assessment processes without allowing for adequate time to respond to and resolve potential issues with assessment vendors.”

To date, Tennessee has moved forward with “large-scale procurements involving millions of dollars under intense time constraints,” the report says, and therefore must try to “mitigate any risks to continuity and student experiences to the fullest extent possible.

The comptroller’s report also includes the results of a survey of teachers across the state, many of whom vented their frustrations about the testing disruptions and urged the state to end the upheaval.

“I really don’t understand why so many other states are able to successfully administer online testing, and Tennessee continues to have so many issues,” said one teacher surveyed.

“I DON’T think we should abandon online testing, I just think we need better leadership to implement this kind of testing and better  partnerships with stronger vendors.

In fact, many states have endured online testing disruptions. But the Tennessee comptroller’s report found fault with the department of education and Questar on a number of grounds:

• Questar made an unauthorized change to text-to-speech software within a platform called Nextera without formally notifying the department, which the auditor said contributed to the online testing disruptions;
• The state did not put enough sufficient, detailed information in its work plan agreement with Questar, a shortcoming that made it less effective in making sure the vendor met its deadlines;
• Questar did not sufficiently staff customer support, which resulted in school officials experiencing testing problems having to wait and abandoning calls (see the chart provided in the state’s report, below), and
• The department did not evaluate whether the vendor had adequate internal controls over processing and storing the agency’s data and whether it was ready to properly administer testing through a software-as-a-service application.

The state’s department of education issued detailed responses to the findings, in some cases agreeing with them, while disputing other conclusions. It also laid out specific procedures it had put in place to strengthen oversight of testing vendors and make sure assessments run smoothly.

The department argued that  its work plan was in fact effective in spelling out requirements for the “incredible number” of tasks associated with statewide assessment, and in holding Questar to account for $4 million in liquidated damages.

In response to the auditor’s finding that Questar made an unauthorized change in the testing software, department officials said they agreed with the need for “strong internal controls and communication” with the vendor. But agency officials believed they had many safeguards in place, and that “appropriate oversight is thwarted when a vendor departs from their own documented processes as was apparent in this situation.”

Questar is a wholly-owned, independently-operated subsidiary of Educational Testing Service.

In a statement, Tennessee Commissioner of Education Candice McQueen added that the state has already made fixes to many of the concerns identified by the comptroller. She also noted that a statewide test administration in the fall had gone smoothly.

“We have learned a lot from this process, and in the last seven months, we have already implemented and are in the process of implementing all of the improvements suggested by the comptroller,” McQueen said. “There was nothing new in this report. This audit process identified the same areas the department had already identified and addressed, such as developing additional processes to better manage our testing vendors, adding more requirements into our testing contracts, and improving customer service.”

 

 

 

 

 

 

 

 

 

 

Questar officials said in a statement in response to the auditor’s report that they hired an outside company to review its processes after the problems emerged in Tennessee. It also said it did not agree with several of the Tennessee comptroller’s findings.

(In a follow-up e-mail to EdWeek Market Brief, for instance, a company spokeswoman said that the testing glitches were not affected by the text-to-speech technology, but rather by “code interaction with a database component which also served text-to-speech.”)

The company conceded it did not have adequate capacity to handle the flood of calls that came in from confused district officials during the testing breakdowns. Questar says it has overhauled its practices for responding since then.

“We are not standing still. Questar Assessment is committed to continually advancing our processes, technology, and security,” Questar Assessment Chief Operating Officer Brad Baumgartner said. “We look forward to serving Tennessee teachers and students this spring with the best testing experience possible.”

Follow EdWeek Market Brief on Twitter @EdMarketBrief or connect with us on LinkedIn.

---

See also:

• Tennessee, Measurement Incorporated Tussle Over Blame for Testing Woes
• Online State Testing: Mostly Smooth, With One Glaring Exception
• Common-Core Testing Contracts Favor Big Vendors