Background and Intent of the Tool:
This online computer-based decision making tool was designed by CCSSO’s Computer-Based subgroup of the Technical Issues in Large Scale Assessment (TILSA) committee during 2006 with the assistance of Pacific Metrics. State testing directors and assessment personnel with experience in computer-based testing provided guidance and input to the tool’s content.

The tool is intended to assist state department of education personnel in making decisions and in implementing plans to move from a paper and pencil testing environment to a computer-based testing environment.

Directions for Use
The tool is designed to be simple and convenient to use.

Users must first register and create a log in and password. After that, the user should go into the tool and work through the sections in order. These sections are:

1. Starting Points
2. Goals
3. Decision Points:
Players and the Environment
Assessment and Psychometrics
Technology
4. Risk Analysis
5. Cost/Benefit Analysis
6. Implementation Plan

It is not necessary to complete the entire content of the tool in one sitting. Nor is it necessary to complete an entire section. The program will save the user’s entries as long as the same log in is used each time. The user may also go back and change or update their responses as appropriate.

At the completion of each section (1-5) a summary report is generated based on the user’s input. That report can be printed out or accessed online as needed.

Section 6: Implementation Plan can be completed online, or alternatively printed out and completed by hand or used as a guide to develop a comprehensive implementation plan for the user’s particular situation.

Annotated Bibliography and Literature Review
CCSSO also conducted a literature review of documents and studies related to computer based testing. The annotated bibliographies are available in .pdfs by clicking on the appropriate link(s) below; to access an entire document from the web, click on the hyperlink (where available).

Literature Review

1:ARTICLES AND REPORTS ON DECISION CONSIDERATIONS

Bennett, R.E. (2002). Inexorable and inevitable: The continuing story of technology and assessment. Journal of Technology, Learning, and Assessment, ı(ı). Available from http://www.jtla.org.

Editors, Education Week on the Web (2003), Legal Twists, Digital Turns, 2003 Editorial Projects in Education, Vol. 22, number 35, page 11-14,16. Available online at http://counts.edweek.org/sreports/tc03/article.Cfm?slug=35impact.h22

Editors, Education Week on the Web (2005), Tech’s Answer to Testing, 2003 Editorial Projects in Education, Vol. 22, number 35, page 8,10. Available online at http://counts.edweek.org/sreports/tc03/article.cfm:slug=35exec.h22

Row, Neil C., (2005), Cheating Online Student Assessment: Beyond Plagiarism

Trotter, Andres, (2005), Published May 23, 2001. Testing Computerized Exams. Education Week on the Web.

Wall, Janet E., (2000) Technology-Delivered Assessment: Diamonds or Rocks? ERIC/CASS Digest. ERIC Identifier: ED446325, ERIC Clearinghouse on Counseling and Student Services Greensboro, NC.

2. ARTICLES AND REPORTS ON DESIGN CONSIDERATIONS

CTB/McGraw-Hill, April, 2003. Computer-Based Testing – Issues and Considerations.

CTB/McGraw-Hill, April 11, 2003, The Computer-Based or Online Administration of paper-Pencil Tests, Journal of Technology, Learning, and Assessment, 2(1). Available from http://www.jtla.org

Quenemoen, R. F., & Lehr, C. A. (2002). Access to computer-based testing for students with disabilities (Synthesis Report 45). Minneapolis, MN: University of Minnesota, Thompson, S. J., Thurlow, M. L., National Center on Educational Outcomes. Retrieved [January, 2006] from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Synthesis45.html

(RAND Education)Hamilton, Laura S., Klein, Stephen P., and Lorie, William (2000), National Science Foundation, Division of Elementary, Secondary and Informal Education grant ESI-9813981. http://www.rand.org/pubs/issue_papers/IP196/

Stover, Del, (2005), Computer-Graded Essays Gaining a Foothold, 2005 National School Board News, National School Boards Association

Thompson, S. J., Thurlow, M. L., Quenemoen, R. F., & Lehr, C. A. (2002). Access to computer-based testing for students with disabilities (Synthesis Report 45). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved January, 2006 from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Synthesis45.html

U.S. Department of Education, National Center for Education Statistics. NAEP Validity Studies: Implications of Electronic Technology for the NAEP Assessment. NCES 2003–16, by Richard P. Durán. Project Officer: Patricia Dabbs. Washington, DC: 2003

3. ARTICLES AND REPORTS ON RESEARCH CONSIDERATIONS: PAPER AND PENCIL VS. ONLINE

Amoroso, Donald L., (2004). Appalachian State University, Boone, North Carolina. Use of Online Assessment Tools to Enhance Student Performance in Large Classes. Proc ISECON 2004, v21 (Newport): §3142 (refereed) © EDSIG, pages 1-8.

Brown , Carol J., (2002), Online Classroom Assessment Project Evaluation Report #1, Prepared for the Bill & Melinda Gates Foundation.

Harcourt Assessment, Inc., (2004), Wang, Shudong, Young, Michael J. , and Brooks, Thomas E., Harcourt Technical Report: SDRT 4/SDMT 4 Administration Mode Comparability Study/Online or Paper: Does Delivery Affect Results?. Available from: http://www.harcourtassessment.com

Loggia, J., Glasnapp, D. R., Yang, X., & Poggio, A. J. (2005). A comparative evaluation of score results from computerized and paper and pencil mathematics testing in a large scale state assessment program. Journal of Technology, Learning, and Assessment, 3(6). Available from http://www.jtla.org

Pearson Educational Measurement., Paek, Pamela, (2005), Recent Trends in Comparability Studies Research Report. Available: http://www.pearsonedmeasurement.com/research/research.htm

Poggio, J., Glasnapp, D. R., Yang, X., & Poggio, A. J. (2005). A Comparative Evaluation of Score Results from Computerized and Paper and Pencil Mathematics Testing in a Large Scale State Assessment Program. Journal of Technology, Learning, and Assessment, 3(6). Available from http://www.jtla.org

Pommerich, M. (2004). Developing computerized versions of paper-and-pencil tests: Mode effects for passage- based tests. Journal of Technology, Learning, and Assessment,2(6).Available from http://www.jtla.org

Wresch, W. (1984) The Computer in Composition Instruction: A Writer’s Tool. Urbana, IL, National Council of Teachers of English.

4. ARTICLES AND REPORTS ON RESEARCH CONSIDERATIONS: SCORING

Zhang, Yuli (Lilly), Powers, Donald E., Wright, Wendi, & Morgan, Rick. Applying the Online (OSN) to Advanced Placement Program ® (AP©) Tests. Educational Testing Service (ETS), April, 2003.

5. ARTICLES AND REPORTS ON RESEARCH CONSIDERATIONS: EFFECTS OF STUDENT FAMILIARITY WITH TECHNOLOGY

Educational Testing Service (ETS), National Center for Education Statistics. (2003), Sandene. Brent., Bennett, Randy., Kaplan, Bruce., and Braswell, James. Online Assessment in Writing.

National Board on Education Testing Public Policy, ), Russell, Michael, (1997), Boston College Massachusetts, Testing Writing on Computers: An Experiment Comparing Student Performance on Tests Conducted via Computer and via Paper-and-Pencil, Volume 5 Number 3 , Available: http://epaa.asu.edu.

O’Dwyer, L.M., Russell, M., Bebell, D., & Tucker-Seeley, K.R. (2005). Examining the Relationship Between Home and School Computer Use and Students’ English/Language Arts Test Scores. Journal of Technology, Learning and Assessment, 3(3). Available from http://www.jtla.org

Russell, Michael., (1997), Testing On Computers: A Follow-up Study Comparing Performance On Computer and On Paper, Volume 7, Number 21. Education Policy Analysis Archives Available: http://epaa.asu.edu.

6. ARTICLES AND REPORTS ON USE OF COMPUTERS IN INSTRUCTION

(ERIC) Kerka, Sandra., Wonacott, Michael E., Grossman, Cheryl., and Judy Wagner. (2000) Clearing House on Adult, Career and Vocational Education, No. Assessing Learners Online – Online Assessment: Principles and Practices, ED-99-CO-0013.

Goldberg, A., Russell, M., & Cook, A. (2003). The effect of computers on student writing: A meta-analysis of studies from 1992 to 2002. Journal of Technology, Learning, and Assessment, 2(1). Available from http://www.jtla.org

Liang, Xin & Kim Creasy (2004). Classroom assessment in web-based instructional environment: instructors’ experience. Practical Assessment, Research & Evaluation, 9(7). Retrieved August 12, 2005 from http://PAREonline.net /getvn.asp?v=9&n=7