|
 |
|
| Back to Standard 1 |
|
Forward to Standard 3 |
|
|
|
|
| Back to Standard 1 |
|
Forward to Standard 3 |
The unit has an assessment system that collects and analyzes data on applicant qualifications, candidate and graduate performance, and unit operations to evaluate and improve the unit and its programs. (For spring 2002, the unit should have a well-developed plan for an assessment system that includes timelines and details about the system components.)
In keeping with the NCATE Transition Plan, the unit has developed an assessment plan that will be implemented over the next several years to improve the unit and each of the programs within the unit. While this assessment plan was developed in preparation for review by NCATE, it is a natural extension of the assessment procedures that were in place in each program. Grants from the Maryland State Department of Education have helped the Teacher Education Programs to develop performance assessments over the past five years. A workgroup was established last year to begin to articulate a system of assessment that would be seamlessly integrated with the department's conceptual framework. The present plan was reviewed in an Education Department meeting held in February 2002.
Due to the unique nature of each program within the unit, the implementation of the assessment plan will be program-specific; however, the program-specific elements are tied to the unit's conceptual framework as well as national standards from the NCATE-member specialty organizations (e.g., International Reading Association, National Council of Teachers of English).
As shown in the preconditions document (G.1) and documented in the response to Standard 1, the department has been collecting data from a variety of internal and external sources to monitor candidate performance. GPA, SAT, and NTE/PRAXIS scores are monitored regularly. Periodically, follow-up surveys have been sent to graduates of Teacher Education and Counseling programs. Candidate performance on national licensing exams has also been recorded and tracked.
For the initial certification programs at the undergraduate level, the assessment system has been performance-based and refined continuously over the past six years (2.1). It includes college-level admission criteria, sophomore review, strong collaboration between Loyola College clinical faculty and school-based mentors during internship, and an extensive portfolio review process based on EDOT (for elementary) or INTASC (for secondary) standards as a culminating event. This process is described in detail in the responses to Standards 1 and 3 and in the accompanying exhibits. The power of this process has been that it allows evaluation of each candidate's achievements based on nationally recognized standards and leads to revision of curriculum and assessment practices to achieve better alignment with those standards. While aggregated information has been used in an informal way, as this process is refined it will allow the standards in the portfolio to be tied in more closely to the candidate's entire experience and the aggregated data from the portfolio reviews will be used more systematically to locate weaknesses in the program. For example, a cursory review of portfolios has indicated a possible weakness with regard to the implementation of information technology and steps are being taken to investigate and remedy that, including increased modeling of technology in all unit courses starting with ED100 Introduction to Education. The assessment system that is planned will provide a structured process to feed information from the portfolio review and other data sources back into program improvement. These processes are now being expanded into the post-baccalaureate programs.
At the advanced level, the department has seven distinct programs: Special Education; Administration and Supervision; Reading; School Counseling, Educational Technology, Curriculum and Instruction, and Montessori Education. All of these programs include admission criteria and most conclude with a culminating or capstone experience (such as the internships in Administration and Supervision, Reading, School Counseling, and Educational Technology) . Each program will formalize an assessment plan to improve interim and/or mid-point evaluations, to tie culminating experiences to the unit conceptual framework and national standards, and to monitor results of the culminating experience to provide for program improvement.
While the department has collected a good deal of data on many of its programs and that information has been used to make positive, programmatic changes to ensure that candidates are well-prepared (G.1), the plan described here will take the work to the next level of assessment - systematic, standards-based, and technologically supported.
Most of the department's programs have advisory boards consisting of members of the community in the program's field. (G.5; 2.2). In the case of the undergraduate initial teacher education programs the PDS Steering Committees and other community members have been involved in program development, development of the assessment system, and implementation of the assessment system (2.3,4). The initial teacher education programs include a portfolio review following the full-year internship. Each portfolio is evaluated by a team that includes at least one external evaluator. In programs other than initial teacher education, the assessment plan includes using advisory boards as part of the process for developing program-specific evaluation and as part of the implementation of the plan.
Each of the department's programs is guided by the standards of nationally recognized specialty organizations as indicated in the Table on pages 12 and 13 of the introductory section of this report. Document 2.5 shows how the unit's conceptual framework is aligned with national and state standards as well. Programmatic assessments will be developed to capture a broad sense of candidates' achievement of these state and national standards. Based on the portfolio evaluation model from the initial teacher education program that requires candidates to collect and organize materials demonstrating their competence in each of the INTASC or EDOT standards, each program will be developing a culminating process based on the standards for its area. Other measures from both internal and external sources will continue to be used.
Given the diversity of programs in the department, one common set of assessment measures is not appropriate for the entire unit. However, each program's assessments will share the common characteristics outlined in the assessment plan. These include:
For undergraduates, admission decisions are made at the college level. Candidates, once admitted, are free to choose a major after their freshman year. Any candidate who has been accepted into the college and elects to major or minor in one of the department's programs is admitted. Decisions regarding advancement and, ultimately program completion are made later in the candidate's career as described below. At the graduate level, candidates are admitted to programs based primarily on their prior academic performance as measured by their undergraduate or graduate GPA (2.7). As of the current catalogue year, applicants must also provide evidence of passing the PRAXIS I examinations at or above the Maryland cutoff score in order to achieve full admission to a post-baccalaureate initial teacher preparation program. There has been a degree of flexibility in admission requirements for advanced programs in an effort to take factors outside of academic performance into account when evaluating the suitability of adult learners for each program. Admission criteria at all levels and for all programs will be reviewed as part of the implementation of the assessment plan and evaluations of the reliability, predictive validity, and fairness of these measures will be assessed. Adjustments to the admission criteria will be made based on the results of these studies and in consideration of the capacity of each program to prepare quality graduates.
Progress through the program of studies is monitored each semester by the candidate's academic advisor. The Records Office prepares a report for each academic advisor and GPA is checked to ensure that candidates remain above the required minimum GPA and, for graduate programs, that the candidate has not earned more than one grade of C or below.
Candidates are asked to evaluate their instructors and mentors each semester. In addition, undergraduates in the Teacher Preparation Programs have been asked to evaluate their programs on a regular basis, rating such elements as coursework, field experience, and internship (2.6).
As part of the unit assessment plan a system will be developed to monitor student progress relative to specific outcomes identified in the conceptual framework and required for compliance with many of the specialty organizations that have reviewed individual programs. Candidates will monitor their own progress relative to outcomes. Instructors will be asked to complete a simple evaluation sheet that will be developed during AY 2002-2003 to assess the achievement of each student relative to those outcomes that are addressed in each specific course.
Each program will determine an interim or midpoint (either dependent on completing particular courses or a certain number of courses) at which candidates will be evaluated. The nature of this evaluation will be determined by each program, based on the applicable specialty organization's standards and the unit's conceptual framework. An example of such a measure is the Sophomore Review process, discussed in the response to Standard 1, that is currently used for undergraduate Elementary Education majors (1.51). This midpoint review includes a check of the candidate's GPA, the evaluation of a writing sample, and an interview with a faculty member. Currently it serves a monitoring and advisory function rather than a decision-making one - this will be evaluated as part of the implementation of the assessment plan.
A midpoint review will be an important element in the overall assessment system since much of the latter half of a candidate's program in many areas consists of clinical work and/or advanced courses. Candidates who progress into that stage of an academic program are generally successful. It is important, therefore, to develop a meaningful review for the midpoint of each program to ensure that appropriate screening for potential achievement of the desired unit outcomes in each candidate.
Internship is a critical phase in many of the programs in the Education Department. Presently, completion of the required coursework and an acceptable GPA serve as the criteria for advancement into the internship. The assessment plan calls for including the ongoing and midpoint evaluation data, which are aligned with unit outcomes and external standards, as part of the decision process for admittance to internship. By using these multiple sources of information to help inform the decision to place a candidate in an internship, faculty will be able to determine appropriate placements better and to customize the experience to each candidate's specific strengths and weaknesses.
Currently all programs except the Curriculum & Instruction program have some form of culminating activity such as an internship. While these activities are generally aligned with specialty organization and/or state standards and the unit conceptual framework, the assessment plan requires each program not only to have in place a culminating activity but also to evaluate performance in that activity based on standards and unit outcomes with some form of exit assessment. In the initial teacher education program, candidates use their internship experience to develop a portfolio that is tied to appropriate standards. The portfolio forces the candidate to assess his/her competence relative to each standard. The portfolio is then reviewed by a team of evaluators and presented in a formal exit interview to the review team. Administration and Supervision (G.11) and Guidance and Counseling (1.40) also use a portfolio review process. The Reading Specialist Program requires practicum candidates to complete extensive case studies as the exit criteria (1.41). All of these evaluations will be reviewed and aligned with the unit outcomes and specialty standards as appropriate as part of the implementation of the assessment plan.
Periodically the department has conducted follow-up surveys of program completers. Results are reported in the preconditions report (G.68). Questions about how well the Loyola program prepared candidates for their career are asked and responses are gathered for reporting. Beginning last year, Federal Title II reporting protocols require the department to aggregate and report pass rates on the PRAXIS I and II for all program completers (1.52). Scores of program completers on other licensing examinations (ISLLC and NCSB) have also been gathered (G.69). One-year follow-ups are required of all programs in the assessment plan. The services of the Office of Career Planning and Placement may be available to assist the department in collecting and aggregating these data (2.8). Additional surveys will also be considered, such as extended phone interviews with samples of graduates from each program.
Similar follow-up will also be conducted after each completer has been out of his/her program for three years. Very often the full effects of a preparation program are not evident until a candidate has been working in his/her field for a number of years. Also, because professionals in education fields have a relatively high rate of leaving for other careers, these follow-ups will be used to learn about programs strengths and weaknesses and to help faculty to plan programs that will deal with long-term retention issues..
The reliability, predictive validity, and fairness of this system will be monitored as part of the ongoing analysis described below.
When the assessment plan is fully implemented, each program in the unit will have admission, ongoing, midpoint, internship, exit, and follow-up evaluations for each student. As these will be built into the programs as part of the requirements for the candidates, the data will be collected regularly. Additionally, follow-up surveys shortly after graduation and three years after graduation will help to provide for important feedback to the program improvement based on the success of departmental graduates. By combining these data with information from licensing exams, a comprehensive set of data will be collected regularly and fed back to the unit and the programs for a continuous process of improvement.
Each program coordinator will produce a comprehensive annual report on program activities and outcomes after the end of the spring semester. The format and content of this report will be developed during the 2002-2003 academic year, and will include:
These reports will be submitted to the department chair and reviewed by the Committee of Coordinators as part of the ongoing evaluation and planning for the unit.
The collection of data, as suggested by the wide range of measures discussed above, will be ongoing and will require the coordination of information across a number of offices at the college. As part of the first year of implementation of the assessment plan, determination will be made as to how best to structure the system for data collection, analysis and reporting. An administrative assistant has been employed by the department to work on such accreditation-related activities, but the possible need for an additional person to coordinate assessment activities will have to be evaluated. A graduate assistant or staff person who is familiar with data processing and statistical software will certainly be needed.
The offices of Admissions, Advising, Records, Institutional Research and Technology Services will all have a part in building a system that can collect, coordinate, and report the data gathered. The department will hold a series of meetings over the summer of 2002 that will determine who can provide key elements of data and how the data from a variety of sources can best be gathered so that a comprehensive reporting system that is accurate and easy to use will result. The goal is an information system that can provide, on a regular and consistent timetable, all data that will be needed for the compilation of the Annual Report from each program.
Each program's faculty, under the guidance of the program coordinator, will review the aggregated data as outlined above and use it to inform program revisions as appropriate. Such use of data has been employed, in a more informal way, by the Teacher Education Program for a number of years. Data will now be available from multiple sources to evaluate the effectiveness of courses, faculty members, adjunct instructors, and clinical experiences. Progress of students toward successful achievement of the unit's desired outcomes will be the standard of performance in all cases. The results of this work within the program will result in the development of the Annual Program Report described above.
Each summer, the Committee of Coordinators will review all annual reports and generate recommendations for curricular, programmatic, and departmental change. Important issues related to departmental personnel and resources, including hiring requests and operating budgets, will be part of this discussion that will have, at its source, the aggregated and multi-faceted data generated by the assessment system.
In one of the first departmental meetings each fall, the chair will report on the summer review of programs. Priorities will be discussed and referred to the Committee of Coordinators, which will set priorities for requests to be made in a formal report to the dean. Curricular changes and their implications for the entire department will also be discussed as a result of the summer review. In addition, needed changes in the assessment instruments, system and processes will be discussed and plans for revision will be made.
|
|
|
|
|
|
| Admission Criteria | The unit will form a task-force consisting of faculty and community members to explore the implications of changes in admissions criteria and to evaluate capacity for each program | The unit will pilot suggested changes to admission criteria based on the task-force's recommendations. |
The unit will monitor performance of candidates admitted under new policy. Results will be reported in annual reports and used to modify programs. |
The unit will evaluate the success of candidates compared with results of piloted admission criteria and implement changes in admission policy based on the results of the study. |
| Interim Assessment | Each program will develop a checklist based on national standards and/or conceptual framework. The checklist will be used: (a) for candidates to monitor their progress in completing the program standards; (b) for advisors to monitor the progress of the students; (c) for instructors to evaluate each candidate's progress on each standard. |
Each new candidate will be required to monitor his/her own progress based on the checklist. Each instructor will complete a checklist for each student in his/her classes. Advisors will monitor student progress. |
Results of candidate progress will be monitored and adjusted as needed. Cumulative results will be included in annual reports and used to modify programs. | Graduation of new candidates (who started in AY 04-05 or later) will be dependent on successful completion of the checklist. Programs will adjust curriculum and experiences based on results. |
| Midpoint Review | Based on task force input, program faculty will develop appropriate mid-point assessments for each program, using state, national, and/or conceptual framework outcomes |
Programs will determine the appropriate time to conduct midpoint evaluations, which, at a minimum, will include review of each candidate's checklist. Programs will implement the midpoint review and report results in annual report. |
Programs will use the newly developed midpoint evaluation process to formally review each candidate's progress and alert candidates to any difficulties. Aggregated results will be included in the annual report. | Results of midpoint reviews will provide feedback for program improvement as needed. |
| Advancement to Internship | Programs requiring internships will determine criteria beyond the currently used GPA and coursework completion standard that should be considered in the decision to advance candidates into the internship. | Programs will include additional criteria, including results of interim and midpoint reviews, into the procedure for advancing candidates to internship. | Results of new procedures will be incorporated into the annual report and changes to procedures will be made accordingly. | Advancement procedures will be monitored and adjusted as needed. |
| Exit Evaluation |
Each program that does not have an advisory board will form one consisting of members of the schools, school systems, candidates, recent graduates, and/or other appropriate members of the community. Each program that does not include a culminating activity will develop one, possibly in the form of an internship, a major project, or a research paper, with input from the advisory board. This culminating activity will be tied to the program's national standards and the unit's conceptual framework. Each program that already includes a culminating activity will refine the activity, if necessary, to align with the program's national standards and the unit's conceptual framework. |
New entrants into each program will be required to complete the newly developed culminating activities prior to program completion. Continuing students from each program will be recruited to pilot the new culminating activities. Each program will determine the nature of all exit criteria, which will include evaluation of the culminating activity and checklists used for interim evaluation. Advisory boards will be consulted as part of this process. |
Pilot data from candidates participating in the culminating activity will be used to refine the culminating activities for each program. Faculty members of the program, members of advisory boards and other appropriate community members will participate in formal exit evaluations of all candidates in programs in which this is feasible. Aggregated results of exit evaluations will be included in the annual reports of each program and used to adjust curriculum and other program aspects. |
Each program will fully implement a culminating activity that is tied to the exit evaluation and program completion. Programs will use the new exit criteria with all candidates who entered the program after the exit evaluation process was included in the College Catalogue. Data will be collected to compare success of exit criteria with success on entry and midpoint criteria. |
| One-Year Follow-up | The unit will develop a basic survey instrument for recent graduates. This survey will be administered to all recent graduates with assistance from the Office of Career Planning and Placement. Data from licensing examinations will be collected and aggregated. | Each program will develop a supplement to the unit-level survey to be administered to recent graduates. These supplements will be based on program-specific national standards. The unit-level survey with the supplement will be administered to all recent graduates. | Both unit level and program-specific surveys will be administered to all recent graduates. Results of these surveys will be used to evaluate program performance via the annual program report. | Both unit level and program-specific surveys will be refined and administered to all recent graduates. Continued use of these results will be built into the annual review process. |
| Three-year Follow-up | The unit will develop a survey instrument to be administered to all graduates who completed their programs three years earlier. This survey will focus on the unit conceptual framework and employment status of the graduate. | The unit survey will be administered to all graduates who completed their programs three years earlier. Results will be aggregated and included in the annual program report. | The unit survey will be administered to all graduates who completed their programs three years earlier. Results will be aggregated and included in the annual program report. | The unit survey will be administered to all graduates who completed their programs three years earlier. Results will be aggregated and included in the annual program report. |
|
|
|
|
| Back to Standard 1 |
|
Forward to Standard 3 |
Last Modified: March 13, 2002 based on the February 25, 2002
Institutional Report
Send comments about this web page to the Education Department
webmaster.