第四次TIMSS 2003 NRC自由反應評分系統研討會會議報告

方泰山,國立台灣師範大學化學系教授兼科學教育中心主任

 

緣起

4th TIMSS-2003 NRC會議主要目標是“Field Test”評分方法之研討會,原計畫是由本研究小組“竹師院”張美玉教授代表,臨時因故,由本人「實務組召集人」方泰山代表自然組出席,另已定本中心洪有情則代表數學組出席。

 

過程

三月十六日深夜,1100由台北搭長榮航空出發,過境曼谷,於三月十七日當地(晚台北7小時)上午1030抵阿姆斯特丹,,再搭下午100 KLM 航空City Hopper200抵比利時布魯塞爾,隨時轉搭地下鐵至布魯塞爾北站,再搭火車至80公里外的西北方向Ghent (根特,Gent),由計程車送到第四次NRC會議地點Sofitel旅館,已是三月十七日下午430,前後花掉整個一天的旅程。

 

當晚630準時參加古色古香城堡地下室的歡迎酒會,預計參加團54個及承辦單位之各個單位主席,人員共計75人,約9.5成參加,相當熱鬧。我國二位代表週旋於各團之間,互相認識,並談及各國之科學教育問題。有幸碰到美國原ETS課程專家,現服務K-12顧問公司,談到很多國家正效法美國的K-12統整課程,如果有做好配套措施的話,沒她認為注定要失敗,必定會降低學生的科教水準,因為(1)學生認知過程,無法趕上,(2)教師無法應付。希望各國能及時煞車。另一,這次亦有“Palestine”及“Israel”同台參與。由於美國已決定支持“Palestine”建國,該國代表亦相當有自信,且非常Active參與。洪教授與本人一直待至930才回旅館。

 

三月十八日上午900TIMSS 2003執行主任Hans主持開幕典禮,地主Flemish社群教育部代表致詞歡迎各會員來此,致贈大家一本“Flemish”之“教育指標”。Mullis報告第三屆NRC會議後TIMSS目前進度。主要是ISC完成了:

 

l        Fiela Test之資料:包括“數學測驗題”“科學測驗題”問卷與各手冊,“自由評分訓練”手冊。(這些是由英語系參加國,美國、奧大利、加、英、迦納、紐西蘭和新加坡)

l        繼續“抽樣”之任務:NRC開始轉譯之工作。

 

說明本次會議之任務:

 

l        回顧Field-Test之各操作程序與其準備工作。

l        Field-Test之數據“處理過程”(前、進行中,與後)各項動作。

l        分發各國“評分手冊”與“訓練手冊”。

 

三月十八、十九日      M4 M8 之部分測驗題評分之訓練(如附件一)

三月廿、廿一日  S4 S8  之部分測驗題評分之訓練(如附件一)

三月廿二日主辦單位安排了一個上午的時間遊覽五十公里外的Bruges古城。Flemish之藝術、巧古力、Lace及教堂、古堡,就是這個西歐古老融合民族的特色。

會議議程如下表:

 

AGENDA

TIMSS 2003

Fourth National Research Coordinators’ (NRC) Meeting

Ghent, Belgium

March 17-22, 2002

 

Day/Date

Time

Topic

Presenter

Sun. March 17

7:30 p.m.

Welcome Reception

Sofitel Gent Belfort Hotel

 

Mon. March 18

9:00 a.m.

Welcome and Introduction

Representative of Ministry of Education, Ministry of the Flemish Community

H. Wagemaker, E. Owen

 

 

Progress Report, Schedule

Overview Constructed-Response-Scoring

M. Martin, I. Mullis

 

10:30 a.m.

Break

 

 

10:45 a.m.

Update on Sampling, Survey Operations, and Data Processing

E. Gonzalez, M. Joncas,   O. Neuschmidt

 

12:15 p.m.

Lunch

 

 

1:45 p.m.

Training for Scoring Constructed Response Items: Mathematics 4th Grade

R. Garden, G. Ruddock,   C. Jones

 

3:15 p.m.

Break

 

 

3:30 p.m.

Training for Scoring Constructed Response Items: Mathematics 4th Grade (cont.)

 

 

5:00 p.m.

Adjourn

 

 

7:00 p.m.

Group Dinner at ‘het Pand’

 

Tues. March 19

9:00 a.m.

Training for Scoring Constructed Response Items: Mathematics 4th Grade (cont.)

R. Garden, G. Ruddock,   C. Jones

 

10:30 a.m.

Break

 

 

10:45 a.m.

Training for Scoring Constructed Response Items: Mathematics 8th Grade

 

 

12:15 p.m.

Lunch

 

 

1:45 p.m.

Training for Scoring Constructed Response Items: Mathematics 8th Grade (cont.)

 

 

3:15 p.m.

Break

 

 

3:30 p.m.

Training for Scoring Constructed Response Items: Mathematics 8th Grade  (cont.)

 

 

5:00 p.m.

Adjourn

 

 

Day/Date

Time

Topic

Presenter

Wed.  March 20

9:00 a.m.

Training for Scoring Constructed Response Items: Science 4th Grade

T. Smith, S. Lie,             C. O’Sullivan

 

10:30 a.m.

Break

 

 

10:45 a.m.

Training for Scoring Constructed Response Items: Science 4th Grade (cont.)

 

 

12:15 p.m.

Lunch

 

 

1:45 p.m.

Training for Scoring Constructed Response Items: Science 4th Grade (cont.)

 

 

3:15 p.m.

Break

 

 

3:30 p.m.

Training for Scoring Constructed Response Items: Science 4th Grade (cont.)

 

 

5:00 p.m.

Adjourn

 

Day/Date

Time

Topic

Presenter

Thurs. March 21

9:00 a.m.

Training for Scoring Constructed Response Items: Science 8th Grade

T. Smith, S. Lie,             C. O’Sullivan

 

10:30 a.m.

Break

 

 

10:45 a.m.

Training for Scoring Constructed Response Items: Science 8th Grade (cont.)

 

 

12:15 p.m.

Lunch

 

 

1:45 p.m.

Training for Scoring Constructed Response Items: Science 8th Grade (cont.)

 

 

3:15 p.m.

Break

 

 

3:30 p.m.

Training for Scoring Constructed Response Items: Science 8th Grade (cont.)

 

 

5:00 p.m.

Adjourn

 

Fri. March 22

8:00 a.m.

Half-day Excursion to Bruges

 

 

***Consultations with staff from the DPC, Statistics Canada, and the ISC will be held Monday through Thursday***

 

 


結果與建議

l          維京評分系統(Constructed Response,建構反應)評分系統

 

第一個碼

l          2類型的CR項目(分數碼)

(1)   2分:(外延反應評分)

2分:完整無誤

1分:部分對

(2)   1分:(問答)

(3)   0分:7-9

 

2

 

1

第二個碼

l          診斷訊息碼

0-5:表出現之頻次類次,配合參數碼標之

9         :為其他,無特殊類別

20-2510-1570-75

l                                          9 無特殊類,亦可和第一個碼配合運用

291979

 

 

l          78=自個兒“診斷碼”(國家碼, 可自選)

l          99 為空白

l          79Erases

 

 

l          評分”時間估量:如(試測)(在2-4星期內)

 

例:G85 ×2201,100

    G47 ×2201,540   1,1001,5402,640

以一人評分:每天7小時,1小時15本,

則需 2,640÷15÷72425

6人,則一星期可完成。

 

如何

(1)                                    減少評分訓練時間?

(2)                                    減少流動(易人改,確保品質)?

(3)                                    二次評分之一致性?

 

建議四個步驟:(分四大群 4Gs: G4M, G4S, G8M,G8S, 每一G, 2-3評分人)

Ⅰ、G1評分#1(即只評B1-3   G31B1-3

    G2評分#2(即只評B4-5/7     G42B4-7

Ⅱ、每一評分每一“隔”冊,記錄在確定表。

Ⅲ、交換冊

Ⅳ、評分每一冊,記錄在“冊”上


l          Field Test (Pilot Test) 架構

 

G8                                                                                         M and S   15  blocks

 

G8

P

P

Book 1

1-13PM12

14-26PS11

27-39AS04

40-52PS12

53-65AM01

66-69PM14A

加“計算機”調查

Book 2

1-12PS07

13-25PM10

26-29PM13A

30-42AM02

43-55AS03

56-61PS13

加“計算機”調查

Book 3

1-13PM08

14-25PS08

26-31PS14

32-44AS01

45-58AM03

59-61PM14

加“Calculator Survey

Book 4

1-14PS09

15-28PM11

29-35PM13

36-49AM04

50-63AS02

64-70PS14A

加“Calculator Survey

Book 5

1-13PS10

14-26PM07

27-40AM05

41-53PM09

54-65AS05

66-74PS13A

加“Calculator Survey

 

T: Test standardized

P: new Pilot test

A: the Alternative of new pilot test

S: Science

M: Mathematics

Number 01-14 : from 28 blocks of test bank
G4                                                                                          M and S  
21 blocks


G4

P

P

Book 1

1-11PS10

12-22PM12

23-31AM01

32-41TM02

42-49AS05

50-53PS13

加“Calculator Survey

Book 2

1-12PM08

13-23PS08

24-33AS01

 

34-40TS03

41-52TM05

53-56PM13

加“Calculator Survey

Book 3

1-11PS10

12-26PM07

27-40AM04

32-42TM03

43-50TS05

51-56PS13A

加“Calculator Survey

Book 4

1- 9PM07

10-20PS09

21-29AS04

30-36TS04

37-47TM06

48-52PM13A

加“Calculator Survey

Book 5

1-12PS07

13-24PM11

25-33AM02

34-44TM04

45-54AS02

55-60PS14

加“Calculator Survey

Book 6

1-11PM09

12-22PS12

23-30AS03

31-38TS02

39-48AM05

49-51PM14

加“Calculator Survey

Book 7

1- 8TS06

9-17AM06

18-22PM14A

23-32AM03

33-41AS06

42-46PS14A

加“Calculator Survey

 


:  TIMSS 2003 “評量2003”(正式題本)架構

 

範疇:

 

專家學者及TIMSS之經驗,把8th(8年級,G8)7小時作答與4th4年級,G45.5小時作答之題庫,分給統計上足夠意義的學生,8th(90分鐘)4th(65分鐘),再加15-30學生問卷,即8th每位學生不超過120分鐘,4th不超過80-95分鐘之測驗時間。

 

如此將所需之“題庫(8th 7hrs,4th 5.5hrs)”分成28區(數理各有14區),其題庫之28區,分區如下(6區為TIMSS 19951999之題庫,其餘8區(57.2%)為新題目,每區都是只會數或理,8th年級有15分鐘測驗題,而4th每區為12分鐘之測驗。數為M1M14,理為S1S14,而1314區為數理混合區,提供應付較長的解題與探究題。

 

涵蓋面

 

28評量區分佈在12冊題本(M1-14 S1-14) Exhibit 7.

   -6區探究:199519998th進展,但4th只有1995進展。

M評量內容佔%比如Exhibit 2

S評量內容佔比如Exhibit 3

TIMSS-2003 試題28 blocks 新舊之分佈如Exhibit 6

TIMSS-2003 試題與問卷測試時間之分佈如Exhibit 8

 

TIMSS: 2003-2011 試題的安全性: Exhibit 9

 

Exhibit 7: TIMSS 2003 Booklet Design- Fourth and Eighth Grade         (G4) 21blocks

Student Booklet

Assessment Blocks

Booklet 1 (M12,S11)

M1

M2

(A) S1

(P)S12

M7

M10

Booklet 2

M2

M3

(T)S2(A)

(P)S11

M13/14

Booklet 3

M3

M4

(T)S3(A)

(P)S10

M8

M11

Booklet 4

M4

M5

(T)S4(A)

(P)S9

M13/14

Booklet 5

M5

M6

(A)S5(T)

(P)S8

M9

M12

Booklet 6

M6

M1

(T)S6(A)

(P)S7

M13/14

Booklet 7

S1

S2

M1

M12

S7

S10

Booklet 8

S2

S3

M2

M11

(P,A)S13/14(P,A)

Booklet 9

S3

S4

M3

M10

S8

S11

Booklet 10

S4

S5

M4

M9

S13/14

Booklet 11

S5

S6

M5

M8

S9

S12

Booklet 12

S6

S1

M6

M7

S13/14

 

 

Exhibit 2: Target Percentages of TIMSS 2003 Mathematics Assessment Devoted to Content and Cognitive Domains by Grade Level

 

 

Fourth Grade

Eighth Grade

Mathematics Content Domains

Number

40%

30%

Algebra*

15%

25%

Measurement

20%

15%

Geometry

15%

15%

Data

10%

15%

Mathematics Cognitive Domains

Knowing Facts and Procedures

20%

15%

Using Concepts

20%

20%

Solving Routine Problems

40%

40%

Reasoning

20%

25%

 

*At fourth grade, the Algebra content domain is called Patterns, Equations, and Relationships.

 

 

Exhibit 3: Target Percentages of TIMSS 2003 Science Assessment Devoted to Content and Cognitive Domains by Grade Level

 

Fourth Grade

Eighth Grade

Science Content Domains

Life Science

45%

30%

Physical Science

35%

*

Chemistry

*

15%

Physics

*

25%

Earth Science

20%

15%

Environmental Science

*

15%

Science Cognitive Domains

Factual Knowledge

40%

30%

Conceptual Understanding

35%

35%

Reasoning and Analysis

25%

35%

 

 

 

*At fourth grade, Physical Science will be assessed as one content area including both physics and chemistry topics. Some understandings related to Environmental Science will be assessed as part of the Life Science and Earth Science content domains at fourth grade.

 

Exhibit 6: TIMSS 2003 Matrix-Sampling Blocks-Fourth and Eighth Grade (G8 S 15blocks)

Source or Type of Item

Mathematics Blocks

Science Blocks

Trend Items (TIMSS 1995 or 1999)

M1

(A) S1

Trend Items (TIMSS 1995 or 1999)

M2

(A) S2

Trend Items (TIMSS 1995 or 1999)

M3

(A) S3

Trend Items (TIMSS 1995 or 1999)

M4

(A) S4

Trend Items (TIMSS 1995 or 1999)

M5

(P) S5

Trend Items (TIMSS 1995 or 1999)

M6

S6

New Replacement Items

M7

(P) S7

New Replacement Items

M8

(P) S8

New Replacement Items

M9

(P) S9

New Replacement Items

M10

(P) S10

New Replacement Items

M11

(P) S11

New Replacement Items

M12

(P) S12

New Replacement Items

M13

(P) S13 (A)

New Replacement Items

M14

(P) S14 (A)

 

28 Blocks

 

Exhibit 9: TIMSS Block Release Schedule-Mathematics and Science, Fourth and Eighth Grades.

Assessment

Year

Released

Blocks

Secure

Blocks

2003

1, 2, 3, 5, 7, 10, 13, 14

4, 6, 8, 9, 11, 12

2007

1, 2, 4, 6, 8, 11, 13, 14

3, 5, 7, 9, 10, 12

2001

1, 2, 3, 5, 9, 12, 13, 14

4, 6, 7, 8, 10, 11

 

 

Exhibit 8: TIMSS 2003 Student Testing Time

Activity

Fourth Grade

Eighth Grade

Student Booklet-Part 1

(Calculators Not Permitted)

36 minutes

45 minutes

Break

Student Booklet-Part 2

(Calculators Permitted-

8th grade only)

36 minutes

45 minutes

Break


評分之問題:兩大類

 

Ⅰ、Multiple-choice (選擇題:45只選1)

    (盡量減低學生閱讀之負擔)

 

Ⅱ、結構-反應題 Constructed-Response

部分給分,瞭解學生學習該“區”評量內容之量化,而非其寫作之好壞,但其表達能力亦相當重要)評分者能有國際一致標準與素質。

31723日之scoring訓練,將1999年各國之評分結果與經驗傳授,以“監測”各國之評分一致性。

(To ensure consistent application of Scoring guides for 1999 items in the 2003 assessment, IEA has achieved samples of student responses from each country; these will be used to train scores in 2003 and to monitor consistent application)

 

Ⅲ、評分“量”之目標:定每個“區”有15分(8th)、12分(4th

如此:每受測年級(共M and S 28 blocks):

例如8th

112區,每一區可由 8個單選(8分)

                                        23個結構反應(每一,12分)     15

                                                1個延續反應(35分)

 

        1314(複合區),可含“複雜題”及inquiry…..       15(共最多30分)

                                                        (多分+少分)

 

評分“核對”項目清單如附件。

 

活動照片集景點滴

               3/16 台北→曼谷→阿姆斯特丹

               3/17 阿姆斯特丹→布魯塞爾機場→Ghent (St. P. Station) Sofitel(會場)

               3/18-21 Sofitel (會場)

               3/22 SofitelBrugesGhent→布魯塞爾

               3/23 布魯塞爾→Viena→曼谷

               3/24 曼谷→台北

 

 

       

               3/17-18 歡迎晚會,Sofitel旅館地下樓古堡及Ghent大學之晚宴

 

 

        

             3/18 第四次NRC會議開幕及會場附近廣場風光

 

 

       

             3/1821 自由試題反應評分研討會與研習

 

 

       

              3/22 Bruges古城遊覽(Flemish之藝術古城、巧古力、Lace及教堂、古堡)