This workshop was organized for postgraduate students, particularly those who are planning to develop module for PhD studies.
In a PhD study that involve module development, the design, development and validation of the module should cover 50 percentage of the entire PhD workload. For example, the a study titled:
Effect of Career Development Module Intervention (Independent Variable) on Career Maturity (Dependent Variable 1), Career Indecision (DV2) and Career Decision Making (DV3) in Saudi Arabia
Prof Sidek shared the ideal structure of Chapter 1 of a the PhD thesis:
1.1 Research Background (should consists of 10 to 20 pages)
- Introduce the title of thesis systematically, by explaining each variable one by one.
1.2 Problem Statement (2.5 to 3 pages)
This table should not be included in the thesis, instead it should be shown in Viva.
No.
|
Issue
|
Ideal
|
Reality
|
Gap
|
1
|
IV
|
“Based on theory”
|
“Based on LR”
|
Theoretical gaps vs practical gaps
|
2
|
DV1
|
“Reports from the Ministry”
|
||
3
|
DV2
|
|||
4
|
DV3
|
1.3 Research Questions
- IV: How to develop a training module for...
- DV1: Does the developed module significantly improve DV1?
- DV2: Does the developed module significantly improve DV2?
- DV3: Does the developed module significantly improve DV3?
1.4 Research Objective
- A PhD thesis that involves module development should have an objective for a Descriptive Study.
- The descriptive study is needed to assure whether the module (IV) is valid and reliable
- Subsequent research objectives should be written to assess whether the module (IV) can improve each DV
1.5 Hypothesis
- No hypothesis for research question 1
-------------------------------------------------------------------------------------
To Prof Sidek, quasi-experiment is not an option at all for evaluating the effect of a module. Instead, students must conduct true experiment. His assertion on the feasibility of conducting true experiment was that if true experiment should not be designed for social science research, then all books written about true experiment should be discarded.
When conducting a training programme, there is an issue for those which do not use module, i.e. the training session can only deliver what the invited speaker or facilitator like to deliver, as opposed to what were intended by the programme organizer.
In a training programme that uses module, all activities must be conducted, and facilitator or speaker cannot change according to his or her preference. In particular, every step in an activity has its allocated time, in which "minute" is applied as the measuring unit. In this sense, when designing a module, module developer must estimate the time needed for running each step in every activity. In a word, module is a means to control time among all trainers.
Creativity is not permitted in the execution of module. To assure no creativity, training of trainers (TOT) must be conducted unless the module developer is the sole facilitator who will implement the module. The TOT of the module MUST walk through the actual time of all activities, as recorded in the module. After that, the module developer must observe every new trainer, at least for the first training.
-----------------------------------------------------------------------------------
There are two types of modules:
1. Module that does not require facilitator: e.g. academic module, teaching module for distance education
- self-directed learning
- examples of feedback:
Tahniah! Anda telah mencapai XXX, sila maju ke aktiviti seterusnya.
Mungkin anda sekarang sudah letih. Sila rehat seketika.
2. Module that requires facilitator (modul berbantukan fasilitator): e.g. training module
Tips for viva: Is there any existing module out there? Are those modules effective or not?
Must conduct need analysis in every public university, through survey or interview
Unless, there was "arahan dari langit"
Lain padang lain belakang, lain lubuk lain ikannya, thus the module should provide what trainees need as opposed to what we want to give. A module can be effective to this group but not effective to other groups.
----------------------------------------------------------------------------------
Tips for writing Chapter 5 Discussion of PhD thesis
- Knowledge Contributions vs Practical Contributions
- Theoretical implication vs practical (application) implications to the nation
---------------------------------------------------------------------------------
Basic structure of a module
Module > Sub-modules (the breakdown must be based on theory) > Activities > Steps (smallest unit)
- It is good to base on more than one theory
*WNR can consider linking her research to career development theory in the field of animation or the creative industry
Examples of step in an activity of a module:
Step
|
Action
|
Duration (minutes)
|
Materials
|
1
|
Introduce activity
|
5
|
Module handout
|
2
|
Distribute survey forms
|
2
|
Survey forms
|
3
|
Briefing
|
3
|
Module handout
|
4
|
Answer questionnaire
|
20
|
Pencils / Pens
|
5
|
Count score
|
5
|
Rubrics
|
6
|
Make graft
|
5
|
Graft sheets
|
7
|
*Interpretation
|
60
|
Instrument descriptors
|
8
|
Draw conclusion (Rumusan)
|
5
|
Module handout
|
Objectives of conducting a pilot study:
- check every step
- observe the actual time needed for each step
The pilot study can be run without control group.
The pilot study can be run without control group.
*WNR can consider using video recording in a pilot study.
In the practice of PLKN nationwide, 14 centres started at the same time, and then ended at the same time for the same activity after 2 hours.
A module should also have Appendices section:
- at the end of each activity
- at the end of the module (neat but not ease of use)
The appendices should include brief interpretation of an activity and all relevant supplements for each step in an activity.
----------------------------------------------------------------------
Module Development Strategy
- should be similar to the strategy for developing research instruments
1. Rational strategy: based on rationalization of literature review
2. Empirical strategy: based on empirical research studies
3. Factorial strategy: based on factorial analysis
*WNR should explore each strategy accordingly before developing her module
After writing the first draft of a module, the module developer should rest for a few days before revisiting the module in order to revise or improve the module.
For non-PhD module, the module should be checked by language expert. 10 targeted trainees should be invited to comment whether they can understand or comprehend the module or not.
In PhD studies, students should combine experts' view and comments given by targeted trainees.
An excellent module should look exactly like a printed book.
-----------------------------------------------------------------
Content validation of an module
Involves at least 3 - 5 experts who are
- an expert in the field of the module (i.e. PhD in developing module)
- an expert in the content knowledge (to assure content validity)
Every expert should examine every step and every content of the module.
The student should elaborate the module development theory, pedagogical theory, and content knowledge theory.
When inviting experts, a formal letter from the supervisor is needed.
-------------------------------------------
Reliability of an module
Use 33 participants (cannot involve in actual study later) to participate the training using actual duration. Administer a survey immediately after the training.
How to develop the questionnaire for the reliability test?
- Based items on objective = one objective one item
- three types of scale:
-- categorical (yes / no): McNemar Test
-- Likert Scale 1 - 5
-- Semantic 0 to 10
Reliability Cronbach's Alpha
-------------------------------------------
Literature Review
Latest
|
Over 5 years
|
|||
f
|
%
|
f
|
%
|
|
Journal
|
||||
Non-journal
|
||||
--------------------------------------------------
Evaluating the effects of a module
When n > or = to 30, mean population = mean sample
Ideal count participants:
2 experimental groups = mass delivery 33 participants + 3 groups of 11 participants each
1 control group = 33 participants without learning at all
*3 participants are added to each group to tackle the mortality factor (assuming 10% of mortality)
The group approach (pendekatan kolompok) should not exceed 12 individuals in every group to avoid blind spot. When training more than 8 people in a group, co-facilitators will be trained and needed to handle blind spots in training.
If the differences between the mass training and group training are not significant, then it would be economical to conduct mass training.
Must assure all participants start at the same beginning point.
Same means across all groups.
If you are the person who conduct the training (without TOT for facilitators), you should get experts to observe your conduct of the training. Alternative, observe through video records.
Conduct follow-up to measure the sustainability of effects, one month (the earliest) after the post-test.
In the case the results drop in the follow-up, it could be caused by participants' fatigue, boring, e.g. the program may be conducted in marathon mode (e.g. 3 continuous days with 12 activities).
Advice to PhD students: module is not aiming to solve world problems. Thus should consider studying Sekolah Berasmara Penuh.
Module
|
Instrument for experiment
|
|
Validity
|
2 – 5 Experts for content validity
|
Develop test items based on DVs
Content experts
|
Reliability
|
30 participants for
Cronbach’s Alpha
|
*r must be > or = .60
Two types of tests:
a) maximum performance test: everyone wants high score, e.g. standard tests like PT3, UPSR, classroom or school-based assessment
b) typical performance test: no pass or fail, e.g. personality test
*WNR can consider conducting the actual study in one academic semester, where pretest is conducted at the beginning of one semester, and then post-test is conducted at the end of the semester.
*WNR must avoid including learning contents which are already in the syllabus.
Types of analysis:
- One-sample dependent T-test for comparing 2 groups
- ANOVA for comparing more than 2 groups
- Post hoc to identify the source of change.
Must assure normal distribution, by discarding extremists, i.e. beyond 1 standard deviation (SD) above and 1 SD below the Mean score.
The remaining scores would be 68%, in which top 16% scores and bottom 16% scores are considered outliners.
For example:
if Mean = 50 and SD = 10, then the range to be included is 40 to 60.
WNR: the recommended research design would be Randomised Pre-test / Post-test Control Group with Follow-up Experiment.
Sampling
Cluster randomized sampling vs simple randomized sampling
Stratified sampling (for gender, age differences,
May consider using a class as unit.
The mean of pre-test scores in all groups must be the same.
Paired randomised sampling (sampel rawak berpasangan): put frequencies all scores into Excel.
Score
|
Frequency
|
Draw
randomly
|
Group
|
150
|
9 out of 10 participants
|
3 participants
|
Experimental Group 1
|
3 participants
|
Experimental Group 2
|
||
3 participants
|
Control Group
|
||
Remaining 1 test taker
|
Discarded from the study
|
Treatment:
Meet participants twice a week.
Back Translation Technique
Method A
Requires 6 expert in the content knowledge (instead of language experts) who master the targeted second language.
First 3 experts translate the instrument, and then take majority votes. Else ask supervisor for help.
Second 3 experts translate the instrument back to the original language.
Must achieve at least 80% similarity
Method B
Two translation committee meetings: Week 1 and then Week 3
Tips for writing Chapter 3
- Discuss threats, no need to be lengthy.
- Combination of methods
- Define each method
- Strengths and weaknesses of each method, should have more strengths than weaknesses to justify why you choose this method
- Subject:
-- population,
-- sample size (how others determined sample size; pros and cons; your sample size)
- sampling technique: paired randomised sampling technique
