How to develop training module - learning from Prof Dr Sidek Mohd Noah

I attended the Bengkel Pembinaan Modul: Kesahan, Kebolehpercayaan & Keberkesanan together with a PhD student on 29 Dec 2018.

This workshop was organized for postgraduate students, particularly those who are planning to develop module for PhD studies.

In a PhD study that involve module development, the design, development and validation of the module should cover 50 percentage of the entire PhD workload. For example, the a study titled:

Effect of Career Development Module Intervention (Independent Variable) on Career Maturity (Dependent Variable 1), Career Indecision (DV2) and Career Decision Making (DV3) in Saudi Arabia 

Prof Sidek shared the ideal structure of Chapter 1 of a the PhD thesis:

1.1 Research Background (should consists of 10 to 20 pages)
- Introduce the title of thesis systematically, by explaining each variable one by one.

1.2 Problem Statement (2.5 to 3 pages)

 This table should not be included in the thesis, instead it should be shown in Viva.

No.	Issue	Ideal	Reality	Gap
1	IV	“Based on theory”	“Based on LR”	Theoretical gaps vs practical gaps
2	DV1		“Reports from the Ministry”
3	DV2
4	DV3

1.3 Research Questions
- IV: How to develop a training module for...
- DV1: Does the developed module significantly improve DV1?
- DV2: Does the developed module significantly improve DV2?
- DV3: Does the developed module significantly improve DV3?

1.4 Research Objective
- A PhD thesis that involves module development should have an objective for a Descriptive Study.
- The descriptive study is needed to assure whether the module (IV) is valid and reliable
- Subsequent research objectives should be written to assess whether the module (IV) can improve each DV

1.5 Hypothesis
- No hypothesis for research question 1

-------------------------------------------------------------------------------------

To Prof Sidek, quasi-experiment is not an option at all for evaluating the effect of a module. Instead, students must conduct true experiment. His assertion on the feasibility of conducting true experiment was that if true experiment should not be designed for social science research, then all books written about true experiment should be discarded.

When conducting a training programme, there is an issue for those which do not use module, i.e. the training session can only deliver what the invited speaker or facilitator like to deliver, as opposed to what were intended by the programme organizer.

In a training programme that uses module, all activities must be conducted, and facilitator or speaker cannot change according to his or her preference. In particular, every step in an activity has its allocated time, in which "minute" is applied as the measuring unit. In this sense, when designing a module, module developer must estimate the time needed for running each step in every activity. In a word, module is a means to control time among all trainers.

Creativity is not permitted in the execution of module. To assure no creativity, training of trainers (TOT) must be conducted unless the module developer is the sole facilitator who will implement the module. The TOT of the module MUST walk through the actual time of all activities, as recorded in the module. After that, the module developer must observe every new trainer, at least for the first training.
-----------------------------------------------------------------------------------

There are two types of modules:
1. Module that does not require facilitator: e.g. academic module, teaching module for distance education
- self-directed learning
- examples of feedback:

Tahniah! Anda telah mencapai XXX, sila maju ke aktiviti seterusnya.

Mungkin anda sekarang sudah letih. Sila rehat seketika.  

2. Module that requires facilitator (modul berbantukan fasilitator): e.g. training module

Tips for viva: Is there any existing module out there? Are those modules effective or not?

Must conduct need analysis in every public university, through survey or interview
Unless, there was "arahan dari langit"

Lain padang lain belakang, lain lubuk lain ikannya, thus the module should provide what trainees need as opposed to what we want to give. A module can be effective to this group but not effective to other groups.
----------------------------------------------------------------------------------

Tips for writing Chapter 5 Discussion of PhD thesis

- Knowledge Contributions vs Practical Contributions
- Theoretical implication vs practical (application) implications to the nation

---------------------------------------------------------------------------------

Basic structure of a module

Module > Sub-modules (the breakdown must be based on theory) > Activities > Steps (smallest unit)

- It is good to base on more than one theory

*WNR can consider linking her research to career development theory in the field of animation or the creative industry

Examples of step in an activity of a module:

Step	Action	Duration (minutes)	Materials
1	Introduce activity	5	Module handout
2	Distribute survey forms	2	Survey forms
3	Briefing	3	Module handout
4	Answer questionnaire	20	Pencils / Pens
5	Count score	5	Rubrics
6	Make graft	5	Graft sheets
7	*Interpretation	60	Instrument descriptors
8	Draw conclusion (Rumusan)	5	Module handout

 * Interpretation is the most important step of this activity

Ideally, an activity should be conducted from 45 minutes to 120 minutes. Every activity in a sub-module must have name, abstract/ synopsis, objectives, apparatus, delivery method (visit, workshop, brainstorming, watching film, retrieving information from the Internet, etc)

The key to successful module delivery is MMA (macam-macam ada)

Objectives of conducting a pilot study:

- check every step

- observe the actual time needed for each step

The pilot study can be run without control group. 

*WNR can consider using video recording in a pilot study.

In the practice of PLKN nationwide, 14 centres started at the same time, and then ended at the same time for the same activity after 2 hours.

A module should also have Appendices section:
- at the end of each activity
- at the end of the module (neat but not ease of use)

The appendices should include brief interpretation of an activity and all relevant supplements for each step in an activity.
----------------------------------------------------------------------

Module Development Strategy

- should be similar to the strategy for developing research instruments

1. Rational strategy: based on rationalization of literature review
2. Empirical strategy: based on empirical research studies
3. Factorial strategy: based on factorial analysis

*WNR should explore each strategy accordingly before developing her module

After writing the first draft of a module, the module developer should rest for a few days before revisiting the module in order to revise or improve the module.

For non-PhD module, the module should be checked by language expert. 10 targeted trainees should be invited to comment whether they can understand or comprehend the module or not.

In PhD studies, students should combine experts' view and comments given by targeted trainees.

An excellent module should look exactly like a printed book.

-----------------------------------------------------------------
Content validation of an module 

Involves at least 3 - 5 experts who are
- an expert in the field of the module (i.e. PhD in developing module)
- an expert in the content knowledge (to assure content validity)

Every expert should examine every step and every content of the module.

The student should elaborate the module development theory, pedagogical theory, and content knowledge theory.

When inviting experts, a formal letter from the supervisor is needed.

-------------------------------------------
Reliability of an module 

Use 33 participants (cannot involve in actual study later) to participate the training using actual duration. Administer a survey immediately after the training.

How to develop the questionnaire for the reliability test?
- Based items on objective = one objective one item
- three types of scale:
-- categorical (yes / no): McNemar Test
-- Likert Scale 1 - 5
-- Semantic 0 to 10

Reliability Cronbach's Alpha

-------------------------------------------
Literature Review 

	Latest		Over 5 years
	f	%	f	%
Journal		51%
Non-journal

--------------------------------------------------
Evaluating the effects of a module 

When n > or = to 30, mean population = mean sample

Ideal count participants:
2 experimental groups = mass delivery 33 participants + 3 groups of 11 participants each
1 control group = 33 participants without learning at all
*3 participants are added to each group to tackle the mortality factor (assuming 10% of mortality)

The group approach (pendekatan kolompok) should not exceed 12 individuals in every group to avoid blind spot. When training more than 8 people in a group, co-facilitators will be trained and needed to handle blind spots in training. 

If the differences between the mass training and group training are not significant, then it would be economical to conduct mass training.

Must assure all participants start at the same beginning point.
Same means across all groups.

If you are the person who conduct the training (without TOT for facilitators), you should get experts to observe your conduct of the training. Alternative, observe through video records. 

Conduct follow-up to measure the sustainability of effects, one month (the earliest) after the post-test.

In the case the results drop in the follow-up, it could be caused by participants' fatigue, boring, e.g. the program may be conducted in marathon mode (e.g. 3 continuous days with 12 activities).

Advice to PhD students: module is not aiming to solve world problems. Thus should consider studying Sekolah Berasmara Penuh.

	Module	Instrument for experiment
Validity	2 – 5 Experts for content validity	Develop test items based on DVs Content experts
Reliability	30 participants for Cronbach’s Alpha

*r must be > or = .60

Two types of tests:
a) maximum performance test: everyone wants high score, e.g. standard tests like PT3, UPSR, classroom or school-based assessment
b) typical performance test: no pass or fail, e.g. personality test

*WNR can consider conducting the actual study in one academic semester, where pretest is conducted at the beginning of one semester, and then post-test is conducted at the end of the semester.
*WNR must avoid including learning contents which are already in the syllabus.



Types of analysis:
- One-sample dependent T-test for comparing 2 groups
- ANOVA for comparing more than 2 groups
- Post hoc to identify the source of change.

Must assure normal distribution, by discarding extremists, i.e. beyond 1 standard deviation (SD) above and 1 SD below the Mean score.
The remaining scores would be 68%, in which top 16% scores and bottom 16% scores are considered outliners.

For example:
if Mean = 50 and SD = 10, then the range to be included is 40 to 60.

WNR: the recommended research design would be Randomised Pre-test / Post-test Control Group with Follow-up Experiment.

Sampling
Cluster randomized sampling vs simple randomized sampling
Stratified sampling (for gender, age differences,

May consider using a class as unit.

The mean of pre-test scores in all groups must be the same.

Paired randomised sampling (sampel rawak berpasangan): put frequencies all scores into Excel.

Score	Frequency	Draw randomly	Group
150	9 out of 10 participants	3 participants	Experimental Group 1
		3 participants	Experimental Group 2
		3 participants	Control Group
		Remaining 1 test taker	Discarded from the study

Conduct pretest to the whole population before grouping to avoid sampling error.

Treatment:
Meet participants twice a week.

Back Translation Technique 
Method A
Requires 6 expert in the content knowledge (instead of language experts) who master the targeted second language.
First 3 experts translate the instrument, and then take majority votes. Else ask supervisor for help.
Second 3 experts translate the instrument back to the original language.
Must achieve at least 80% similarity

Method B
Two translation committee meetings: Week 1 and then Week 3

Tips for writing Chapter 3
- Discuss threats, no need to be lengthy.
- Combination of methods
- Define each method
- Strengths and weaknesses of each method, should have more strengths than weaknesses to justify why you choose this method
- Subject:
-- population,
-- sample size (how others determined sample size; pros and cons; your sample size)
- sampling technique: paired randomised sampling technique

How to measure creativity

TTCT:

• Fluency 
• Flexibility
• Originality
• Elaboration

------------------------------------------------------------------------

Tony Buzan's suggestion:

·         Novelty: being new, original or unusual.

·         Quantity: the amount of ideas created.

·         Practicality: the feasibility of things or ideas.

·         Aesthetics: the beauty of the creation or ideas.

------------------------------------------------------------------------------

Anatoly Guin & Mark Barkan:

R = PK x PC x (1+M)(1+t)

where
R: result
PK: personal knowledge
PC: physical capability
M: method
t: tool


Speed is one important
----------------------------------------------------------------------------------
Product-oriented problem-solving:
D.U.M.B.S., suggested by Robest Yong
- Doable,
- Usable,
- Marketable,
- Bankable and
- Sustainable

(no scale, no rubrics, no indicators) 

How to develop a research instrument

1. Determine the objective for needing an instrument. Why do you need the instrument?
- In preschool Malaysia, there is no existing standard instrument in DSKP
- An instrument is needed to afford preschool teachers to provide constructive feedback to individual pupils, encouraging them to thinking creatively.

2. Determine the constructs of the instrument. What do you want to measure?
- You actually do not know, so you tried to see how others had done this in similar contexts (literature review).
- Determine similar contexts and different contexts in order to qualify and disqualify particular existing instruments as references for building your instrument.
- At the end of the selection, two existing instruments were found suitable for use in the adaptation. 
- So how did you adapt? First, examine the constructs of each referenced instrument and determine whether individual constructs are suitable for measuring creative thinking in colouring activity.
- After that, a list of suitable constructs was made. Can you use these constructs directly? If yes, use with justification; else modify these constructs according to your measurement objective.
- Need to justify why limit your instrument to have only four constructs

3. Meanwhile, with reference to the chosen instruments for adaptation, determine the scale to be used for your need of measurement.   
- Can you use the scale directly? If yes, use it with justification; else modify the scale according to your measurement objective.

4. Develop test items / indicators for each construct  

Instrument
> scale for the overall instrument
>> rubrics for each reading
> constructs
>> indicators of each construct

Cara menjawab komen pemeriksa (selepas VIVA)

Langkah 1: Masukkan semua komen dalam laporan pemeriksa, wakil dekan dan catatan penyelia semasa viva ke dalam Laporan Pembetulan Tesis

Langkah 2: Buat justifikasi kepada setiap komen pemeriksa, sama ada setuju atau tidak dengan hujah berkenaan.

Langkah 3: Bagi hujah atau komen yang tidak disetujui (pemeriksa yang tidak hadir) dan disokong oleh penyelia, huraikan dengan bukti kenapa tidak setuju dengan komen atau hujah berkenaan dalam Laporan Pembetulan. 
 
Langkah 4: Bagi hujah atau komen yang disetujui, buat pemurnian (separa setuju) atau pembetulan (setuju sepenuhnya) pada tesis (dengan warna teks yang menonjol) mengikut komen yang tercatat dalam Laporan Pembetulan.

Langkah 5: Tunjukkan Laporan Pembetulan bersama dengan tesis yang telah dimurnikan dan dibetulkan kepada penyelia, sehingga mendapat pengesahan pembetulan daripadanya.

Langkah 6: Tunjukkan Laporan Pembetulan bersama dengan tesis yang telah disahkan oleh penyelia kepada pemeriksa dalam dan/atau luar sehingga mendapat pengesahan pembetulan daripadanya.

Reaction to "Vernacular, religious schools must go"

I read a letter forwarded by a columnist from Sin Chew Daily, titled "Vernacular, religious schools must go".

The writer's argument contains a cause-effect mistake. Accusing the lack of critical thinking and quality questioning skills among Malaysian for the presence of vernacular and religious schools demonstrate the writer's lack of logical thinking. According to the theory of Multiple Intelligences proposed by Howard Gardner, linguistic intelligence and logical-mathematical intelligence are developed in parallel to other forms of intelligences (see https://howardgardner.com/multiple-intelligences/). These intelligences are complementary to each other, rather than competing to each other. Instead of condemning specific school types, we should provide opportunities to Malaysian to nurture and develop logical-mathematical thinking, critical thinking and multiple linguistic skills. These skills would afford Malaysian to respond rationally to current issues and avoid making improper cause-effect mistakes.  

Algorithm Humans live by

Algorithms are step-by-step instructions that seek the best possible solution to a problem.

Inspiring concepts to me:
Problem that inspires them
Emotional motivation and informational motivation
Computational cognitive science
Where rule is relaxed.
Information imbalances
Computational kindness

What is PhD?

PhD is essentially a self-actualization journey, in which you go really deep to understand about yourself by:

- understanding your past achievements and failures, 

- understanding your current strengths and needs, and 

- predicting future trends of what you are good at now.

Philosophy = Philo- + sophy
where Philo means “love” and sophy means “wisdom”

To me, loving wisdom means loving what, when, where, why and how to use knowledge.

This is aligned to my life motto: Do the right thing at the right time and use the right method.

Get it done first, then get it right

When I was pursuing my PhD at Warwick Institute of Education, I always kept what Prof David Wray taught me: “get it written before getting it right”.

I documented my written progress, especially when I modified or change my research design, as shown in the following figure. I changed 10 times in my PhD, before finding what I actually wanted to study. However, the difference between a lot of PhD students and me is: I worked rapidly at the first 6 months of PhD. According to my supervisors and seniors, the changes I made might take 2 years of other students. My secret was: I read a lot during first 6 months, and I asked a lot of questions to myself and others. 

I found most of graduate students or potential graduate students facing writing block, in fact, that happens to me once a while. 

So, to prospective graduate students who do not have a specific topic or scope of study, I may offer a tentative topic or scope for him or her to start with. 

Now I start to see the benefits of the US PhD education system: students typically spend 2.5 years taking courses, and develop a scope and topic of interests along this period, while getting exposure to most if not all generic research methods—quantitative, qualitative, mixed methods, design-based research, action research, etc. With these intensive exposure and experience, graduate students should be able to develop himself or herself and prepare a solid thesis at the end of the 5th year. 

Thesis structure

Attended academic writing sessions: 'Thesis Structure' & 'How not to write' on 11 July 2008. Herewith my notes:

Thesis Structure

Structure: the interrelation or arrangement of parts in a complex entity; a notion / reality covering the recognition.

Categories of knowledge in thesis writing:

1. Why am I doing this? Reasons? 2. What is known and unknown? 3. What is my intention to discover / investigate? 4. What are my methods?	EXPLORATION
5. What have I found? 6. What does it mean? (to you? to everybody else?) 7. So what? Possible applications / recommendations? What contribution doest it make to knowledge? What's next?	RESULTS

 

Proportion of thesis:	Percentage	Words
Introduction (write as a whole at the end) Methods (write competantly instead of tentatively) Findings Discussion (the significant of your findings)/analysis Conclusion (second last)	10% - 20% (very conceptual subject) around 20% (has to be very detailed) 10% - 20% 30% - 40% 20%	8000 - 16000 16000 8000 - 16000 24000 - 32000 16000

The two 'pillars' of thesis structure: INTRODUCTION & CONCLUSION

How to build a theoretical model

My note in March 2008 (from my research diary)

A model is a simplified (often mathematical) description of a system etc., to assist calculations and predictions.

Steps in building a theoretical model:-

1.  The variables to be used in characterising and understanding the process must be specified.
2. The forms of the relationships connecting these variables must be specified.
3. Ignorance and the need for simplicity will ensure that all relationships other than identities are subject to error and so, for purposes of efficient statistical estimation, these error terms must be specified.
4. The parameters of the model must be estimated and the extent of its identification ascertained; if this is inadequate, the model must be reformulated.
5. Finally, the model must be kept up to date and used, so that an impression can be formed of its robustness and reliability.

Theoretical models are of many kinds: static or dynamic; partial or complete; aggregated or disaggregated; deterministic or stochastic; descriptive or optimising.