BC Ferries - a Case Study in Blended Maritime Assessment - Part 1
This article presents a case study of assessment practices employed by British Columbia Ferry Services Inc. (BC Ferries) - one of the largest ferry systems in the world. BC Ferries has, arguably, one of the most advanced and successful maritime training and assessment programs in existence.
Assessment in the maritime industry is a huge subject. One could write about it for months and still only touch some of the major topics. Even so, I conclude this series on assessment (for now - no doubt I will revisit it again soon) with a case study of British Columbia Ferry Services Inc. (BC Ferries) - one of the largest ferry systems in the world. BC Ferries has, arguably, one of the most advanced and successful maritime training and assessment programs in existence. By examining their assessment practices, and the practices of others, we can raise the level of knowledge and discussion on maritime assessment to the benefit of all.
This first article in the case study presentation will introduce the BC Ferries assessment process and describe how BC Ferries combines assessment techniques in order to provide objective, standardized and comprehensive assessment. The articles which follow will continue the case study by discussing:
- how BC Ferries trains it assessors,
- how it uses assessment at two key points in the training process,
- how and why it separates summative assessment from training,
- how it supports the assessors with comprehensive assessment instructions and materials,
- how it encourages the development and exchange of best practice assessment through the use of a trainer/assessor community,
- and how it uses assessment metrics and feedback to inform continuous improvement.
In order to be notified of these upcoming articles (and all upcoming articles from this maritime training blog), please click here.
BC Ferries owns 36 vessels operating over 25 routes. Every year approximately 21 million passengers and 8 million vehicles are transported safely through the efforts of 4,000 marine personnel. Every year new and existing employees must be trained on knowledge and skills pertaining to BC Ferries job routines, vessels, routes and terminals.
Until a few years ago, BC Ferries employed training and assessment techniques which are quite common in the maritime industry. New employees, or those advancing to higher positions, were trained primarily through a process of job shadowing. The prospective employee would spend anywhere from a few days to two or more weeks following one or more existing employees engaged in the performance of their duties.
Following job shadowing, the prospective employee was then assessed by a “clearer”. The clearance process was guided by a high-level checklist of competencies. The clearer, using the checklist as a guide, would ask the candidate to confirm or demonstrate that they possessed the various competencies or knowledge listed on the checklist.
A clearer who is an expert mariner and conscientious assessor will ensure that only qualified candidates (in their opinion) are cleared for duty. However, the success of the process is highly dependent on the aptitude and performance of the clearer. It is up to the clearer to interpret the checklist to decide what constitutes sufficient knowledge or adequate performance of a skill. This interpretation will vary from clearer to clearer, and be subject to competing influences such as operational constraints, time availability and personal biases. Overall, this type of assessment process can be highly variable, difficult to control, and may be impossible to measure.
In order to bring more rigor and standardization to training and assessment, BC Ferries created what they call the “SEA Program”. SEA, here, stands for “Standardized Education and Assessment”. The SEA program is a comprehensive blended (eLearning and face-to-face training) program for both training and assessment. The program moves away from job shadowing toward a multi-phase approach including eLearning, face-to-face training with dedicated trainers, and a multi-part assessment process. This program has been discussed to some extent in a previous article. I will cover various aspects of it again in more depth in future articles, but for this post I will concentrate on the assessment aspects of SEA.
In order to achieve BC Ferries goals of rigor, consistency and objectivity of assessment, BC Ferries has employed a combination of techniques which have been discussed in this series on assessment.
Assessment must cover both required skills and knowledge. Different assessment techniques are required for each. Likewise, different assessments techniques have varying levels of flexibility and objectivity. As such, it would be difficult or impossible to provide both comprehensive and objective assessment using only a single technique. BC Ferries, therefore applies a combination of techniques with different strengths - each one designed to contribute in its own way to the overall process. Any one of these techniques, in isolation, would provide an incomplete assessment. However, used in combination, all of the assessment goals can be met.
The techniques used to assess every candidate are the following:
- An objective written examination.
- An oral scenario-based examination.
- A demonstrative examination.
- A meeting with the master.
Each component of the clearance must be passed in order for the candidate to be cleared.
The written examination is a vessel and route-specific (or terminal-specific, depending on the candidate’s department) randomized, computer-graded, multiple-choice examination. The candidate has a fixed time to complete the exam which may consist of anywhere from 40 to 80 questions. In order to pass the examination, the candidate must achieve a grade of 80% or more.
The written exam contributes to the overall assessment process through its ability to test knowledge in a highly objective manner. The exam in randomized in order to ensure that candidates do not have any knowledge of the specific exam questions prior to the exam. Because there is no interpretation involved in grading the exam, all candidates are assessed on a level playing field and the results are completely objective.
In this clearance phase, the candidate is presented with a selection of ten realistic scenarios they may face as a working employee, and are asked for the appropriate reaction. They must correctly answer eight of the ten scenarios. A large list of scenarios is available on the SEA trainer resource website, and each scenario includes important points that should be covered by the candidate in their response. The clearer will select ten scenarios from those provided in order to ensure that assessments vary from one candidate to another. Example scenarios include asking the candidate what they would do when presented with an order from the master which they believe to be unsafe, or what their reaction would be if the vessel began to move away from the dock while passengers were still on the overhead walkway.
These kinds of oral scenario-based questions are less objective than are multiple-choice examinations. However, they are a valuable technique in that they are very useful in assessing a candidate's ability to assimilate their knowledge and use reason to formulate a valid plan of action in the face of an unanticipated scenario. Thus they test not only knowledge, but also the ability to apply and communicate that knowledge.
In this clearance phase, the candidate is asked to demonstrate knowledge or competencies through a set of “clearance activities”. A large list of clearance activities is defined and available on the SEA trainer website from which the clearer may choose. When the clearer selects the activity, they indicate the vessel and route the candidate is being cleared for, and the SEA website dynamically updates the activity to include vessel- and route-specific activity information. This creates a highly targeted set of assessment activities.
Some of these activities are marked as optional, and other are marked as mandatory. The mandatory activities are those which are considered to be essential safety-critical components of all employees’ knowledge or skills. The optional activities are also important, and serve to ensure that testing is comprehensive and that no two assessments are exactly alike. The clearer takes the candidate through all of the mandatory activities as well as through a selection of optional activities. Examples of clearance activities include demonstration of competency with life rafts, donning and doffing of fire fighting equipment, and so on. Each activity lists specific components of a successful demonstration in order to remove as much subjectivity as possible.
This clearance phase, like the oral scenarios, is less objective than multiple choice examinations. However, demonstrative examinations are an excellent way of assessing skills (as opposed to knowledge) and therefore constitute an important component of a well rounded assessment. As indicated above, subjectivity is removed to the greatest extent possible by providing specific information (sometimes to the level of vessel- or route-specific) on what constitutes a passing grade in the demonstration. In order to pass this phase, all activities marked as mandatory must be passed.
The meeting with the master is an opportunity for the master to meet and ask questions of the candidate. The master is encouraged to either ask questions of his or her own, or to ask questions from scenarios which were not covered during the oral scenario-based examination. The clearer is expected to attend this meeting between the candidate and master.
Clearly this meeting is highly subjective. However, it is a critical part of assessment in that it allows the master to use his or her own professional judgement and experience to assess the candidate. As with the other clearance phases, a meeting with the master would, on its own, be an incomplete assessment process. However, it contributes valuable assessment information as one part of a more varied and comprehensive process.
Combining assessment techniques, such as BC Ferries does, provides a more standard, objective and comprehensive assessment process than could be provided by any single assessment technique alone. Multiple choice exams test knowledge and are highly objective. Oral scenario-based exams test the ability to communicate effectively and assimilate knowledge into a plan of action. Demonstrative clearance activities assess critical skills. And finally, the meeting with the master allows for that “hard to quantify” human assessment that an experienced professional master can contribute. Together, all important aspects of assessment are covered.
As I indicated in the introduction, I will continue this case study of BC Ferries assessment techniques by looking at many other aspects of BC Ferries assessment practices including the support which BC Ferries provides for it assessors, how training and summative assessment are kept distinct, the use of assessment metrics, and so on. If you would like to be notified of these upcoming articles (and all upcoming articles from this maritime training blog), please click here.
Until then - thanks for reading, take care and have a great day!
# # #
About The Author:
Murray Goldberg is the founder and President of Marine Learning Systems (www.marinels.com), the creator of MarineLMS - the learning management system designed specifically for maritime industry training. Murray began research in eLearning in 1995 as a faculty member of Computer Science at the University of British Columbia. He went on to create WebCT, the world’s first commercially successful LMS for higher education; serving 14 million students in 80 countries. Now, in Marine Learning Systems, Murray is hoping to play a part in advancing the art and science of learning in the maritime industry.