Maximize Field Operational Test (FOT) success by creating a sound experimental design and data acquisition plan.
Experience from a Field Operational Test (FOT) in Michigan to evaluate a Roll Advisor and Control (RA&C) system for commercial vehicles.
- Develop a sound experimental design.
There are a number of important things to consider when developing an experimental design. Selecting a long enough period for data collection is important to help eliminate seasonal effects from the main phenomena being studied. In the current study, there were two ways that the experimental design limited the strength of the conclusions. The FOT was conducted essentially within a one-year period, with one phase conducted as the weather was warming and the other phase conducted as the days shortened. Separating the seasonal effects from the RA&C’s effects could not be done with absolute certainty. Secondly, the demographics of the drivers were uniform. All were experienced, and all were within a twenty-year age span. A better test of the educational benefits of the RSA would have been to study two similar groups of novice drivers, one with the system and one without. Battelle, the evaluator of the FOT, understands that any experiment must be planned within constraints, and the partnership worked around these constraints as well as possible.
- Automate data collection as much as possible.
In this study, data collection was largely automatic. UMTRI reviewed the data to make sure all was working well. On ordinary days it took no help from Praxair personnel. In the cases where valid data were missing or the data collected were invalid, the partnership could readily trace the path through the server at the terminal to diagnose and fix the problem.
- When planning the FOT size, carefully consider the trade-off between ease of data collection and statistical power.
During this FOT, the use of a single terminal kept logistics to a manageable level. There were two occasions when a Battelle staff member interviewed the drivers. In both instances, nearly all of them were seen in a single trip of only a couple days. A benefit of the limited delivery area was that there were many road segments through which each driver passed many times. There were, however, disadvantages to the selection of a single terminal. Only six tractors, driven for six months each in the control and treatment phases, produced a fairly small number of serious events to analyze. By the nature of statistics, confidently estimating small changes, as were expected and observed in this FOT, requires large numbers of near-rollover incidents, which were simply not available. In this sense, the compact size of the FOT fleet was largely responsible for the lack of statistical significance in the final safety benefits estimate. Also, while the convenience of the terminal’s proximity to both UMTRI and Battelle had its benefits, the RA&C was tested exclusively in the Great Lakes region of the country, which is fairly free of hills and curves. As with many experiments, there is an inherent trade-off: a larger experiment provides richer data, but a smaller one is easier to conduct.
Published By: U.S. DOT Federal Highway Administration
Prepared by Battelle for the U.S. DOT FHWA
Source Date: September 2003
EDL Number: 13871
Other Reference Number: FHWA-OP-03-179URL: http://ntl.bts.gov/lib/jpodocs/repts_te/13871.html
Average User Rating