FESTA handbook Introduction

Back to FESTA Handbook main page: FESTA handbook

1. Introduction

Field Operational Tests (FOTs) were introduced several years ago as an evaluation method for driver support systems and functions with the aim of proving that such systems can deliver real-world benefits. A number of such FOTs have been conducted at a regional, national and international (European) level, to evaluate a range of systems, particularly a variety of driver support systems. These FOTs have proven to be highly valuable and have been identified as an important means of verifying the real-world impacts of new systems, including at a European level and as a means to verify that previously conducted R&D has the potential to deliver identifiable benefits. The key deliverable for "FESTA", (Field opErational teSt supporT Action), has been to produce comprehensive guidance to facilitate the successful delivery of FOTs. This, the latest version of the FESTA handbook has drawn from experience and knowledge across numerous FOTs in order to revise and update the best practice presented here. This Handbook is the result of a joint effort of several research institutes, OEMs and other stakeholders from across Europe to prepare a common methodology for European FOTs. It is also highly relevant, and it is hoped useful, for FOTs conducted at a regional or national level within Europe as well as outside Europe.

For the purposes of this Handbook, a “Field Operational Test” (FOT) is defined as:

A study undertaken to evaluate a function, or functions, under normal operating conditions in road traffic environments typically encountered by the participants using study design so as to identify real world effect and benefits.

This means that it must be possible to compare the effects that the function has on traffic with a baseline condition during which the function is not operating. In order to achieve this, the participants’ control over or interaction with the function(s) has to be manipulated by the research team. “Normal operating conditions” implies that the participants use the platforms during their daily routines, that data logging works autonomously and that the participants do not receive special instructions about how and where to drive. Except for some specific occasions, there is no experimenter in the vehicle, and typically the study period extends over at least a number of weeks. FOTs must however consider whether it is still possible to achieve a baseline where all studied systems are off due to the proliferation of functions in standard vehicles).

The main purpose of this Handbook is to provide guidelines for conduct of FOTs and (NDS) although, for simplicity, in the following, the name FOT is used and differences between the two kind of studies are summarized when relevant. It walks the reader through the whole process of planning, preparing, executing, analysing and reporting an FOT and it gives information about aspects that are especially relevant for a study of this magnitude, such as administrative, logistic, legal and ethical issues. Another aspect of the Handbook is to pave the road for standardization of some aspects of FOTs, which would be helpful for cross-FOT comparisons. It has to be kept in mind, though, that many traffic parameters in different European countries differ substantially.

In Figure 1.1 the steps that need to be carried out during an FOT are presented in the form of V diagram, where there is correspondence between the levels on the left-hand and right-hand sides. The steps will be explained in detail in the different chapters of the Handbook. For orientation purposes, a copy of the figure is provided in the beginning of each chapter highlighting which step of the FOT. Chain is described in the current chapter. The FOT Implementation Plan takes up all the steps and integrates them into one big table which can be used as a reference when actually carrying out an FOT

Figure 1.1: The steps that typically have to be considered when conducting an FOT. The large arrows indicate the time line.

In order to make the picture more complete a horizontal bar has been added on top of the diagram that in principle summarises the context in which the FOT is supposed to take place. For instance, the choice of a function to be tested implies that there is either a problem that is to be addressed and that the chosen function is defined to solve the problem or that a policy objective is stated and that the function tested can be used to reach the objective. An FOT can always be related to a wider perspective view than is defined by just a description of the function to be tested.

The top of the V covers setting up a goal for the study and selecting a suitable research team, and also the last steps that include an overall analysis of the systems and functions tested and the socio-economic impact assessment, dealing with the more general aspects of an FOT and with aggregation of the results. The further down on FOT Chain V-Shape the steps are located, the more they focus on aspects with a high level of detail, such as which performance indicators to choose, or how to store the data in a database. The ethical and legal issues have the strongest impact on those aspects, where the actual contact with the participants, the data handling and potential data sharing takes place.

The representation of the FESTA methodology in the form of a V does not mean that designing and performing an FOT is always a linear process. Decisions made at a certain stage of the FESTA-V influence the next steps and it is likely that there will be a need to sometimes go back and redo some steps. Especially on the left-hand side of the V iteration may be necessary. For example, one may find out that the measures and sensors available do not make it possible to investigate the hypotheses defined earlier, performance performance indicators may be needed. Also the right hand side of the FESTA-V may influence the decisions to be made on the left-hand side. The need to assess the socio-economic impact may influence the definition of functions, use-cases, research questions and other elements of the left-hand side. Consideration of the resources available for data analysis may also lead to revision of the left-hand side. There is, however, the question “when does one stop the iterative process?” From a research perspective, this is a continuous process. However, from a project management perspective, boundaries have to be set to reflect budget constraints and timing aspects.

The first step in the FESTA-V is the identification of functions to be tested. Sometimes this may not be the best step to start with. For example, an FOT may not be driven by the technical systems that need to be tested but by a research question or an impact area. When there is a large set of functions available from which a few need to be selected as candidates for testing, definition of the research questions may help the selection process. For example if safety is to be investigated, different functions may have a higher priority than when traffic efficiency is the main focus.

In conclusion, the FESTA-V provides a static picture of the complex design and execution of an FOT . but in reality a more iterative process will be needed, with a starting point suited to the specific aims of a project.

The FESTA Handbook is not meant to be a substitute for consultations with experts, organising a good and capable research team, and carrying out specific investigations into the legal and ethical issues that apply to the current question and situation. It is not an exhaustive action list, and each FOT has its own special issues and concerns that have to be dealt with on an individual basis. Nor is the advice in it necessarily perfect and representative of the state of the art. On many issues, there will be scope for disagreement with the recommendations or use of alternative sources of advice. But it would certainly be preferable for major departures from the advice to be justified to funding agencies and major stakeholders.

The FESTA project consortium decided early in the project that the primary focus of the FESTA Handbook would be on the evaluation of Advanced Driver Assistance Systems (ADAS) and In-Vehicle Information Systems for vehicles — both in the form of autonomous systems and of Cooperative Systems. It was also agreed that the FESTA Handbook should be relevant to the evaluation of Original Equipment Manufacturer (OEM), aftermarket and nomadic systems. The Handbook is therefore designed specifically to guide the evaluation of impacts of such systems, and is less relevant to the evaluation of electronic road infrastructure such as Variable Message Signs (VMS). However, it will be seen that many of the activities identified in the Handbook are common to the evaluation of most vehicle- and infrastructure-based ICT technologies).

In conclusion, the FESTA Handbook gives an overview and general guidelines concerning the conduct of an FOT. FOTs are designed to contribute to the identification and verification of solutions to a problem, and this handbook is intended to provide a formalized and practical framework, and not a cook book: the methodology described will necessarily have to be adapted to the specific case, in order to increase the efficiency of the approach or to tackle data incompleteness or inconsistency. Furthermore, the results of an FOT may have to be integrated with external sources of information, to achieve a wider perspective, and an increased relevance for tackling the problem at hand.

In addition to the Handbook itself, more detailed work, which was produced during the FESTA project and later in the FOT-Net projects, and which is referenced in the Handbook, is included in the annexes to the Handbook and the FESTA deliverables.

1.1. The Naturalistic Driving Study in relationship to the Field Operational Test and the FESTA methodology

1.1.1 Definition

The Naturalistic Driving Study (NDS) or observation is a relatively new research method using advanced technology for in-vehicle unobtrusive recording of driver (or rider) behaviour during ordinary driving in everyday traffic situations. This method yields unprecedented knowledge primarily related to road safety, but also to environmentally friendly driving/riding and to traffic management. A central focus of naturalistic driving studies is to understand explanatory factors associated with crashes and predict involvement in crashes. The naturalistic method “refers to a method of observation that captures driver behaviour in a way that does not interfere with the various influences that govern those behaviours.” (Boyle et al., 2009). Naturalistic driving studies are defined as “those undertaken using unobtrusive observation or with observation taking place in a natural setting” (Dingus et al., 2006). The following characteristics have been chosen to define the Naturalistic Driving approach in the recent EU project PROLOGUE (Sagberg et al., 2011):

• Unobtrusive recording of driver and vehicle parameters
• Normal driving, i.e. driving purpose and driving destinations as defined by the driver, and driving taking place on roads open to ordinary traffic, and with the vehicle that the driver normally uses (owned, leased, or company vehicle)
• No observer present in the vehicle

1.1.2 Relation to FOTs

Naturalistic Driving Studies tend to focus on crash-explanatory factors, and Field Operational Tests generally focus on evaluation of systems or functions. However the collected data in both types of studies can be used for many alternative purposes such as analysis of Environment, Efficiency and Mobility impacts. NDS and FOTs are preferably seen as different methods because (a) the study design is different (participant selection, experimental conditions, vehicle sample, etc.), and (b) the research questions and hypotheses are different. In particular, the main difference relates to degree of experimental control found in the study, as illustrated in Figure 1.2. It is recognized that Naturalistic Driving Studies and Field Operational Tests have some common methodological aspects and that there are gradual transitions between FOTs and NDS.

Figure 1.2: A partially overlapping relationship between Field Operational Tests and Naturalistic Driving Studies along a continuum of experimental control.

A more generic methodological approach is emerging whereby naturalistic data collected by either NDS or FOTs can be used for similar purposes. For example, an NDS may be used to evaluate the impact of on-market intelligent safety functions (Antin et al., 2011) and an FOT may be used to study crash causation (Olson et al., 2009). Even though there may be differences in purpose between NDS and FOT studies, the technology for driver behaviour observation may be the same, and consequently experiences from FOTs have been important inputs to the planning of a large-scale European NDS (Sagberg et al., 2011).

Between the FOT and the NDS, lies the Naturalistic FOT. The Naturalistic Field Operational Test is defined as a study undertaken using unobtrusive observation in a natural setting, typically to evaluate the relationship between (permanent or temporary) driver-, vehicle-, or environment factors with crash risk, driving behaviour, and countermeasure effectiveness (Victor et al. 2010). This definition accommodates for both accident research oriented research of explanatory factors associated with crashes (common in NDS), as well as for the evaluation and development-oriented research on new technology and solutions (common in FOTs). N-FOT studies can for example, assess the relationship of an in-vehicle system (dynamic vehicle factor) or age (static driver factor) or distraction (dynamic driver factor) or speed cameras (static environment factor) with crash-risk, driving behaviour and/or countermeasure effectiveness. Environment sensing and video are believed to be essential for identifying near collisions and other incidents, and for validating that intelligent vehicle systems (e.g. collision warning, lane departure warning and intelligent speed adaptation) perform as expected.

1.1.3 NDS and the FESTA V

Although the FESTA V was originally developed as an implementation planning tool for FOT studies, it is a highly relevant and useful tool also for naturalistic driving studies. However, some modifications are required. These modifications are primarily related to the shift in focus away from function assessment in FOTs towards behaviour assessment in NDS. Broadly speaking, the bottom or tip of the FESTA V is most relevant for NDS and the top of the FESTA V is less directly relevant. Figure 1.3 Illustrates how the FESTA V is modified by removing four steps which are more relevant for FOTs.

The horizontal Context bar (Figure 1.3) is also useful for the NDS, it is recommended that activities which are dealing with the more general aspects of an NDS and with high level aggregation of the results take place within this horizontal Context bar. According to Sagberg et al. (2011) this type of activity includes:

• User/Stakeholder Identification,
• Topics Selection,
• Dissemination, and
• Identification of Constraints, such as available technologies and budgets.

The FESTA V has two “scaling up” steps: Impact Assessment, and Socioeconomic Cost Benefit Analysis. Naturalistic driving studies also have scaling up activities, such as aggregation of research questions results, and analysis of the implications of results. For example:

• Provide an integrated overview of the NDS findings.
• Use findings to identify new and more efficient safety and sustainability measures related to vehicles, drivers and road infrastructure (such as incentive to encourage adoption of new technologies, education of drivers, regulation enforcement).
• Identify ways for measures and tools to improve safety and sustainability of road transport in Europe based on NDS results.
• Demonstrate how Naturalistic Driving can be used in industrial development of safety and sustainability functions and services.

Although it can be debated whether these activities should be carried out in a separate step or box in the FESTA V, it is suggested that these activities should take place in the 'research questions and hypotheses testing step and the horizontal Context bar.

For further detailed examples of how the FESTA V can be interpreted for Naturalistic Driving Study purposes see Sagberg et al. (2011) and Victor et al. (2010).

Figure 1.3: Modification of the FESTA V to match Naturalistic Driving Study purposes