Difference between revisions of "FESTA handbook Introduction"
|Line 2:||Line 2:|
In Japan and in the United States Field Operational Tests ([http://wiki.fot-net.eu/index.php?title=FOT FOT]s)<ref>Key items have internal links and, when availbale, a link to the FOT-Net Wiki glossary</ref> have been introduced as an evaluation method for driver support [http://wiki.fot-net.eu/index.php?title=System systems]<ref>A System is defined as "a combination of hardware and software enabling one or more functions"</ref> and [http://wiki.fot-net.eu/index.php?title=Function functions]<ref>A Function is defined as "an implementation of a set of rules to achieve a specified goal"</ref> several years ago with the aim of proving that such systems can deliver real-world benefits. In Europe too, [http://wiki.fot-net.eu/index.php?title=FOT FOT]s have been conducted at a national or regional level, particularly on speed support [http://wiki.fot-net.eu/index.php?title=System systems] and lane departure warning [http://wiki.fot-net.eu/index.php?title=System systems]. These [
In Japan and in the United States Field Operational Tests ([http://wiki.fot-net.eu/index.php?title=FOT FOT]s)<ref>Key items have internal links and, when availbale, a link to the FOT-Net Wiki glossary</ref> have been introduced as an evaluation method for driver support [http://wiki.fot-net.eu/index.php?title=System systems]<ref>A System is defined as "a combination of hardware and software enabling one or more functions"</ref> and [http://wiki.fot-net.eu/index.php?title=Function functions]<ref>A Function is defined as "an implementation of a set of rules to achieve a specified goal"</ref> several years ago with the aim of proving that such systems can deliver real-world benefits. In Europe too, [http://wiki.fot-net.eu/index.php?title=FOT FOT]s have been conducted at a national or regional level, particularly on speed support [http://wiki.fot-net.eu/index.php?title=System systems] and lane departure warning [http://wiki.fot-net.eu/index.php?title=System systems]. These [FOT]s have proven to be highly valuable. Recently [http://wiki.fot-net.eu/index.php?title=FOT FOT]s have been identified as an important means of verifying the real-world impacts of new [http://wiki.fot-net.eu/index.php?title=System systems] at a European level and in particular to verify that European R&D has the potential to deliver identifiable benefits. This Handbook is the result of a joint effort of several research institutes, [http://wiki.fot-net.eu/index.php?title=FESTA_handbook_List_of_Abbreviations#OEM OEM]s and other stakeholders from across Europe to prepare a common methodology for European [http://wiki.fot-net.eu/index.php?title=FOT FOT]s. It is also highly relevant, and it is hoped useful, for [http://wiki.fot-net.eu/index.php?title=FOT FOT]s conducted at a regional or national level within Europe as well as outside Europe.
For the purposes of this Handbook, a “Field Operational Test” (FOT) is defined as:
For the purposes of this Handbook, a “Field Operational Test” (FOT) is defined as:
Revision as of 09:41, 15 January 2014
Back to FESTA Handbook main page: FESTA handbook
In Japan and in the United States Field Operational Tests (FOTs) have been introduced as an evaluation method for driver support systems and functions several years ago with the aim of proving that such systems can deliver real-world benefits. In Europe too, FOTs have been conducted at a national or regional level, particularly on speed support systems and lane departure warning systems. These FOTs have proven to be highly valuable. Recently FOTs have been identified as an important means of verifying the real-world impacts of new systems at a European level and in particular to verify that European R&D has the potential to deliver identifiable benefits. This Handbook is the result of a joint effort of several research institutes, OEMs and other stakeholders from across Europe to prepare a common methodology for European FOTs. It is also highly relevant, and it is hoped useful, for FOTs conducted at a regional or national level within Europe as well as outside Europe.
For the purposes of this Handbook, a “Field Operational Test” (FOT) is defined as:
|A study undertaken to evaluate a function, or functions, under normal operating conditions in environments typically encountered by the participants using quasi-experimental methods.|
This means that it must be possible to compare the effects that the function has on traffic with a baseline condition during which the function is not operating. In order to achieve this, the participants’ control over or interaction with the function(s) has to be manipulated by the research team. “Normal operating conditions” implies that the participants use the platforms during their daily routines, that data logging works autonomously and that the participants do not receive special instructions about how and where to drive. Except for some specific occasions, there is no experimenter in the vehicle, and typically the study period extends over at least a number of weeks.
The main purpose of this Handbook is to provide guidelines for the conduction of FOTs It walks the reader through the whole process of planning, preparing, executing, analysing and reporting an FOT and it gives information about aspects that are especially relevant for a study of this magnitude, such as administrative, logistic, legal and ethical issues. Another aspect of the Handbook is to pave the road for standardisation of some aspects of FOTs, which would be helpful for cross-FOT comparisons. It has to be kept in mind, though, that many traffic parameters in different European countries differ substantially.
In Figure 1.1 the steps that need to be carried out during an FOT are presented. They will be explained in detail in the different chapters of the Handbook. For orientation purposes, a copy of the figure is provided in the beginning of each chapter highlighting which step of the FOT . Chain is described in the current chapter. The FOT Implementation Plan takes up all the steps and integrates them into one big table which can be used as a reference when actually carrying out an FOT
Figure 1.1: The steps that typically have to be considered when conducting an FOT. The large arrows indicate the time line.
In order to make the picture more complete a horizontal bar should be added on top of the diagram that in principle summarises the context in which the FOT is supposed to take place. For instance, the choice of a function to be tested implies that there is either a problem that is to be addressed and that the chosen function is defined to solve the problem or that a policy objective is stated and that the function tested can be used to reach the objective. An FOT can always be related to a wider view on the exercise than is defined by just a description of the function to be tested.
This can be summarised as the first steps, which include setting up a goal for the study and selecting a suitable research team, and also the last steps that include an overall analysis of the systems and functions tested and the socio-economic impact assessment, dealing with the more general aspects of an FOT and with aggregation of the results. The further down on FOT Chain V-Shape the steps are located, the more they focus on aspects with a high level of detail, like which performance indicators to choose, or how to store the data in a database. The ethical and legal issues have the strongest impact on those high-level aspects, where the actual contact with the participants and the data handling takes place.
The representation of the FESTA methodology in the form of a V does not mean that designing and performing an FOT is always a linear process. Decisions made at a certain stage of the FESTA-V influence the next steps and it is inevitable to sometimes go back and redo some steps. Especially in the left-hand side of the V iteration may be necessary. For example, one may find out that the measures and sensors available do not make it possible to investigate the hypotheses defined earlier, so adjustments to the hypotheses or performance indicators may be needed. Also the right hand side of the FESTA-V may influence the decisions to be made at the left-hand side. The question of the socio-economic impact may influence the definition of functions, use-cases, research questions and other elements of the left-hand side. Regarding the resources available for data analysis may also lead to revision of the left-hand side. There is, however, the question “when does one stop the iterative process?” From a research perspective, this is a continuous process. However, from a project management perspective, boundaries have to be set to reflect budget constraints and timing aspects.
The first step in the FESTA-V is the identification of functions to be tested. Sometimes this may not be the best step to start with. For example, an FOT may not be driven by the technical systems that need to be tested but by a research question or an impact area. When there is a large set of functions available from which a few need to be selected as candidates for testing, definition of the research questions may help the selection process. For example if safety is to be investigated, other functions may have a higher priority than when traffic efficiency is the main focus.
In conclusion, the FESTA-V provides a static picture of the complex design and execution of an FOT . but in reality a more iterative process will be needed, with a starting point suited to the specific aims of a project.
The FESTA Handbook is not meant to be a substitute for consultations with experts, organising a good and capable research team, and carrying out specific investigations into the legal and ethical issues that apply to the current question and situation. It is not an exhaustive action list, and each FOT has its own special issues and concerns that have to be dealt with on an individual basis. Nor is the advice in it necessarily perfect and representative of the state of the art. On many issues, there will be scope for disagreement with the recommendations or use of alternative sources of advice. But it would certainly be preferable for major departures from the advice to be justified to funding agencies and major stakeholders.
The FESTA project consortium decided early in the project that the primary focus of the FESTA Handbook would be on the evaluation of Advanced Driver Assistance Systems (ADAS) and In-Vehicle Information Systems for vehicles — both in the form of autonomous systems and of Cooperative Systems. It was also agreed that the FESTA Handbook should be relevant to the evaluation of Original Equipment Manufacturer (OEM), aftermarket and nomadic systems. The Handbook is therefore designed specifically to guide the evaluation of such systems, and is less relevant to the evaluation of electronic road infrastructure such as Variable Message Signs (VMS). However, it will be seen that many of the activities identified in the Handbook are common to the evaluation of most vehicle- and infrastructure-based ICT technologies).
In conclusion, the FESTA Handbook gives an overview and general guidelines concerning the conduction of an FOT . FOTs are designed to participate to the solution of a problem, and this handbook is intended to provide a formalised and practical framework, and not a cook book: the methodology described will necessarily have to be adapted to the specific case, in order to increase the efficiency of the approach or tackle data incompleteness or inconsistency. Even more, the results of an FOT may have to be integrated with external sources of information, to achieve a wider perspective, and an increased relevance for tackling the problem at hand.
In addition to the Handbook itself, more detailed work, which was produced during the FESTA project, and which is referenced in the Handbook, is included in the annexes to the Handbook and the FESTA deliverables.
The Naturalistic Driving Study in relationship to the Field Operational Test and the FESTA methodology
The Naturalistic Driving Study (NDS) or observation is a relatively new research method using advanced technology for in-vehicle unobtrusive recording of driver (or rider) behaviour during ordinary driving in traffic. This method yields unprecedented knowledge primarily related to road safety, but also to environmentally friendly driving/riding and to traffic management. A central focus in naturalistic driving studies is to understand explanatory factors associated with crashes and predict involvement in crashes. The naturalistic method “refers to a method of observation that captures driver behavior in a way that does not interfere with the various influences that govern those behaviors.” (Boyle et al., 2009). Naturalistic driving studies are defined as “those undertaken using unobtrusive observation or with observation taking place in a natural setting” (Dingus et al., 2006). The following characteristics have been chosen to define the Naturalistic Driving approach in the recent EU project PROLOGUE (Sagberg et al., 2011):
- Unobtrusive recording of driver and vehicle parameters
- Normal driving, i.e. driving purpose and driving destinations as defined by the driver, and driving taking place on roads open to ordinary traffic, and with the vehicle that the driver normally uses (owned, leased, or company vehicle)
- No observer present in the vehicle
Relation to FOTs
Naturalistic Driving Studies tend to focus on crash-explanatory factors, and Field Operational Tests generally focus on evaluation of systems or functions, however the collected data in both types of studies can be used for many alternative purposes such as analysis of Environment, Efficiency and Mobility impacts. NDS and FOTs are preferably seen as different methods because (a) the study design is different (participant selection, experimental conditions, vehicle sample etc), and (b) the research questions and hypotheses are different. In particular, the main difference relates to degree of experimental control found in the study, as illustrated in Figure 1.2. It is recognized that Naturalistic Driving Studies and Field Operational Tests are merging methodologically and that there are gradual transitions between FOTs and NDS.
Figure 1.2: A partially overlapping relationship between Field Operational Tests and Naturalistic Driving Studies along a continuum of experimental control.
A more generic methodological approach is emerging whereby naturalistic data collected by either NDS or FOTs can be used for similar purposes. For example, an NDS may be used to evaluate the impact of on-market intelligent safety functions (Antin et al., 2011) and an FOT may be used to study crash causation (Olson et al., 2009). Even though there may be differences in purpose between NDS and FOT studies, the technology for driver behaviour observation may be the same, and consequently experiences from FOTs are important inputs to the planning of a large-scale European NDS (Sagberg et al., 2011).
Between the FOT and the NDS, as in, lies the Naturalistic FOT. The Naturalistic Field Operational Test is defined as a study undertaken using unobtrusive observation in a natural setting, typically to evaluate the relationship between (permanent or temporary) driver-, vehicle-, or environment factors with crash risk, driving behavior, and countermeasure effectiveness (Victor et al. 2010). This definition accomodates for both accident research oriented research of explanatory factors associated with crashes (common in NDS), as well as for the evaluation and development-oriented research on new technology and solutions (common in FOTs). N-FOT studies can for example, assess the relationship of an in-vehicle system (dynamic vehicle factor) or age (static driver factor) or distraction (dynamic driver factor) or speed cameras (static environment factor) with crash-risk, driving behavior and/or countermeasure effectiveness. Environment sensing and video are believed to be essential for identifying near collisions and other incidents, and for validating that intelligent vehicle systems (e.g. collision warning, lane departure warning and intelligent speed adaptation) perform as expected.
NDS and the FESTA V
Although the FESTA V was originally developed as an implementation plan for FOT studies, it is a highly relevant and useful tool also for natualistic driving studies. However, some modifications are required. These modifications are primarily related to the shift in focus away from function assessment in FOTs towards behavior assessment in NDS. Broadly speaking, the bottom or tip of the FESTA V is most relevant for NDS and the top of the FESTA V is less directly relevant. Figure 1.3 Illustrates how the FESTA V is modified by removing four steps which are more relevant for FOTs.
The horizontal Context bar (Figure 1.3) is also useful for the NDS, it is recommended that activities which are dealing with the more general aspects of an NDS and with high level aggregation of the results take place within this horizontal Context bar. According to Sagberg et al. (2010) this type of activity includes:
- User/Stakeholder Identification,
- Topics Selection,
- Dissemination, and
- Identification of Constraints, such as available technologies and budgets.
The FESTA V has two “scaling up” steps – Impact Assessment, and Socioeconomic Cost Benefit Analysis. Naturalistic driving studies also have scaling up activities, such as aggregation of research questions results, and analysis of the implications of results. For example:
- Provide an integrated overview of the Naturalistic driving study findings;
- Use findings to identify new and more efficient safety and sustainability measures related to vehicles, drivers and road infrastructure (like incentives w.r.t. new technologies, education of drivers, regulation enforcement).
- Identify ways for measures and tools to improve safety and sustainability of road transport in Europe based on naturalistic driving study results.
- Demonstrate how Naturalistic driving can be used in industrial development of safety and sustainability functions and services
Although it can be debated whether these activities should be carried out in a separate step or box in the FESTA V, it is suggested that these activities should take place in the 'research questions and hypotheses testing step and the horizontal Context bar.
Figure 1.3: Modification of the FESTA V to match Naturalistic Driving Study purposes
- Key items have internal links and, when availbale, a link to the FOT-Net Wiki glossary
- A System is defined as "a combination of hardware and software enabling one or more functions"
- A Function is defined as "an implementation of a set of rules to achieve a specified goal"