FESTA handbook From Functions to Hypotheses

Back to FESTA Handbook main page: FESTA handbook

4. From Functions to Hypotheses

The final objective of an FOT is to evaluate in-vehicle functions based on Information Communication Technology (ICT) in order to address specific research questions. These research questions can be related to safety, environment, mobility, traffic efficiency, usage, and acceptance. By addressing the research questions, FOTs promise to furnish the major stakeholders (customers, public authorities, OEMs, suppliers, and the scientific community) with valuable information able to improve their policy-making and market strategies. Individuating the most relevant functions and connected hypotheses to successfully address the above-mentioned research questions is one of the major challenges in an FOT . In this Chapter, the process of individuating the functions to be tested in an FOT and the relevant connected hypotheses will be elucidated. Specifically, the reader will be guided in the process of

1) selecting the functions to be tested,

2) defining the connected use cases to test these functions,

3) identifying the research questions related to these use cases,

4) formulating the hypotheses associated to these research questions , and

5) linking these hypotheses to the correspondent performance indicators.

The FOT chain shows specifically the steps reported above.

The steps may be influenced by other elements of the FOT chain. The selection of functions may be driven by the socio-economic impact that is expected, or by the research questions. When details are filled in later on in the process, it may be necessary to re-visit earlier steps, for example limits in resources or technical capabilities may lead to a decision to limit the amount of functions, use cases and hypotheses.

---

Applying this Process to Naturalistic Driving Studies This chapter is written from the perspective of FOTs on various kinds of systems. By contrast, NDS investigate driving itself without the constraints of the experimental conditions that are normally required in FOTs. In FOTs, there is a natural progression that starts with a specification of the functions to be evaluated, and then moves on to the environments and situations in which the tests will be conducted and the experimental design that will be employed. The specified research questions follow in a natural progression, and, in forming the hypotheses, there is generally an anticipation of the direction of change, e.g. that a driver assistance system will promote safety or fuel economy. In framing hypotheses for FOTs, we seek to gain a deeper understanding not only of what changes occur but of how they occur, i.e. of how a particular function influences user behaviour. By contrast NDS are much more exploratory. They aim for deeper understanding of driving behaviour as it relates to safety and in some cases environmental aspects of driving. They can thus be likened to other observational studies in traffic. Of course there is a kind of experimental design even in NDS: the vehicles and participants have to be selected and the choices made here will be determined by the focus of the study and the research topics being addressed. NDS focus on how drivers manage their driving in changing situations and environments and on how breakdowns in the safe operation of vehicles come about. Hence they are primarily interested in why problems occur and do not occur. This makes research questions, as opposed to hypotheses, the natural focus of NDS. And those research questions will tend to be initially quite broad — what is the relationship between distraction and safety-critical events or why do young drivers have a problem with loss-of-control on curves at night-time. The RQs can then be further specified in the form of sub-RQs and so on. From those sub-RQs, appropriate performance indicators and measures can be defined. This then leads to the specification of the Data Acquisition System and associated subjective and objective data. One of the challenges in NDS is prioritising the research questions. It is comparatively easy to generate hundreds of RQs and sub-RQs. The difficulty then is likely to be setting the boundaries of the study by determining which ones are high priority and can be addressed within the resources available. However, if the final database is sufficiently broad and flexible, it can become a valuable resource for the investigation of additional topics and RQs. Those engaged in NDS may benefit from consulting some sections of this chapter, such as the material on situations. Chapter 5 is generally applicable to NDS.

---

4.1 Systems and functions

In the last few years, the number of ICT functions available for use on standard vehicles and more generally while travelling has been rapidly increasing. ICT functions are intrinsically designed to provide the driver or traveller with new, additional information. However, the extent to which this increased amount of information from these ICT functions results in clear and positive effects on safety, environment, mobility, usage, and acceptance in real traffic situation is unknown. FOTs warrant to evaluate, for the first time, these ICT functions in a real traffic situation during naturalistic driving. In this handbook we refer to 1) in-vehicle, 2) cooperative, and 3) nomadic systems intended as a combination of hardware and software enabling one or more ICT functions. Depending on the different systems implementing a specific function, different challenges may have to be faced during the FOT design.

It is important to note that NDS in the future will increasingly include new systems as they increase in market penetration. For example, future NDS will be able to be used for with-and-without analyses of systems that are new today, but commonplace tomorrow (e.g. Forward collision warning may be as commonplace as ABS tomorrow).

4.1.1 Vehicle systems

Vehicle Systems are a combination of hardware and software enabling one or more functions aimed at increasing driver’s safety and comfort. Vehicle Systems promise

1) to increase road safety by increasing driver’s attention in potentially hazardous scenarios (such as the Forward Collision Warning),

2) to improve the driver’s comfort by automating some of the operational driving tasks (such as the Adaptive Cruise Control function),

3) to increase driver mobility by furnishing timely traffic information (such as the Dynamic Navigation function), and

4) to increase safety in critical situation by automating the vehicle response (such as the Collision Mitigation function).

Vehicle systems are becoming more and more standard equipment, even in middle class vehicles and commercial vehicles. However, their impact on the driver, the traffic system, the society, and the environment in the short-, but especially in the long-term is not fully understood. FOTs can help quantifying the impact of vehicle systems on driver’s workload and to understand how different functions interact with each other in a real complex traffic situation. Further, FOTs will expose these functions to many improbable scenarios which are not possible to be tested during the functions evaluation phase.

4.1.2 Cooperative systems

Cooperative Systems are vehicle systems based on vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I) and infrastructure-to-vehicle (I2V) communication technology. Communication technology has enabled a new class of in-vehicle information and warnings which are more precise in terms of time and location. It is foreseen, that the integration of Cooperative Systems with autonomous functions will provide a new level of ADAS. Infrastructure based information tells the driver for example what is the appropriate speed to keep on a specific part of road or warns the driver in case of ice on the road or fog.

One of the initial objectives of V2V, V2I, and I2V is to increase road safety. The development of safety-critical V2V systems in Europe has been mainly promoted by the Car-to-Car Communication Consortium (C2C-CC). More recently, the EC has placed emphasis on the application of Cooperative Systems to achieve environmental and efficiency impacts.

Currently (January 2014), the major stakeholder umbrella organizations are working together to achieve the objective, stated in the MoU signed by the car manufactures inside (C2C-CC), to start real life application of cooperative systems by 2015.

A key point in Cooperative Systems is standardisation. The minimum set of standards needed for Day1 implementation and requested by the Mandate (N. 453) issued by European Commission are going to be finalised during 2014 by ETSI and CEN. Activities for harmonisation with U.S. and Japanese standards are on going. (see the COMeSafety Website for information). Cooperative Systems differ in several areas to conventional areas, which directly influences the FOT planning and operation.

Firstly, there are currently no Cooperative Systems available in the vehicle market – this implies that all participating vehicles in the FOT have to be equipped manually with prototype systems. This is costly and makes it difficult to find suitable test vehicle fleets. Also, drivers are not aware of such and have no knowledge of their functionality. Before starting the FOT , drivers have to be educated on the usage of Cooperative Systems.

Another difficult issue lies in the essence of cooperation: all V2V functions rely on more than one vehicle being in communication range, and V2I relies on roadside-stations. Before starting the FOT , it is hard to estimate the number of vehicles in one area and how often vehicles will pass a Road-Side Unit (RSU). This penetration rate is a crucial factor in designing and evaluating an FOT .

Current estimates from recent V2X FOTs predict a minimal penetration rate of 10% for V2X functions to show a noticeable effect on traffic safety and efficiency. Given the size of the FOT area and the average distribution of vehicles, the number of participating vehicles should usually be considerably higher.

The penetration rate also influences the Frequency of events (FoE). In Cooperative Systems FOTs, the FoE is roughly related to the number of necessary vehicles for a function. While some functions work with only two vehicles (e.g. a slow vehicle warning), other functions require several more (e.g. traffic-jam ahead warning). For certain functions the combined frequency of events might make a naturalistic FOT unfeasible.

A proper assessment of the penetration rate impact can be derived from dedicated simulations. A simulation environment should consider traffic effects, communication models, applications and their respective influence on each other. Such dedicated simulation environments are able to predict the frequency of events for the developed functions and thus support the FOT design and setup process greatly in Cooperative Systems.

There are several implications derived from this effect to the test setup and execution of Cooperative Systems FOTs:

• The number of equipped vehicles is considerably higher or the FOT needs to run considerably longer to collect enough data to be representative.
• It might be very difficult or even not possible to run the FOT uncontrolled for some functions(e.g. Emergency electronic brake light,..) FOT. Controlled testing implies the usage of dedicated tools to specify the test cases, the test scenarios and to run the test. They also need to be linked to the vehicles and drivers to control and monitor the running test. (See 6.5 for more details)

It is most likely that the initial market drive will be for V2I-based applications until there is a critical mass of equipped cars on the road. FOTs can assess the technological and business feasibility of Cooperative Systems and may be necessary to complete the validation of such systems. In fact testing to validate Cooperative Systems may require very complicated set-up which may not be possible unless in a real-traffic set-up.

One important issue is the installation of infrastructure devices that could be needed. Permission for the testing and team involvement should be planned in advance considering the impact on normal traffic during the installation phase. In the testing phase proper logistic support is needed in order to allow technical people to operate with debugging equipment in order to tune the systems. Assessment of RSUs and antennas positioning could be rather time-consuming. Protection and maintenance of road-side equipment should be ensured.

A specific complexity of Cooperative Systems is due to the distributed nature of involved components. It is not only sufficient to log the vehicle state and the functionstate, but to completely evaluate the functionality of a Cooperative Systems it is also necessary to handle all incoming and outgoing V2X and the Local Dynamic Map state on the vehicle side. These data are provided to or received from other nodes of the V2X network (a node is defined as ITS Vehicle Station or ITS Roadside station depending on the location) directly influence the system functionality. Consequently, the scope of logging has to be widened to cover the data provided by ITSStations located in the relevant geographical areas. The logging system has to be able to cope with the broader scale both in ITS station and ITS area scope.

Each ITS needs to be equipped with a logging device. This device or software component has to be able to write a common log destination of data provided by all important internal sources and the network:

• Vehicles Data (CAN Bus)
• Position and Time (if not present on CAN)
• Networking: Incoming and Outgoing Messages (CAM, DENM)
• Local Dynamic Map Status (List of all neighbour stations, List of all events)
• function Status (to monitor how applicants are working)
• Other Facilities (connection to backend-services, etc.)

It is obvious, that all these various sources provide a huge amount of data. This is a major consideration for the logging system on each station and should influence it in two ways:

• Qualitative: Each log entry should be reduced to the minimum (entry type ID, timestamp, value). It is possible to further reduce the required space, e.g. by only writing the time and not the date into each entry. It can also be useful to compress the written files. It is strongly advised to use a “strict” logging scheme, which only allows pre-defined log entry types, as this simplifies post-processing and also allows saving disk space.
• Quantitative: Logging does not have to record all values all the time. Although data storage cost is lowering more and more, if necessary, it is possible to reduce the used space drastically by using pre-defined logging profiles. These are generated with a mapping from those performance indicators needed for evaluation to the available measurements. For each test a logging profile can be pre-selected, which contains only those measurements needed.

In more complex settings the logging system can also automatically switch profiles depending on the measures values (a critical situation can be detected from measurements and a wider logging profile can be selected for a given time). It is also possible to select the log profile directly in controlled testing to guarantee that all necessary measures are logged. The quantity of data to be collected is discussed in section 2 of the Chapter on Data Analysis and Modelling where two approaches (“space mission” and “minimum data set collection”) and related issues, are presented.

The ITS Roadside Station provides similar measures (except vehicle data) and it is advised to use a similar architecture for the logging system. In contrast, the measures available to the ITS Central Station can differ widely and are specific to each FOT. Roadside sensors may be needed to provide information related to interaction with non-connected vehicles and other situational variables.

For controlled testing scenarios it is advisable to include monitoring features in the logging component. Monitoring is a real-time transfer of selected information available to the logger. In a central counterpart to monitoring this information is displayed to visualize the current test progress. Typically the monitoring value set is a subset of logging and should at least include vehicle ID, position, heading and speed. In this minimal set the test operator can follow vehicle progress through the test. More advisable monitoring sets include direct information from developed functions to monitor in real-time, if the test intent (to trigger the developed functions in a specified way) has been met. Test operators can thus determine if a test was successful (and the detailed logs are valuable for evaluation) or if tests have to be repeated right away. Sophisticated test control and monitoring tools support operators in this task.

Log files in Cooperative Systems FOTs can quickly take up gigabytes of data. A rough estimate for a typical Cooperative Systems is 500Mb per day per vehicle. It is important to design the test and logging system to cope with this data amount. Log files have to be saved locally on vehicle and roadside station, have to be transferred to the storage and have to be integrated into a central database. Throughout this process it has to be assured, that no data can be lost due to a single point of failure.

The transfer of data from vehicles to the central storage might exceed possible bandwidth from 3G connections. Other possible ways include transfer over Wi-Fi and manual transfer by USB Stick or Drive. Log data should be compressed to speed up transfer times. Depending on the vehicle availability (average run-time over day, time in between access to vehicles) one solution has to be chosen and the system has to be specified in a way, that the in-vehicle storage can fit enough data to hold all log-files until extraction. When large international FOT are planned, Cooperative Systems may require interoperability tests. The costs, the organizational and legal aspects (prototypes in foreign countries) should not be underestimated when planning these testing sessions.

4.1.3 Nomadic devices

The use of so-called Nomadic Devices for transport and traffic related applications has become increasingly commonplace in the last few years. The first wave of such devices were dedicated Satnavs, also known as Personal Navigation Devices (PNDs). The second wave of functions have been a large variety of “apps” for Smartphones. In addition some PND suppliers have formed alliances with vehicle manufacturers so that their products may be fitted as original equipment. Whether fitted as original equipment or functioning on Smartphones, the hardware platforms tend to operate autonomously of the vehicle. Indeed many of these devices can provide traveller functionality — for pedestrian route-finding, for public transport information, to locate points of interest — outside the vehicle.

The functionality of PNDs and Smartphones has been evolving over time. Initially, the functionality of PNDs was focussed on guiding the driver from geographic point A to a point B, using buttons and the screen to set the desired destination. That functionality has now evolved so that the devices may offer a media player, media viewer, mobile phones integration, and may be voice activated. They can also provide routing and navigation support outside the vehicle. Smartphones started from an emphasis on telephony, supported by additional functions such as an address book and texting. Additional functions(photography, navigation, radio, media centre, etc.) were then continuously added. Now that GPS is generally provided, they are navigation-capable, so that navigation apps are commonplace. On the other side, PNDs are broadening in capability and now tend to include a mobile phone SIM card. Thus there is a large overlap in capability and functionality.

This provides a capability to download map updates but also gives cooperative-system-like functionality in the form of live traffic updates and other environmental information. The same mobile phone link can be used by service providers to acquire real-time traffic conditions. Smartphones, with their built-in networking capability, can offer the same functionality.

As well as links to mobile networks, Nomadic Devices are now available with Bluetooth, Wi-Fi networking, assisted GPS, speech recognition and high-end operating systems with the capability to provide dynamic geo-referenced applications to assist the driver and traveler.

Specifically tailored aftermarket devices, sometimes offered by the same manufacturers as personal PNDs, are now targeted at the fleet market for management of logistics and other fleet functions. Some of these devices provide links to CAN data. An increasing focus of such systems is fuel economy and feedback to drivers on their efficiency in driving. Another sector for aftermarket devices is the pay-as-you-drive insurance market.

Nomadic Devices need to be evaluated from the perspectives of user behaviour and acceptance, safety (particularly in regard to HMI issues), travel and traffic impacts and environmental implications. It also needs to be recognised that such devices can have broad mobility implications, both in terms of the strategic level of driving (route choice) and in terms of trip generation and mode choice. Any evaluation of usage needs to consider the potential for both in-vehicle and out-of-vehicle usage of these devices.

One of the critical characteristics of PNDs and Smartphones is how far the device is integrated within the vehicle. Many devices use specific mounting kits for in-vehicle installation (connection to power supply, GPS and in some cases the vehicle’s audio system). Typically PNDs are mounted with a suction cup directly to the windscreen, while cradles for smartphones may also attach via a suction cup. As a result they may impede the driver’s field of view increasing the risk of accidents. There is also often a problem with small screen size and inadequate audio volume. Nevertheless, the popularity of these devices has not been affected. To increase usability and reduce negative side effects, an Automotive Bluetooth profile has been developed so that such devices can use the vehicle’s built-in HMI. This can offer improvements both in user input to the device (stalks, buttons, speed recognition, link to hands-free mobile phone, etc.) and in device output (using integrated audio, and screen).

The general ease of use of a device will have a major influence on acceptance and willingness to pay. Here ease of use refers not just to the usability while driving but to the user experience in all aspects of usage — pre-trip, in-trip and post-trip. Post-trip functionality is very relevant to usage in the fleet market and to support for and feedback on eco-driving.

Due to the above-mentioned fast innovation cycle, FOTs studying Nomadic Devices may require state of the art planning in order to keep up with the introduction of new features and functions. They will also need to consider the surrounding infrastructure since (rather like many Cooperative Systems) many functions rely on information and support from the outside. Weather forecasts, traffic information, updates on road conditions, dynamic speed limit information and speed advice are all dependent on service providers.

4.1.4 Combinations of functions

There are many FOTs which investigate the impacts of a combination of functions sometimes because systems and functions come in a bundle. One such common bundle is the combination of Adaptive Cruise Control and Forward Collision Warning. Both functions make use of the same sensors, and indeed second-generation ACCs generally implement a warning function to indicate to the driver when the deceleration demanded of the ACC in order to prevent a collision with the preceding vehicle is beyond the function’s designed capability. In other cases, an FOT may be investigating a function that resides on a platform which offers many other functions. This is almost invariably the case when studying functions that reside on Nomadic Devices such as a smartphone or SatNav. In some cases a project will create a new function and provide to users on a standard consumer Nomadic Device. It is not practical or reasonable to demand of users that they do not use the full functionality of the device and attempts to disable features may well annoy participants.

Therefore in planning the evaluation, it is important to consider how functions may interact with each other and how those interactions might affect user behaviour. This needs to be done at the stage of an FOT when research questions and hypotheses are initially formulated. What needs to be considered is:

• Can the effects of the various functions be disentangled? Note that it may not be possible or feasible to do so, particularly if the functions are closely coupled together.
• Does the experimental design need to be modified to enable both the single effects of each functionto be investigated as well as the effect of the functions in combination?

Some systems are now so integrated that it is no longer possible to disentangle them completely. An ACC that does not incorporate FCW is no longer on offer. An ACC + FCW system can be driven with the ACC off but cannot be driven in ACC mode with FCW off. Indeed disabling the FCW functionality within an ACC could be considered unethical and may be impossible for practical reasons.

Where bundles cannot be disentangled, it is only possible to investigate and report on the impacts of the combined systems. But when, for example, there are separate apps on a Nomadic Device, then consideration needs to be given to how those apps might interact. The investigation can be carried out in a naturalistic manner — i.e. the participants are free to choose when and when not to use the applications and functions that are not the immediate focus of the FOT. Alternatively, it can be carried out in a more experimental manner by means of instructions to participants, systematically enabling and disabling the functions, or carrying out controlled drives with the functionality of the system(s) carefully manipulated.

In formulating hypotheses, it is useful to think both of the singular effects of a system or function and of the synergistic effects that one function may have on another. Thus the recommended procedure is to start with the individual functions and then to proceed to combinatory effects. The process should therefore be to:

1. Begin separately with the individual functions, generating a list of research questions and hypotheses
2. Examine commonalities and conflicts between the systems and functions and derive hypotheses from those commonalities and conflicts:
   1. Can they generate simultaneous messages and/or warnings?
   2. Do they have a common interface and can they both be activated simultaneously?
   3. Are the same performance indicators relevant to each system and/or function?
   4. Are there common factors influencing usage?
3. Distinguish between hypothesis additive effects when the two systems interact with each other and multiplicative effects when the presence of a second system will alter the effects of the first. Additive (or subtractive) effects mean that the size of the effects will change. Multiplicative effects mean that the relationship is different, i.e. that there is an interaction in statistical terms. Consider situations in which such multiplicative effects might be important.

The application of this procedure should produce a comprehensive set of hypotheses on how the functions might interact and should affect the subsequent experimental design. As in all such work on the preparation of research questions and hypotheses, the reasonableness of pursuing every possible combination in a structured experimental design needs to be considered. Any additional function can impose a huge cost in terms of the increase in the number of possible combinations. It can be argued that there is a rationale also for multiple baselines, i.e. with all functions off, and with one function at a time off.

Complex experimental designs have large practical costs associated with them and the benefits of such designs need to be carefully considered. The costs can be in the form of the number of different baselines that may be required and of the time needed for data collection on each combination. A full experimental design in which ordering effects are considered may well be totally impossible for practical reasons. This can all lead to excessive time required for FOT execution. But another side effect can be the sheer difficulty of getting the participants to comply with all the different conditions of the experimental design. These are arguments for using a more naturalistic approach in which the participants are free to use whatever combination of systems and functions they choose. Of course system and function state will need to be recorded. This naturalistic approach has some disadvantages:

• Not all combinations may occur and not all participants may experience each combination
• It may be hard to take care of seasonal effects
• There may be insufficient sample sizes in some conditions so that experimental power is inadequate

However, the naturalistic approach also has advantages in that:

• Participant compliance will generally be assured
• The frequency with which the various combinations are used can provide useful information for the scaling-up process

Alternatives which offer greater efficiency than a full experimental design can be proposed. Laboratory experiments on driving simulators can be adopted to examine synergistic effects and a priori analysis can be applied at an early stage in an FOT to identify combinations that are of particular interest and which should therefore be the focus of attention. This can lead to a satisfactory but incomplete experimental design.

4.2 General methodology

The main advantage of an FOT is that it has the potential to give insight into system performance in naturalistic driving situations, as free as possible from any artefact resulting from noticeable measurement equipment or observers in the car. Therefore the first step when planning an FOT is to identify systems and functions where considerable knowledge about their impacts and effects in realistic (driving) situations is of major interest, but is still lacking (see Section 1.2.1). Another domain for FOTs is the area of systems and functions which need a certain penetration rate to work at all, like especially Cooperative Systems.

After the identification of the functions and system, which should be tested in an FOT . the goal is to define statistically testable hypotheses and find measurable indicators to test the hypotheses. To reach this goal, several steps need to be taken, starting from a description of the functions down to an adequate level of detail (see Section 4.2.1). This means that the main aspects of the functions, its intended benefits and the intrinsic limitations have to be described to fully understand objectives and limitations and to derive reasonable use cases.

Secondly, these use cases need to be defined (see Section 4.2.2). Use cases are a means to describe the boundary conditions under which a function is intended to be analysed. A general starting point is given by the functional specifications from the function description part. But it might also be of interest how a function performs when certain preconditions are not met and to identify unintended and unforeseen effects.

Starting from the use case definitions specific research questions need to be identified (see Section 1.2.3). Research questions are general question to be answered by compiling and testing related specific hypotheses. While research questions are phrased as real questions ending with a question mark, hypotheses are statements which can either be true or false. This will be tested by statistical means (see Chapter Data Analysis and Modelling). One might already have a very clear idea from the beginning which hypotheses are to be tested in a very specific situation during the FOT . However, this very focused view might result in an extreme limited experimental design, where important unintended effects will not be considered. The process to define hypotheses developed in FESTA aims to prevent these potential issues (see Section 4.2.4).

Finally, hypotheses can only be tested by means of reasonable indicators (see Section 4.2.5).

These steps are shown as parts of the complete FOT and are elaborated further in the following sections. FESTA deliverables D3, D4 and D5 provide additional detail on the application of the FESTA methodology to identify functions and systems and to develop hypotheses for the experimental design. All steps, from the description of the systems and functions, the development of use cases and scenarios, as well as the research questions and hypotheses and the proposal of related performance indicators are covered.

In the FESTA methodology, function description is a starting point. However, this may pose some problems. Not all FOTs are testing one pre-defined function; sometimes a set of functions or systems are to be evaluated. Or in [ naturalistic driving studies], the focus of the study may not be on functions but on driving behaviour in general. Stakeholders may have different ideas about the functions they want to test, and function descriptions are not always clear.

One issue is whether a function is generic or manufacturer-specific. In other words, how far should the particular manufacturer’s specification and functionality be considered. The specifics of the function can clearly have an impact on acceptance and behaviour. An example here might be an LDW that gives auditory and visual warnings and a different one that provides haptic feedback through the steering wheel. It is possible that one design will turn out be more effective than the other.

When it is possible to start with a clear function description, this will allow a detailed planning of the data collection plans, of the experimental design and hence of the costs. But it is of course still necessary to further specify research questions, hypotheses and performance indicators. In the case that functions are not well-defined or a decision has not made before the start of the project as to which functions to investigate, it may be more advisable to start with defining the research questions and select functions that seem most useful in answering these questions.

The issue of how many functions can be investigated in an FOT is a matter of the resources to be deployed and of whether the impacts of each function is to be investigated separately or alternatively whether the functions are to be treated as a package. Multiple functions can have interaction effects with each other and combinations of functions can therefore have impacts which are not simply the sum of the individual effects. On the other hand it may not be feasible to get the functions to work separately — for example FCW is now generally provided in combination with ACC so that when using ACC it is not possible to switch off FCW (see section 1.1.4).

4.2.1 Step 1: Selection and description of functions

Usually it is quite clear from the beginning what functions or at least what type of functions will be the object of an FOT . However, to select the specific functions but also in case the type of functions has not yet been decided, a Stakeholders Analysis is recommended. During this analysis, the needs of the different stakeholders need to be identified and merged into a common requirements description. Stakeholders are those whose interests are affected by the issue or those whose activities strongly affect the issue, those who possess information, resources and expertise needed for strategy formulation and implementation, and those who control relevant implementations or instruments, like customers, public authorities, OEMs, suppliers, and the scientific community. It is of vital importance that all relevant stakeholders are included in the analysis to guarantee that the selection process will not itself bias from the beginning the appraisal of the gained results.

It is recommended to evaluate the stakeholders’ needs by means of questionnaires, workshops or well documented interviews of stakeholders’ representatives. It is also quite important to describe the selection process sufficiently to prevent from misjudgment.

The basis for all following steps is a sufficient description of the selected functions. For these purposes it is suggested to collect the necessary information into a spreadsheet, structured in two parts:

• A first one, the functional classification, where a short high level description of the main aspects of the function should be given. This information is usually provided through the system specifications given by the system vendor or OEM.
• The second part of the description comprise limitations, boundary conditions and additional information which is necessary to understand how the function works.

The boundary conditions part should describe where and under which circumstances the system/ function will operate according to its specifications, where the FOT should take place and which type of data needs to be recorded during the FOT to enable a good interpretation of the results. It consists of:

• Infrastructure requirements, Cooperative Systems and Nomadic Devices requirements. Here all required actors besides the actual system need to mentioned, which might have an impact on system performance, service availability or similar. It is intended to trigger the consideration of factors which are external to the system/ function under evaluation;
• Demographical requirements / driver requirements: Especially the user or driver recruitment needs to take into account, whether a function is particularly designed for a specific group of users or drivers. Drivers differ on a large variety of characteristics, which may all have an influence on how they drive and use different systems and services. These differences may be important to take into account when planning an FOT. Four categories of driver characteristics may be distinguished:

-Demographic characteristics: gender, age, country, educational level, income, socio-cultural background, life and living situation, etc.;

-Driving experience, and driving situation and motivation: experience in years and in mileage, professional, tourist, with or without passengers and children etc.;

-Personality traits and physical characteristics: sensation seeking, locus of control, cognitive skills, physical impairments or weaknesses etc.;

-Attitudes and intentions: attitudes towards safety, environment, technology etc.

• Geographical Requirements/ Road Context: This description is necessary for systems which, concerning their functionality, depend strongly on the horizontal or vertical curves of the road layout or on the road type. For example, certain speed limit information systems depend largely on the availability of speed limit information in a digital map, which is up to now only commercially available on high class roads.
• Geographical Requirements/ environmental restrictions: Certain systems are especially designed for specific environmental conditions or, on the other hand, specifications might indicate that the system under evaluation will not work under certain environmental conditions. In this case the location of the FOTneeds to be selected carefully and the relevant data must be recorded during the FOT. For example, most of the functions using perception system will be affected by adverse weather conditions. If this is the case it is necessary to log relevant data and take it into account for later data analysis.
• Geographical Requirements/ Traffic Context: The performance of certain systems might depend on the traffic context, that is, the traffic density (e.g. given by the Level of Service) or might even be designed to work in specific traffic densities only. Like the other geographical requirements, this needs to be taken into account when an FOT is planned, performed and the data is analysed.
• Other Limitations: All other limitations need to be mentioned, which might have considerable impact on the performance of functions or systems, since these limitations have major impact on the experimental design and data analysis.

4.2.2 Step 2: Definition of use cases and situations

FOTs will typically test technically mature ICT systems. Therefore, systems and functions to be tested are on the market or close to market and can be easily implemented. But the list grows too long if all possible implementation variations and technologies are considered separately. The use cases put the systems and functions at a suitable level of abstraction in order to group technology-independent functionalities and answer more holistic research questions described later.

Table 4-1: Use Cases, Situations, Scenarios, and their mutual dependence.

A use case is a textual presentation or a story about the usage of the system told from an end user’s perspective. Jacobson et al. (1995) defined use cases as follows: “When a user uses the system, she or he will perform a behaviourally related sequence of transactions in a dialogue with the system. We call such a special sequence a use case.” Use cases are technology-independent and the implementation of the system is not described. Use cases provide a tool for people with different background (e.g. software developers and non-technology oriented people) to communicate with each other. Use cases form the basic test case set for the system testing. There are number of different ways to define a use case. Use cases in FESTA are very general descriptions, like e.g. “car following”. This general description needs to be refined to a reasonable level of detail. This refinement is done by describing so called situations (see Table 4.1). It is the detailed scenario description which triggers the development of specific hypotheses for later analysis.

The situational descriptors are selected in a way that relevant information can be gathered to distinguish between main differences while evaluating systems. The situational descriptors can be distinguished as static and dynamic, where static describe attributes which will not change significantly during one ride of the vehicle, such as age or gender of the driver. Nevertheless this information needs to be stated, since it is one of the main inputs to filter the huge amounts of data in the later stage of data analysis. The second type of attribute is dynamic, since it can change during a ride of the vehicle, such as the system action status (system on or off), the traffic conditions, road characteristics or the environmental situation.

The situations are defined as a combination of certain characteristics of a use case. situations can be derived from use cases compiling a reasonable permutation of the use cases characteristics. The identification of possible situations is covered from three viewpoints:

1systems and vehicle specification,

2environmental conditions specification and

3driver characteristics and status specification.

The situational descriptors in FESTA conform to the following structure:

IDENTIFICATION AND DESCRIPTION
Use case name	A name for identification purposes.
Description	General description of the use cases with necessary depth of information to get a quick overview.
Occurrence	Information about the anticipated quantity of occurrences has implications for the amount of data to analyse.
SYSTEMS AND VEHICLES
System status	Depending on the hypotheses the analysis might concentrate on situations where the system is activated or present. Example: ON/OFF (baseline) or IDLE/ON/OFF
System action status	Depending on the hypotheses the analysis might want to compare the driving performance between different system statuses, e.g. whether the system is actively controlling the vehicle or not. Example: acting/ not acting (meaning e.g. ACC controlling car speed or not)
System/ function characteristics	Depending on the hypotheses an analysis of systemor driver performance with respect to special system/function characteristics might be conducted, e.g. examining differences in system performance between Nomadic Devices (phone, Smartphone,PND ...) or effects that depend on vehicle type. Example: passenger vehicle/ truck/ bus
Interaction between systems	System and especially driver behaviour might change depending on whether the system under evaluation is the only active support system or whether interactions between two or more systems are foreseen. Example: interaction between Blind Spot Warning and Lane Departure Warning.
ENVIRONMENTAL CONDITIONS
Traffic conditions	Performance of some systems might differ depending on traffic density. Others might only be reasonable with a minimal traffic density. Example: Level of Service A and B
Environmental situation	system performance differs depending on lighting and weather conditions like rain/ snowfall/ icy roads, etc. Example: normal/ adverse weather conditions
Road characteristics	E.g. type of road gradient, super elevation, curvature, curviness, since some systems are dedicated to improve driving performance in curves etc. Example: urban roads/ rural roads/ highways
Geographical characteristics	Information about geographical characteristics relevant for testing the systems. Example: mountainous/ flat areas, metropolises with high street canyons.
DRIVER CHARACTERISTICS AND STATUS
Driver specification	Characteristics of the users have an impact on the driving performance. Even if no specific impacts are expected of certain characteristics, some outcomes may be explained better with more knowledge about the participants. A minimum set of data such as age, gender, income group and educational level is easy to gather from participants. Information about driving experience is also important. In section 5.5.6 more details are reported about driver characteristics and about usage of questionnaires for better understanding of driver behaviour and acceptance evaluation.
Driver status	Mindset of the driver Example: attentive/ distracted/ impaired
Purpose, distance, duration	Describes the different attributes of a trip (time between ignition on and ignition off). All three aspects have an impact on driver behaviour and hence on patterns in the data.

A set of basic rules has been set for the design of the situations for an FOT :

1 Complementary: situations are not allowed to overlap.

2 Entirety: the sum of all situations should describe the complete use case.

3 Baseline: The same situation without the use of the systems (system off or non-present) is defined as the baseline. The baseline is the basis for the benefit assessment of the system and the comparison between systems. Therefore, for the same use case, there can be many baselines depending on the number of situations.

4 Comparability: functions compared in an FOT need to have the same use cases and therefore same baseline and situations.

5 Variability of situation parameters: depending on the point of view (user, trip, vehicle, single FOT , multiple FOTs, etc…), attributes describing a situation can vary considerably or not.

This list is non-exhaustive and might be extended if necessary.

Finally, out of all the possible situations, one will need to select the relevant ones for scenarios of interest in an FOT . The scenarios are defined as a use cases in a specific situation and therefore one or more scenarios should be considered from each use case. All other situations should be considered out of the scope of the FOT study. However, if possible data should still be collected in all situations in case an alternative study would like to reuse the same data.

The process of defining the use cases will help the FOT for the next steps: the definition of the research questions and hypotheses and finally the identification of the needed indicators. The scenarios as they are defined at this stage of the FOT are not detailed enough for data analysis purposes. For this reason, after the definition of the indicators, the scenarios (and their situations) will need to be further described in terms of events for data analysis purposes. Only then, the scenarios can be classified with a quantitative measurement tools in function of the defined indicators.

4.2.3. Step 3: Identification of the research questions

The research questions specific to an FOT can only be identified once the overall goal of an FOT has been established.

In general terms the goal of any FOT is to investigate the impacts of mature ICT technologies in real use. The core research questions should therefore focus on impacts but there are other questions that ‘surround’ this core. The range of possible questions is listed below. This list below should be considered a first step in any FOT and not a comprehensive set of questions.

LEVEL OF SYSTEM USAGE

Which factors affect usage of the functions? Examples are

• Portion of journeys where system is used

• Types of road on which system is used

• Traffic density

• Headway

• Weather condition

• Ambient lighting

How do driver characteristics affect usage of the functions? Examples are

• Personal characteristics ( e. g. age, vision)

• Socio-economic characteristics ( e. g. family, friends, employment status)

• Journey-related characteristics ( e.g. other car occupants, shared driving)

IMPACTS OF SYSTEM USAGE

What are the impacts on safety?

• Exposure

• Risk of accident or injury

• Accidents

What are the impacts on personal mobility?

• Individual driving behaviour

• Travel behaviour

• Comfort

What are the impacts on traffic efficiency?

• Traffic flow (speed, travel time, punctuality)

• Traffic volume

• Accessibility

What are the impacts on the environment?

• CO2 emissions

• Pollutions

• Noise

IMPLICATIONS OF MEASURED IMPACTS

What are the implications for policy?

• Policy decisions

• Laws, directives and enforcement

• Future funding

• Public authority implications

• Emergency service implications

What are the implications for business models?

• Predictions for system uptake

• User expectations

• Pricing models

What are the implications for system design and development?

• HMI design and usability

• Perceived value of service

• Device design

• Communications networks

• Interoperability issues

What are the implications for the public?

• Public information/education

• Changes in legislation

• Inclusive access to systems

• Data protection

When defining the research questions the trade-off in terms of process and consequences between an in-depth analysis of few research questions vs. general analysis of many http://wiki.fot-net.eu/index.php?title=Research_question research questions] should be considered (especially for uncontrolled NDS data), bearing in mind that focus on few research questions often produces scientifically sound results.

4.2.4 Step 4: Creation of hypotheses

Once the key research questions for the FOT have been identified, hypotheses can be derived. The process of formulating hypotheses translates the general research questions into more specific and statistically testable hypotheses.

FESTA distinguishes between more general and open research questions and more specific hypotheses. The definition of a research questions in the case of an FOT is "a general question to be answered by compiling and testing related specific hypotheses". An example would be: "Does having a Forward Collision Warning system improve safety in driving?" A hypotheses is here defined as: "a specific statement linking a cause to an effect and based on a mechanism linking the two. It is applied to one or more functions and can be tested with statistical means by analysing specific performance indicators in specific scenarios. A hypotheses is expected to predict the direction of the expected change." The term "function" is used because a particular system may have a number of distinct functions - for example, one system could provide both Adaptive Cruise Control and a Forward Collision Warning. It is also the case that one function can be provided by different systems. An example of a hypotheses might be: "Forward Collision Warning will have the direct effect of an increase in minimum Time to Collision TTC".

There is no process that can assure that all the “correct” hypotheses are formulated. To a large extent, creating hypotheses is an intuitive process, in which a combination of knowledge and judgement is applied. Nevertheless, a number of recommendations can be made about how this process should be conducted. These recommendations have been tested in FESTA FOT - Net workshop and modified based on the experience of and feedback from FOTs.

Two complementary ways to develop hypotheses have been used. Both ways need to be followed, while it is not of importance which step is taken first. One of the steps follows the sequential check of specific areas in which functions can have an impact; the other step is fully based on the description of specific scenarios. While the one step results mainly in general hypotheses, the other step triggers the development of very specific hypotheses in specific driving situations or scenarios.

4.2.4.1 Top-down approaches

The six areas approach

The six areas of impact defined by FESTA are based on ( Draskóczy et al., 1998). Although this approach was originally was designed for formulating hypotheses on traffic safety impacts, it is in fact equally applicable for efficiency and environmental impacts. The six areas are:

• Direct effects of a system on the user and driving.
• Indirect (behavioural adaptation) effects of the system on the user.
• Indirect (behavioural adaptation) effects of the system on the non-user (imitating effect).
• Modification of interaction between users and non-users (including vulnerable road users).
• Modifying accident consequences (e.g. by improving rescue, etc. — note that this can affect efficiency and environment as well as safety).
• Effects of combination with other systems.

It is not of particular importance to which of these areas a particular hypotheses is allocated. The six areas are instead to be used as a checklist to ensure consideration of multiple aspects of system impact.

In applying this procedure, it should be noted that:

• Area 1 includes the human-machine interaction aspects of system use.
• The driving task (see Table 4.1) can be defined, following Michon et al. (1985) into the three levels of strategic, tactical (manoeuvring) and control (operational) aspects. All three levels need to be considered. All are affected by the input from external conditions, which because they are external to the driver can include the vehicle and devices within the vehicle.

The Strategic Level covers longer-term planning and responses to traffic conditions includes potential modifications to:

• Mode choice
• Route choice
• Exposure (frequency and/or length of travel)

The Tactical (manoeuvring) Level includes potential modifications to speed choice and the effects of such modification on manoeuvring and interaction with other road users. These typically happens in time span of seconds.

• Consideration should be given to such mediating factors as user/driver state, experience, journey purpose, etc.

Table 4-1: The three-level model of the driving task, based on Michon (1985)

It should also be noted that the effects of system use may be:

• Short-term or long-term in terms of duration and
• Intended or unintended in terms of system design.

Impact area approach

Another useful top-down approach starts from the most relevant impacts which are: Efficiency, Environment, Safety, Mobility and User Uptake. The basic principle for generating hypotheses using this top-down approach lies in a theoretical understanding of the factors that influence the different impact areas. It should be noted that there is likely to be overlaps of these factorsamong the impact areas under consideration and hence the same research questions and resulting hypotheses will be applicable across more than one impact area. The approach will result in generic research questions that are independent of the any system functionality. The procedure for generating hypotheses in this top-down approach is as follows:

• The impact area should be considered in its entire context and primary measures affecting that area identified.
• Secondary factors of these measures are then identified that can be used to explain the variations in the primary measures.
• Finally the variables affecting the secondary measures are identified.
• The variables identified from the basis of the generic research questions "Is there a change in the variable?" and the hypotheses based upon an anticipated effect of the variable "The variable will increase/decrease."

This procedure should be undertaken for each of the impact areas. Using the Safety Impact as an example:

• The primary measures affecting safety would be "Number of events (accident, near misses) that occur" and the "Severity of the event.
• Secondary factors affecting the first of these measures would, for example be

'Exposure of the vehicle on the road', 'The driving style of the driver', 'The distraction of the driver from the driving task' and 'Any interaction with the fitted device'.

Considering the factor 'Exposure', this can be measured with the following variables:

'Length of journey', 'Number of trips undertaken' and 'Road type used'.

These variable lead to the following research questions:

• Does the system affect the length (miles) of journeys?
• Does the system affect the duration (hours) of journeys?
• Does the system affect the number of journeys undertaken?
• Does the system affect the road type used?

This leads to the generic hypotheses that can be tested in a statistical manner. The direction each hypotheses should take (e.g. increase or decrease) is based upon the anticipated effect once the top-down approach is integrated with the bottom-up (system defined) approach.

• Journey lengths will increase/decrease when the system is used compared to when it is not used.
• Journey duration will increase/decrease when the system is used compared to when it is not used.
• The number of journey will increase/decrease when the system is used compared to when it is not used.
• The use of rural roads/motorways/major roads will increase/decrease when the system is used compared to when it is not used.

4.2.4.2 Bottom-up: the use case approach

This process leads to the development of hypotheses concerning specific scenarios. These scenarios are derived from the combination of Use Cases and situations (see Table 4-1). Scenarios should be covered systematically. It is recommended that a structured approach be used in scenario development and that an Excel spreadsheet is used as a record.

4.2.4.3 Prioritising the hypothesis

A complete list of the hypotheses that have been developed should be recorded. If it is considered that some are too trivial or too expensive to address in the subsequent study design and data collection, the reasons for not covering them should be recorded. It should also be noted that there are standardised techniques for observing driving behaviour with manual observers which may be less resource intensive than using dedicated data recording. Observations using such techniques can be carried out at various times during the study, preferably along a fixed route. A huge number of research questions and associated hypotheses from the top-down and the bottom-up approaches will be developed. A key task is to integrate both sets of hypotheses in the context of each FOT. It is envisaged that the bottom-up approach will form the basis of the hypotheses list for an FOT and that the top-down approach will be used in order to check that nothing significant for a particular impact area has been omitted. After the integration has taken place, the list of hypotheses is still likely to be large. In order to derive a final, manageable set of research questions and hypotheses that can be applied through the various test sites, a cost-benefit approach is proposed. Using this approach, an assessment is made regarding the likely "costs" of collecting the data. Costs can be represented in terms of effort required to derive a performance indicator expressed predominantly in terms of resources. This should be offset against the likely "benefit" that proving/disproving the hypotheses will have. This is measured by way of the likely contribution towards providing a significant answer the research question and thus the level of contribution to the impact assessment. To some degree, this will depend upon the stakeholder needs and requirements, and therefore a prioritization of their needs should be considered.

4.2.4.4 Summary

The basic set of recommendations is:

1. A structured approach should be applied linking a top-down approach at the global system level with a bottom-up approach which looks more at the system states and what can arise from them. FESTA considers it mandatory to combine the two approaches.
2. A multidisciplinary team should jointly develop the hypotheses. A workshop at which participants can brainstorm and debate is recommended to achieve this. Participants in the process should include design engineers, traffic engineers and behavioural scientists, ideally including both behavioural psychologists and human factors experts.
3. The process should iterate between the top-down and bottom-up approaches. It is not particularly important which is performed first, but it is important to cross-check one approach by using the other.
4. An important output of the process is the initial selection of the performance indicatorsto be used in testing the hypotheses.

4.2.5 Step 5: Link hypotheses with indicators for quantitative analyses

Some of the hypotheses will already incorporate an indicator which needs to be measured, e.g. a very concrete like “The function will increase time-to-collision (TTC)”. In this case it is obvious which indicator to choose, while the method to measure TTC might include complicated procedures and/ or costly measurement equipment. The Chapter on Performance Indicators gives an overview about many reasonable indicators. One should consider these indicators when planning the experimental design, since a detailed description how to calculate the indicators from measurements is also provided.

Other hypotheses might be rather unspecific, but still reasonable after rephrasing into testable ones. This rephrasing goes hand in hand with the identification of related reasonable indicators. For example, a hypothesis like “The function will increase lane changing performance” is not directly testable, since “lane change performance” is not an indicator itself. Hence, surrogate measures must be identified to evaluate lane change performance. These surrogate measures or indicators can e.g. be found in publications of corresponding research projects. If appropriate information cannot be found or is not accessible, new performance indicators need to be developed. Those indicators and the measurement methodology must be valid, reliable and sensitive, that is, the measurement must actually measure what it is supposed to measure, they must be reproducible and the measures must be sensitive to changes of the variable. A sensitivity analysis should be performed beforehand during a pilot study to make sure that the new performance indicator is suitable. When one or more surrogate measures have been identified, the initial hypothesis can be reformulated into one or more testable hypotheses. In the above mentioned example, reasonable indicators associated to “lane change performance” might be: use of turning indicator or the number of lane change warnings. The initial hypothesis will then be reformulated into: “The system will increase the use of the turning indicator.” and “During the system use, the number of lane departure warnings will decrease”. The next step is then to evaluate how the indicators “use of turning indicator” and “lane departure warnings” can be measured. In this context, the chapter on Performance Indicators provides useful information.

4.2.6 Iteration

Iteration is especially important when defining research questions and hypotheses, because usually a selection has to be made from the large amount of possible hypotheses, both based on their relation with the main impact areas and research questions and on practical issues. Another important iteration point is the impact areas. The final question on the impact assessment may drive the design of the FOT in all its aspects. When practical issues, such as for example which data-loggers to use, make certain choices hard to realise, iteration to earlier stages is necessary. Cost-benefits analyses and feasibility assessment of different options for the FOT may also drive the design. It is important that there is a good communication between the project members who are in charge of defining research questions and hypotheses and the ones who will be analysing the data in order to assure that the questions can indeed be answered.