FESTA handbook From Functions to Hypotheses

From Functions to Hypotheses

The final objective of an FOT is to evaluate in-vehicle functions based on Information Communication Technology ([ ICT]) in order to address specific research questions. These research questions can be related to safety, environment, mobility, traffic efficiency, usage, and acceptance. By addressing the research questions, FOTs promise to furnish the major stakeholders (customers, public authorities, [ OEM]s, suppliers, and the scientific community) with valuable information able to improve their policy-making and market strategies. Individuating the most relevant functions and connected hypotheses to successfully address the above-mentioned research questions is one of the major challenges in an FOT . In this Chapter, the process of individuating the functions to be tested in an FOT and the relevant connected hypotheses will be elucidated. Specifically, the reader will be guided in the process of

1) selecting the functions to be tested,

2) defining the connected use cases to test these functions,

3) identifying the research questions related to these use cases,

4) formulating the hypotheses associated to these research questions , and

5) linking these hypotheses to the correspondent performance indicators.

The FOT chain shows specifically the steps reported above.

The steps may be influenced by other elements of the [ FOT] chain. The selection of functions may be driven by the socio-economic impact that is expected, or by the research questions. When details are filled in later on in the process, it may be necessary to re-visit earlier steps, for example limits in resources or technical capabilities may lead to a decision to limit the amount of functions, use cases and hypotheses.

Systems and functions

In the last few years, the number of [ ICT] functions available for use on standard vehicles and more generally while travelling has been rapidly increasing. [ ICT] functions are intrinsically designed to provide the driver or traveller with new, additional information. However, the extent to which this increased amount of information from these [ ICT] functions results in clear and positive effects on safety, environment, mobility, usage, and acceptance in real traffic situation is unknown. FOTs warrant to evaluate, for the first time, these [ ICT] functions in a real traffic situation during naturalistic driving. In this handbook we refer to 1) in-vehicle, 2) cooperative, and 3) nomadic systems intended as a combination of hardware and software enabling one or more [ ICT] functions. Depending on the different systems implementing a specific function, different challenges may have to be faced during the FOT design.

It is important to note that [ NDS] in the future will increasingly include new systems as they increase in market penetration. For example, future [ NDS] will be able to be used for with-and-without analyses of systems that are new today, but commonplace tomorrow (e.g. Forward collision warning may be as commonplace as ABS tomorrow).

Vehicle systems

Vehicle Systems are a combination of hardware and software enabling one or more functions aimed at increasing driver’s safety and comfort. Vehicle Systems promise

1) to increase driver safety by increasing driver’s attention in potentially hazardous scenarios (such as the Forward Collision Warning),

2) to improve the driver’s comfort by automating some of the operational driving tasks (such as the Adaptive Cruise Control function),

3) to increase driver mobility by furnishing timely traffic information (such as the Dynamic Navigation function), and

4) to increase safety in critical situation by automating the vehicle response (such as the Collision Mitigation function).

Vehicle systems are becoming more and more standard equipment, even in middle class vehicles and commercial vehicles. However, their impact on the driver, the traffic system, the society, and the environment in the short-, but especially in the long-term is not fully understood. FOTs can help quantifying the impact of vehicle systems on driver’s workload and to understand how different functions interact with each other in a real complex traffic situation. Further, FOTs will expose these functions to many improbable scenarios which are not possible to be tested during the functions evaluation phase.

Cooperative systems

[ Cooperative Systems] are vehicle systems based on vehicle-to-vehicle ([ V2V]), vehicle-to-infrastructure ([ V2I]) and infrastructure-to-vehicle ([ I2V]) communication technology. Communication technology has enabled a new class of in-vehicle information and warnings which are more precise in terms of time and location. It is foreseen, that the integration of [ cooperative systems] with autonomous functions will provide a new level of [ ADAS]. Infrastructure based information tells the driver for example what is the appropriate speed to keep on a specific part of road or warns the driver in case of ice on the road or fog. At European level, cooperative functions and systems have been developed in several projects under the 6th and 7th Framework Programme, namely: CVIS, PRE-DRIVE C2X, PReVENT, SAFESPOT, and WATCHOVER.

One of the initial objectives of [ V2V], [ V2I], and [ I2V] is to increase road safety. The development of safety-critical [ V2V] systems in Europe has been mainly promoted by the Car-to-Car Communication Consortium ([ C2C-CC]). More recently, the EC has placed emphasis on the application of [ cooperative systems] to achieve environmental and efficiency impacts.

The lower level communication protocols are based on [ IEEE] 802.11p working in the 5.9 GHz. frequency range.

A key point in [ cooperative systems] is standardisation. In October 2007 Technical Committee for Intelligent Transport Systems ([ ITS]) was launched by [ ETSI] with the target to turn research results into standards and in October 2009 European Commission published a Mandate (N. 453) which is setting deadlines for the standardisation activities. The mandate has been accepted by [ ETSI] and [ CEN] and the activity is currently (2011) ongoing. One of the objectives of the mandate is harmonisation with U.S. and Japanese standards as well as taking into account the ISO activity which includes the [ CALM] (Continuous Air interface for Long and Medium range communication) architecture.

[ Cooperative systems] differ in several areas to conventional areas, which directly influences the FOT planning and operation.

Firstly, there are currently no [ cooperative systems] available in the vehicle market – this implies that all participating vehicles in the FOT have to be equipped manually with prototype systems. This is costly and makes it difficult to find suitable test vehicle fleets. Also, drivers are not aware of such and have no knowledge of their functionality. Before starting the FOT , drivers have to be educated on the usage of [ cooperative systems].

Another difficult issue lies in the essence of cooperation: all [ V2V] functions rely on more than one vehicle being in communication range, and [ V2I] relies on roadside-stations. Before starting the FOT , it is hard to estimate the number of vehicles in one area and how often vehicles will pass a Road-Side Unit ([ RSU]). This penetration rate is a crucial factor in designing and evaluating an FOT .

Current estimates from recent [ V2X] FOTs predict a minimal penetration rate of 10% for [ V2X] functions to show a noticeable effect on traffic safety and efficiency. Given the size of the FOT area and the average distribution of vehicles, the number of participating vehicles should usually be considerably higher.

The penetration rate also influences the Frequency of events (FoE). In [ cooperative systems] FOTs, the FoE is roughly related to the number of necessary vehicles for a function. While some functions work with only two vehicles (e.g. a slow vehicle warning), other functions require several more (e.g. traffic-jam ahead warning). For certain functions the combined frequency of events might make a naturalistic FOT unfeasible.

A proper assessment of the penetration rate impact can be derived from dedicated simulations. A simulation environment should consider traffic effects, communication models, applications and their respective influence on each other. Such dedicated simulation environments are able to predict the frequency of events for the developed functions and thus support the FOT design and setup process greatly in cooperative systems.

There are several implications derived from this effect to the test setup and execution of [ cooperative systems] FOTs:

• The number of equipped vehicles is considerably higher or the FOT needs to run considerably longer to collect enough data to be representative.
• It might be very difficult or even not possible to run the FOT uncontrolled for some functions(e.g. Emergency electronic brake light,..)[ FOT]. Controlled testing implies the usage of dedicated tools to specify the test cases, the test scenarios and to run the test. They also need to be linked to the vehicles and drivers to control and monitor the running test. (See 6.5 for more details)

It is most likely that the initial market drive will be for [ V2I]-based applications until there is a critical mass of equipped cars on the road. FOTs can assess the technological and business feasibility of [ cooperative systems] and may be necessary to complete the validation of such systems. In fact testing to validate [ cooperative systems] may require very complicated set-up which may not be possible unless in a real-traffic set-up.

One important issue is the installation of infrastructure devices that could be needed. Permission for the testing and team involvement should be planned in advance considering the impact on normal traffic during the installation phase. In the testing phase proper logistic support is needed in order to allow technical people to operate with debugging equipment in order to tune the systems. Assessment of [ RSU]s and antennas positioning could be rather time-consuming. Protection and maintenance of road-side equipment should be ensured.

In comparison to [ ADAS] functions or [ nomadic devices], [ cooperative systems] are more complex to analyze due to the distributed nature of involved components. In former cases it can be sufficient to log the vehicle state and the functionstate. In contrast, to completely evaluate the functionality of a [ cooperative system] it is also necessary to handle all incoming and outgoing [ V2X] traffic and the Local Dynamic Map state on the vehicle side. As other vehicles and Roadside stations directly influence the system functionality the scope of logging has to be adapted as well. While a focus on single vehicles can be sufficient in other FOTs, it has to be widened to complete [ ITS] areas in [ cooperative systems]. The logging system has to be able to cope with the broader scale both in [ ITS] station and [ ITS] area scope.

Each [ ITS] Vehicle Station needs to be equipped with a logging device. This device or software component has to be able to write a common log destination from all important sources on the Application Unit and Communication Unit:

• Vehicles Data ([ CAN] Bus)
• Position and Time (if not present on [ CAN])
• Networking: Incoming and Outgoing Messages (CAM, DENM)
• Local Dynamic Map Status (List of all neighbours, List of all events)
• function Status
• Other Facilities (connection to backend-services, etc.)

It is obvious, that all these various sources provide a huge amount of data. This is a major consideration for the logging system on each station and should influence it in two ways:

• Qualitative: Each log entry should be reduced to the minimum (entry type ID, timestamp, value). It is possible to further reduce the required space, e.g. by only writing the time and not the date into each entry. It can also be useful to compress the written files. It is strongly advised to use a “strict” logging scheme, which only allows pre-defined log entry types, as this simplifies post-processing and also allows to save disk space.
• Quantitative: Logging does not have to record all values all the time. Although data storage cost is lowering more and more, if necessary, it is possible to reduce the used space drastically by using pre-defined logging profiles. These are generated with a mapping from those performance indicators needed for evaluation to the available measurements. For each test a logging profile can be pre-selected, which contains only those measurements needed.

In more complex settings the logging system can also automatically switch profiles depending on the measures values (a critical situation can be detected from measurements and a wider logging profile can be selected for a given time). It is also possible to select the log profile directly in controlled testing to guarantee that all necessary measures are logged. The quantity of data to be collected is discussed in section 1.2 of the Chapter on Data Analysis and Modelling where two approaches (“space mission” and “minimum data set collection”) and related issues, are presented.

The [ ITS] Roadside Station provides similar measures (except vehicle data) and it is advised to use a similar architecture for the logging system. In contrast, the measures available to the [ ITS] Central Station can differ widely and are specific to each FOT. Roadside sensors may be needed to provide information related to interaction with non-connected vehicles and other situational variables.

For controlled testing scenarios it is advisable to include monitoring features to the logging component. Monitoring is a real-time transfer of selected information available to the logger. In a central counterpart to monitoring this information is displayed to visualize the current test progress. Typically the monitoring value set is a subset of logging and should at least include vehicle ID, position, heading and speed. In this minimal set the test operator can follow vehicle progress through the test. More advisable monitoring sets include direct information from developed functions to monitor in real-time, if the test intent (to trigger the developed functions in a specified way) has been met. Test operators can thus determine if a test was successful (and the detailed logs are valuable for evaluation) or if tests have to be repeated right away. Sophisticated test control and monitoring tools support operators in this task.

Log files in [ cooperative systems] FOTs can quickly take up gigabytes of data. A rough estimate for a typical [ cooperative system] is 500Mb per day per vehicle. It is important to design the test and logging system to cope with this data amount. Log files have to be saved locally on vehicle and roadside station, have to be transferred to the storage and have to be integrated into a central database. Throughout this process it has to be assured, that no data can be lost due to a single point of failure.

The transfer of data from vehicles to the central storage might exceed possible bandwidth from 3G connections. Other possible ways include transfer over WiFi and manual transfer by [ USB] Stick or Drive. Log data should be compressed to speed up transfer times. Depending on the vehicle availability (average run-time over day, time in between access to vehicles) one solution has to be chosen and the system has to be specified in a way, that the in-vehicle storage can fit enough data to hold all log-files until extraction. When large international FOT are planned, [ cooperative systems] may require interoperability tests. The costs, the organizational and legal aspects (prototypes in foreign countries) should not be underestimated when planning these testing sessions.

Nomadic devices

The use of so-called [ Nomadic Devices] for transport and traffic related applications has become increasingly commonplace in the last few years. The first wave of such devices were dedicated Satnavs, also known as Personal Navigation Devices ([ PND]s). The second wave of functions have been a large variety of “apps” for Smartphones.  In addition some [ PND] suppliers have formed alliances with vehicle manufacturers so that their products may be fitted as original equipment. Whether fitted as original equipment or functioning on Smartphones, the hardware platforms tend to operate autonomously of the vehicle. Indeed many of these devices can provide traveller functionality — for pedestrian route-finding, for public transport information, to locate points of interest — outside the vehicle. 

The functionality of [ PND]s and Smartphones has been evolving over time. Initially, the functionality of [ PND]s was focussed on guiding the driver from geographic point A to a point B, using buttons and the screen to set the desired destination. That functionality has now evolved so that the devices may offer a media player, media viewer, mobile phones integration, and may be voice activated. They can also provide routing and navigation support outside the vehicle. Smartphones started from an emphasis on telephony, supported by additional functions such as an address book and texting. Additional functions(photography, navigation, radio, media centre, etc.) were then continuously added. Now that [ GPS] is generally provided, they are navigation-capable, so that navigation apps are commonplace. On the other side, [ PND]s are broadening in capability and now tend to include a mobile phone SIM card. Thus there is a large overlap in capability and functionality.

This provides a capability to download map updates but also gives cooperative-system-like functionality in the form of live traffic updates and other environmental information. The same mobile phone link can be used by service providers to acquire real-time traffic conditions. Smartphones, with their built-in networking capability, can offer the same functionality.

As well as links to mobile networks, [ nomadic devices] are now available with Bluetooth, WiFi networking, assisted [ GPS], speech recognition and high-end operating systems with the capability to provide dynamic geo-referenced applications to assist the driver and traveller. Research on [ nomadic device] integration in the vehicle environment has been provided through projects like AIDE, GST, and CVIS. Current FOT projects, such as TeleFOT or EuroFOT, are assessing usability, acceptance and impacts.

Specifically tailored aftermarket devices, sometimes offered by the same manufacturers as personal [ PND]s, are now targeted at the fleet market for management of logistics and other fleet functions. Some of these devices provide links to [ CAN] data. An increasing focus of such systems is fuel economy and feedback to drivers on their efficiency in driving. Another sector for aftermarket devices is the pay-as-you-drive insurance market.

[ Nomadic devices] need to be evaluated from the perspectives of user behaviour and acceptance, safety (particularly in regard to [ HMI] issues), travel and traffic impacts and environmental implications. It also needs to be recognised that such devices can have broad mobility implications, both in terms of the strategic level of driving (route choice) and in terms of trip generation and mode choice. Any evaluation of usage needs to consider the potential for both in-vehicle and out-of-vehicle usage of these devices.

One of the critical characteristics of [ PND]s and Smartphones is how far the device is integrated within the vehicle. Many devices use specific mounting kits for in-vehicle installation (connection to power supply, [ GPS] and in some cases the vehicle’s audio system). Typically [ PND]s are mounted with a suction cup directly to the windscreen, while cradles for smartphones may also attach via a suction cup. As a result they may impede the driver’s field of view increasing the risk of accidents. There is also often a problem with small screen size and inadequate audio volume. Nevertheless, the popularity of these devices has not been affected. To increase usability and reduce negative side effects, an Automotive Bluetooth profile has been developed so that such devices can use the vehicle’s built-in [ HMI]. This can offer improvements both in user input to the device (stalks, buttons, speed recognition, link to hands-free mobile phone, etc.) and in device output (using integrated audio, and screen).

The general ease of use of a device will have a major influence on acceptance and willingness to pay. Here ease of use refers not just to the usability while driving but to the user experience in all aspects of usage — pre-trip, in-trip and post-trip. Post-trip functionality is very relevant to usage in the fleet market and to support for and feedback on eco-driving.

Due to the above-mentioned fast innovation cycle, FOTs studying [ nomadic devices] may require state of the art planning in order to keep up with the introduction of new features and functions. They will also need to consider the surrounding infrastructure since (rather like many [ cooperative systems]) many functions rely on information and support from the outside. Weather forecasts, traffic information, updates on road conditions, dynamic speed limit information and speed advice are all dependent on service providers.

Combinations of functions

There are many FOTs which investigate the impacts of a combination of functions sometimes because systems and functions come in a bundle. One such common bundle is the combination of Adaptive Cruise Control and Forward Collision Warning. Both functions make use of the same sensors, and indeed second-generation [ ACC]s generally implement a warning function to indicate to the driver when the deceleration demanded of the [ ACC] in order to prevent a collision with the preceding vehicle is beyond the function’s designed capability. In other cases, an FOT may be investigating a function that resides on a platform which offers many other functions. This is almost invariably the case when studying functions that reside on [ nomadic device] such as a smartphone or SatNav. In some cases a project will create a new function or “app” and provide the function to users on a standard consumer [ nomadic device]. It is not practical or reasonable to demand of users that they do not use the full functionality of the device and attempts to disable features may well annoy participants.

Therefore in planning the evaluation, it is important to consider how functions may interact with each other and how those interactions might affect user behaviour. This needs to be done at the stage of an FOT when research questions and hypotheses are initially formulated. What needs to be considered is:

1. Can the effects of the various functions be disentangled? Note that it may not be possible or feasible to do so, particularly if the functions are closely coupled together.
2. Does the experimental design need to be modified to enable both the single effects of each functionto be investigated as well as the effect of the functions in combination?

Some systems are now so integrated that it is no longer possible to disentangle them completely. An [ ACC] that does not incorporate [ FCW] is no longer on offer. An [ ACC] + [ FCW] system can be driven with the [ ACC] off but cannot be driven in [ ACC] mode with [ FCW] off. Indeed disabling the [ FCW] functionality within an [ ACC] could be considered unethical and may be impossible for practical reasons.

Where bundles cannot be disentangled, it is only possible to investigate and report on the impacts of the combined systems. But when, for example, there are separate apps on a [ nomadic device], then consideration needs to be given to how those apps might interact. The investigation can be carried out in a naturalistic manner — i.e. the participants are free to choose when and when not to use the applications and functions that are not the immediate focus of the FOT. Alternatively, it can be carried out in a more experimental manner by means of instructions to participants, systematically enabling and disabling the functions, or carrying out controlled drives with the functionality of the system(s) carefully manipulated.

In formulating hypotheses, it is useful to think both of the singular effects of a system or function and of the synergistic effects that one function may have on another. Thus the recommended procedure is to start with the individual functions and then to proceed to combinatory effects. The process should therefore be to:

1. Begin separately with the individual functions, generating a list of research questions and hypotheses
2. Examine commonalities and conflicts between the systems and functions and derive hypotheses from those commonalities and conflicts:
   1. Can they generate simultaneous messages and/or warnings?
   2. Do they have a common interface and can they both be activated simultaneously?
   3. Are the same performance indicators relevant to each system and/or function?
   4. Are there common factors influencing usage?
3. Distinguish between hypothesis additive effects when the two systems interact with each other and multiplicative effects when the presence of a second system will alter the effects of the first. Additive (or subtractive) effects mean that the size of the effects will change. Multiplicative effects mean that the relationship is different, i.e. that there is an interaction in statistical terms. Consider situations in which such multiplicative effects might be important.

The application of this procedure should produce a comprehensive set of hypotheses on how the functions might interact and should affect the subsequent experimental design. As in all such work on the preparation of research questions and hypotheses, the reasonableness of pursuing every possible combination in a structured experimental design needs to be considered. Any additional function can impose a huge cost in terms of the increase in the number of possible combinations. It can be argued that there is a rationale also for multiple baselines, i.e. with all functions off, and with one function at a time off.

Complex experimental designs have large practical costs associated with them and the benefits of such designs need to be carefully considered. The costs can be in the form of the number of different baselines that may be required and of the time needed for data collection on each combination. A full experimental design in which ordering effects are considered may well be totally impossible for practical reasons. This can all lead to excessive time required for FOT execution. But another side effect can be the sheer difficulty of getting the participants to comply with all the different conditions of the experimental design. These are arguments for using a more naturalistic approach in which the participants are free to use whatever combination of systems and functions they choose. Of course system and function state will need to be recorded. This naturalistic approach has some disadvantages:

• Not all combinations may occur and not all participants may experience each combination
• It may be hard to take care of seasonal effects
• There may be insufficient sample sizes in some conditions so that experimental power is inadequate

However, the naturalistic approach also has advantages in that:

• Participant compliance will generally be assured
• The frequency with which the various combinations are used can provide useful information for the scaling-up process

Alternatives which offer greater efficiency than a full experimental design can be proposed. Laboratory experiments on driving simulators can be adopted to examine synergistic effects and a priori analysis can be applied at an early stage in an FOT to identify combinations that are of particular interest and which should therefore be the focus of attention. This can lead to a satisfactory but incomplete experimental design.

1. General methodology

The main advantage of an FOT is that it has the potential to give insight in system performance in naturalistic driving situations, as free as possible from any artefact resulting from noticeable measurement equipment or observers in the car. Therefore the first step when planning an FOT is to identify systems and functions where considerable knowledge about their impacts and effects in realistic (driving) situations is of major interest, but is still lacking (see Section 1.2.1). Another domain for FOTs is the area of systems and functions which need a certain penetration rate to work at all, like especially [ cooperative system].

After the identification of the functions and system, which should be tested in an FOT . the goal is to define statistically testable hypotheses and find measurable indicators to test the hypotheses. To reach this goal, several steps need to be taken, starting from a description of the functions down to an adequate level of detail (see Section 1.2.1). This means that the main aspects of the functions, its intended benefits and the intrinsic limitations have to be described to fully understand objectives and limitations and to derive reasonable use cases.

Secondly, these use cases need to be defined (see Section 1.2.2). Use cases are a means to describe the boundary conditions under which a function is intended to be analysed. A general starting point is given by the functional specifications from the function description part. But it might also be of interest how a function performs when certain preconditions are not met and to identify unintended and unforeseen effects.

Starting from the use case definitions specific research questions need to be identified (see Section 1.2.3). [ Research questions] are general question to be answered by compiling and testing related specific hypotheses. While research questions are phrased as real questions ending with a question mark, hypotheses are statements which can either be true or false. This will be tested by statistical means (see Chapter Data Analysis and Modelling). One might already have a very clear idea from the beginning which hypotheses are to be tested in a very specific situation during the FOT . However, this very focused view might result in an extreme limited experimental design, where important unintended effects will not be considered. The process to define hypotheses developed in FESTA aims to prevent these potential issues (see Section 1.2.4).

Finally, hypotheses can only be tested by means of reasonable indicators (see Section 1.2.5).

These steps are shown as parts of the complete FOT and are elaborated further in the following sections. The major steps from research questions and hypotheses to performance indicators are also summarised in Annex C . FESTA deliverables - D3 Vehicle systems - Final.pdf D3, - D4 Cooperative Systems - Final.pdf D4 and D5 final.pdf D5 provide additional detail on the application of the FESTA methodology to identify functions and systems and to develop hypotheses for the experimental design. All steps, from the description of the systems and functions, the development of use cases and scenarios, as well as the research questions and hypotheses and the proposal of related performance indicators have been accomplished.

In the FESTA methodology, function description is a starting point, however, this may pose some problems. Not all FOTs are testing one pre-defined function, sometimes a set of functions or systems are to be evaluated. Or in [ naturalistic driving studies], the focus of the study may not be on functions but on driving behaviour in general. Stakeholders may have different ideas about the functions they want to test, and function descriptions are not always clear.

One issue is whether a function is generic or manufacturer-specific. In other words, how far should the particular manufacturer’s specification and functionality be considered. The specifics of the function can clearly have an impact on acceptance and behaviour. An example here might be an [ LDW] that gives auditory and visual warnings and one that provides haptic feedback through the steering wheel. It is possible that one design will turn out be more effective than another.

When it is possible to start with a clear function description, this will allow a detailed planning of the data collection plans, of the experimental design and hence of the costs. But it is of course still necessary to further specify research questions, hypotheses and performance indicators. In the case functions are not well-defined or a decision has not made before the start of the project which functions to investigate, it may be more advisable to start with defining the research questions and select functions that seem most useful in answering these questions.

The issue of how many functions can be investigated in an FOT is a matter of the resources to be deployed and of whether the impacts of each function is to be investigated separately or alternatively whether the functions are to be treated as a package. Multiple functions can have interaction effects with each other and combinations of functions can therefore have impacts which are not simply the sum of the individual effects. On the other hand it may not be feasible to get the functions to work separately — for example [ FCW] is now generally provided in combination with [ ACC] so that when using [ ACC] it is not possible to switch off [ FCW] (see section 1.1.4).

1. Step 1: Selection and description of functions

Usually it is quite clear from the beginning what functions or at least what type of functions will be the object of an FOT . However, to select the specific functions but also in case the type of functions has not yet been decided, a Stakeholders Analysis is recommended. During this analysis, the needs of the different stakeholders need to be identified and merged into a common requirements description. Stakeholders are those whose interests are affected by the issue or those whose activities strongly affect the issue, those who possess information, resources and expertise needed for strategy formulation and implementation, and those who control relevant implementations or instruments, like customers, public authorities, [ OEM]s, suppliers, and the scientific community. It is of vital importance that all relevant stakeholders are included in the analysis to guarantee that the selection process will not itself bias from the beginning the appraisal of the gained results.

It is recommended to evaluate the stakeholders’ needs by means of questionnaires, workshops or well documented interviews of stakeholders’ representatives. It is also quite important to describe the selection process sufficiently to prevent from misjudgement.

The basis for all following steps is a sufficient description of the selected functions. For these purposes a spreadsheet template has been prepared and is presented in the AnnexB to collect the necessary information. It provides two main parts: First, the functional classification, where a short high level description of the main aspects of the function should be given. This information is usually provided through the system specifications given by the system vendor or [ OEM]. The second part of the description comprises of limitations, boundary conditions and additional information which is necessary to understand how the function works.

The boundary conditions part describes where and under which circumstances the system/ function will operate according to its specifications, where the FOT should take place and which type of data needs to be recorded during the FOT to enable a good interpretation of the results. It consists of:

• Infrastructure requirements, [ cooperative systems] and [ nomadic devices] requirements. Here all required actors besides the actual system need to mentioned, which might have an impact on system performance, service availability or similar. It is intended to trigger the consideration of factors which are external to the system/ function under evaluation;
• Demographical requirements / driver requirements: Especially the user or driver recruitment needs to take into account, whether a function is particularly designed for a specific group of users or drivers. Drivers differ on a large variety of characteristics, which may all have an influence on how they drive and use different systems and services. These differences may be important to take into account when planning an FOT. Four categories of driver characteristics may be distinguished:

Demographic characteristics: gender, age, country, educational level, income, socio-cultural background, life and living situation, etc.;

Driving experience, and driving situation and motivation: experience in years and in mileage, professional, tourist, with or without passengers and children etc.;

Personality traits and physical characteristics: sensation seeking, locus of control, cognitive skills, physical impairments or weaknesses etc.;

Attitudes and intentions: attitudes towards safety, environment, technology etc.

• Geographical Requirements/ Road Context: This description is necessary for systems which, concerning their functionality, depend strongly on the horizontal or vertical curves of the road layout or on the road type. For example, certain speed limit information systems depend largely on the availability of speed limit information in a digital map, which is up to now only commercially available on high class roads.
• Geographical Requirements/ environmental restrictions: Certain systems are especially designed for specific environmental conditions or, on the other hand, specifications might indicate that the system under evaluation will not work under certain environmental conditions. In this case the location of the FOTneeds to be selected carefully and the relevant data must be recorded during the FOT. e.g., most of the functions using perception system will be affected by adverse weather conditions. If this is the case it is necessary to log respective data and take it into account for later data analysis.
• Geographical Requirements/ Traffic Context: The performance of certain systems might depend on the traffic context, that is, the traffic density (e.g. given by the Level of Service) or might even be designed to work in specific traffic densities only. Like the other geographical requirements, this needs to be taken into account when an FOT is planned, performed and the data is analysed.
• Other Limitations: All other limitations need to be mentioned, which might have considerable impact on the performance of functions or systems, since these limitations have major impact on the experimental design and data analysis.

  1. Step 2: Definition of use cases and situations

FOTs will test technically mature [ ICT] systems. Therefore, systems and functions to be tested are on the market or close to market and can be easily implemented. But the list grows too long if all possible implementation variations and technologies are considered separately. The use cases put the systems and functions at a suitable level of abstraction in order to group technology-independent functionalities and answer more holistic research questions described later.

Table 4-1: Use Cases, Situations, Scenarios, and their mutual dependence.

A use case is a textual presentation or a story about the usage of the system told from an end user’s perspective. [ Jacobson et al] (1995) defined use cases as follows: “When a user uses the system, she or he will perform a behaviourally related sequence of transactions in a dialogue with the system. We call such a special sequence a use case.” Use cases are technology-independent and the implementation of the system is not described. Use cases provide a tool for people with different background (e.g. software developers and non-technology oriented people) to communicate with each other. Use cases form the basic test case set for the system testing. There are number of different ways to define a use case. Use cases in FESTA are very general descriptions, like e.g. “car following”. This general description needs to be refined to a reasonable level of detail. This refinement is done by describing so called situations (see Table 4-1). It is the detailed scenario description which triggers the development of specific hypotheses for later analysis.

The situational descriptors are selected in a way that relevant information can be gathered to distinguish between main differences while evaluating systems. The situational descriptors can be distinguished as static and dynamic, where static describe attributes which will not change significantly during one ride of the vehicle, such as age or gender of the driver. Nevertheless this information needs to be stated, since it is one of the main inputs to filter the huge amounts of data in the later stage of data analysis. The second type of attribute is dynamic, since it can change during a ride of the vehicle, such as the system action status (system on or off), the traffic conditions, road characteristics or the environmental situation.

The situations are defined as a combination of certain characteristics of a use case. situations can be derived from use cases compiling a reasonable permutation of the use cases characteristics. The identification of possible situations is covered from three viewpoints:

1systems and vehicle specification,

2environmental conditions specification and

3driver characteristics and status specification.

The situational descriptors in FESTA conforms to the following structure: A set of basic rules has been set for the design of the situations for an FOT :

1 Complementary: situations are not allowed to overlap.

2 Entirety: the sum of all situations should describe the complete use case.

3 Baseline: The same situation without the use of the systems (system off or non-present) is defined as the baseline. The baseline is the basis for the benefit assessment of the system and the comparison between systems. Therefore, for the same use case, there can be many baselines depending on the number of situations.

4 Comparability: functions compared in an FOT need to have the same use cases and therefore same baseline and situations.

5 Variability of situation parameters: depending on the point of view (user, trip, vehicle, single FOT , multiple FOTs, etc…), attributes describing a situation can vary considerably or not.

This list is non-exhaustive and might be extended if necessary.

Finally, out of all the possible situations, one will need to select the relevant ones for scenarios of interest in an FOT . The scenarios are defined as a use cases in a specific situation and therefore one or more scenarios should be considered from each use case. All other situations should be considered out of the scope of the FOT study. However, if possible data should still be collected in all situations in case an alternative study would like to reuse the same data.

During FESTA a list of functions and use cases was produced based on technically mature [ ICT] systems and functions on the market. The list was consolidated based on the feedback from a stakeholders workshop and a dedicated questionnaire.

The process of defining the use cases will help the FOT for the next steps: the definition of the research questions and hypotheses and finally the identification of the needed indicators. The scenarios as they are defined at this stage of the FOT are not detailed enough for data analysis purposes. For this reason, after the definition of the indicators, the scenarios (and their situations) will need to be further described in terms of events for data analysis purposes. Only then, the scenarios can be classified with a quantitative measurement tools in function of the defined indicators.

1. Step 3: Identification of the research questions

The research questions specific to an FOT can only be identified once the overall goal of an FOT has been established.

In general terms the goal of any FOT is to investigate the impacts of mature [ ICT] technologies in real use. The core research questions should therefore focus on impacts but there are other questions that ‘surround’ this core. The range of possible questions is listed below. This list below should be considered a first step in any FOT and not a comprehensive set of questions.

ADD TABLE

LEVEL OF SYSTEM USAGE Which factors affect usage of the functions? Examples are • Purpose of journeys where system is used • Familiarity with routes where system is used • Portion of journey for which system is used • Types of road on which system is used • Traffic density • Headway • Weather condition • Ambient lighting How do driver characteristics affect usage of the functions? Examples are • Personal characteristics ( e. g. age, vision) • Socio-economic characteristics ( e. g. family, friends, employment status) • Journey-related characteristics ( e.g. other car occupants, shared driving) IMPACTS OF SYSTEM USAGE What are the impacts on safety? • Exposure • Risk of accident or injury • Incidents and near accidents • Accidents What are the impacts on personal mobility? • Individual driving behaviour • Travel behaviour • Comfort What are the impacts on traffic efficiency? • Traffic flow (speed, travel time, punctuality) • Traffic volume • Accessibility What are the impacts on the environment? • CO2 emissions • Particles • Noise IMPLICATIONS OF MEASURED IMPACTS What are the implications for policy? • Policy decisions • Laws, directives and enforcement • Future funding • Public authority implications • Emergency service implications What are the implications for business models? • Predictions for system uptake • User expectations • Pricing models What are the implications for system design and development? • HMI design and usability • Perceived value of service • Device design • Communications networks • Interoperability issues What are the implications for the public • Public information/education • Changes in legislation • Inclusive access to systems • Data protection

1. Step 4: Creation of hypotheses

Once the key research questions for the FOT have been identified, hypotheses can be derived. The process of formulating hypotheses translates the general research questions into more specific and statistically testable hypotheses.

There is no process that can assure that all the “correct” hypotheses are formulated. To a large extent, creating hypotheses is an intuitive process, in which a combination of knowledge and judgement is applied. Nevertheless, a number of recommendations can be made about how this process should be conducted. These recommendations have been tested in a FESTA workshop and modified based on the experience of and feedback from that workshop.

Two complementary ways to develop hypotheses have been used. Both ways need to be followed, while it is not of importance which step is taken first. One of the steps follows the sequential check of specific areas in which functions can have an impact; the other step is fully based on the description of specific scenarios. While the one step results mainly in general hypotheses, the other step triggers the development of very specific hypotheses in specific driving situations or scenarios.

Deriving hypotheses from the scenarios

The main reasoning to describe functions, their use cases, situations and scenarios in detail according to Steps 1 and 2 is to trigger the generation of hypotheses for very specific scenarios. The hypotheses generation should be conducted by a team of experts, consisting of human factors experts, development engineers and traffic engineers and all of them need to fully understand the functions/ systems with all aspects and limitations.

scenarios should be covered systematically. It is recommended that a structured approach be used and that the situations are checked sequentially for related hypotheses.

The six areas of impact

The six areas of impact defined by FESTA are based on [ Draskóczy et al]. (1998). Although this approach was originally designed for formulating hypotheses on traffic safety impacts, it is in fact equally applicable for efficiency and environmental impacts.

The six areas are:

• Direct effects of a system on the user and driving.
• Indirect (behavioural adaptation) effects of the system on the user.
• Indirect (behavioural adaptation) effects of the system on the non-user (imitating effect).
• Modification of interaction between users and non-users (including vulnerable road users).
• Modifying accident consequences (e.g. by improving rescue, etc. — note that this can affect efficiency and environment as well as safety).
• Effects of combination with other systems.

It is not of particular importance to which of these areas a particular hypotheses is allocated. The six areas are instead to be used as a checklist to ensure consideration of multiple aspects of system impact.

In applying this procedure, it should be noted that:

• Area 1 includes the human-machine interaction aspects of system use.
• The driving task (see Table 4-2) can be defined, following [ Michon] (1985) into the three levels of strategic, tactical and control (operational) aspects.
• Consideration should be given to such mediating factors as user/driver state, experience, journey purpose, etc.

It should also be noted that the effects of system use may be:

• Short-term or long-term in terms of duration and
• Intended or unintended in terms of system design.

This additional step for hypotheses generation assures that very general hypotheses are not forgotten as well as hypotheses on unintended, short term and long term effects. It is intended to serve as a means for crosschecking.

Table 4-2: Levels of the Driving Task by Michon (1985)

Prioritising the hypotheses

The prioritization among the generated hypotheses is a difficult process. No specific advice can be given on how to proceed, but there are some general guidelines:

A complete list of the hypotheses that have been developed should be recorded. If it is considered that some are too trivial or too expensive to address in the subsequent study design and data collection, the reasons for not covering them should be recorded. In general, it should be left to the judgement of the experts acting as hypotheses generators which hypotheses are likely to reflect the real driving situation. Those should then be prioritized, keeping in mind that also unintended effects are very important.

1. ''''''''''Step 5: Link hypotheses with indicators for quantitative analyses

Some of the hypotheses will already incorporate an indicator which needs to be measured, e.g. a very concrete like “The function will increase time-to-collision ([ TTC])”. In this case it is obvious which indicator to choose, while the method to measure [ TTC] might include complicated procedures and/ or costly measurement equipment. The Chapter on Performance Indicators gives an overview about many reasonable indicators. One should consider these indicators when planning the experimental design, since a detailed description how to calculate the indicators from measurements is also provided.

Other hypotheses might be rather unspecific, but still reasonable after rephrasing into testable ones. This rephrasing goes hand in hand with the identification of related reasonable indicators. For example, a hypothesis like “The function will increase lane changing performance” is not directly testable, since “lane change performance” is not an indicator itself. Hence, surrogate measures must be identified to evaluate lane change performance. These surrogate measures or indicators can e.g. be found in publications of corresponding research projects. If appropriate information cannot be found or is not accessible, new performance indicators need to be developed. Those indicators and the measurement methodology must be valid, reliable and sensitive, that is, the measurement must actually measure what it is supposed to measure, they must be reproducible and the measurments must be sensitive to changes of the variable. A sensitivity analysis should be performed beforehand during a pilot study to make sure that the new performance indicator is suitable. When one or more surrogate measures have been identified, the initial hypothesis can be reformulated into one or more testable hypotheses. In the above mentioned example, reasonable indicators associated to “lane change performance” might be: use of turning indicator or the number of lane change warnings. The initial hypothesis will then be reformulated into: “The system will increase the use of the turning indicator.” and “During the system use, the number of lane departure warnings will decrease”. The next step is then to evaluate how the indicators “use of turning indicator” and “lane departure warnings” can be measured. In this context, the chapter on Performance Indicators provides useful information.

Iteration

Iteration is especially important when defining research questions and hypotheses, because usually a selection has to be made from the large amount of possible hypotheses, both based on their relation with the main impact areas and research questions and on practical issues. Another important iteration point is the impact areas. The final question on the impact assessment may drive the design of the FOT in all its aspects. When practical issues, such as for example which data-loggers to use, make certain choices hard to realise, iteration to earlier stages is necessary. Cost-benefits analyses and feasibility assessment of different options for the FOT may also drive the design. It is important that there is a good communication between the project members who are in charge of defining research questions and hypotheses and the ones who will be analysing the data in order to assure that the questions can indeed be answered.