Human Factors Evaluation of Level 2 and Level 3 Automated Driving Concepts

From FOT-Net WIKI
Jump to: navigation, search
Contact details

Myra Blanco, Jon Atwood, Holland M. Vasquez, Tammy E. Trimble, Vikki L. Fitchett, Joshua Radlbeck, Gregory M. Fitch, & Sheldon M. Russell. Virginia Tech Transportation Institute (VTTI) Center for Automated Vehicle Systems, 3500 Transportation Research Plaza Blacksburg, Virginia 24061.

Timing and duration of tests

Experiment 1: One 90-minute driving session, ~ 175 minutes for the entire experimental session Experiment 2: Three 1-hour driving sessions, ~ 310 minutes for the entire experimental session Experiment 3: Three 30-minute driving sessions, ~200 minutes for the entire experimental session

Location(s) of tests

Experiments 1 and 2: Milford Proving Ground circle track in Milford, MI Experiment 3: Virginia Smart Road test track in Blacksburg, VA

Tested automation

Tested functionalities

Experiment 1: Alerting Operators to Regain Control of the Vehicle Experiment 2: System Prompt Effectiveness Over Time Experiment 3: Human-Automation System Performance Over Time

Level of automation tested

Experiments 1 and 2: Level 2 – Combined Function Automation (NHTSA) Experiment 3: Level 3 - Limited Self-Driving Automation (NHTSA)

Tested use cases

Experiment 1: Take control alerts (cautionary, imminent, and staged ) and alert modality (unimodal and multimodal) Experiment 2: Prompt conditions: 2-seconds, 7-seconds, or no prompts; event type (alert, no alert, and no lane drift); and three driving sessions. Experiment 3: Take control alert (staged, imminent-external threat, and imminent-no external threat; and three driving sessions.

Tested transport system

Prototype vehicles outfitted with Level 2 and Level 3 automation capabilities on a test track simulating a mixed traffic highway driving condition.

Purposes of testing
  • Assessment of driver behaviour (human, vehicle) / road user behaviour
  • Assessment of user (driver, traveller, etc.) acceptance, usability, take up, etc.
  • Six research questions: 1. How do drivers interact with and operate vehicles that offer Level 2 and Level 3 automation? 2. What are the system performance risks from operator involvement in secondary tasks? 3. What are the most effective hand-off strategies between the system and the operator? 4. How do operators engage, disengage, and re-engage with the driving task in response to the various states of Level 2 and Level 3 automation? 5. How do operators perform under various operational concepts with Level 2 and Level 3 automation? 6. What are the most effective human-machine interface concepts?

Methodology

Definition of baseline

Not applicable.

Test design

Experiment 1: All participants completed one driving session during which they received a total of 19 system alerts. Specifically, each participant experienced six Cautionary alerts, six Staged alerts, and six Imminent alerts. For each of those alert types, participants experienced three unimodal alerts (visual only) and three multimodal alerts (visual + haptic). After receiving these 18 alerts, each participant received an Imminent multimodal alert coupled with an experimenter-triggered lane drift, resulting in a total of 19 alerts. The study was designed to mimic worst-case scenarios when conditions for monitoring the roadway were decreased. During the driving session, participants were instructed to perform non-driving-related tasks (e.g., e-mail, Web browsing) and were, at times, presented with alerts stating that they must take control of the vehicle. Three forms of alerts were presented: Cautionary, Imminent, and Staged. The Cautionary alerts provided information to the participants that a potential problem was detected. The Imminent alerts provided the participants with a message that an active fault was detected. The Staged alerts transitioned from a cautionary alert phase to an imminent alert phase.

Experiment 2: Each participant completed three successive driving sessions, and each session included one of the following: a lane drift with an alert, a lane drift without an alert, or no lane drift. Participants experienced each of these conditions once during the experiment. In addition, there were also three different prompt conditions that were used with the driver monitoring system, and each participant experienced only one prompt condition, either 2-second, 7-second, or No Prompts. The prompt timing was based on previous distraction research (2-second prompts) (e.g., Klauer et al., 2006) and expert opinion (7-second prompts). Additionally, the study was designed to mimic worst-case scenarios when conditions for monitoring the roadway are decreased. During the sessions, participants were given tasks to be completed using a tablet computer. During these tasks, participants received prompts based on their predetermined prompt condition (either 2-second, 7-second, or No Prompts). For the 2- or 7-second prompt conditions, participants received prompts after periods of inattention for the corresponding amount of time. Participants given the No Prompts condition did not receive any prompts and they were free to behave as they thought was appropriate. In addition to these prompts, at a random time during one predetermined session, the participant received an alert for a surprise left lane drift, consisting of a haptic seat alert and a flashing red LED. In a different predetermined session, the participant experienced a surprise lane drift with no alert, which consisted only of a left lane drift without any alert and with the prompting system disabled. The experimenter-injected lane drift was used to simulate a lane-keeping performance issue combined with a failure of the prompting system. Note that, to the participants with the 2-second and 7-second prompt conditions, the alert that they received along with the lane drift was indistinguishable from the prompts that they had been receiving based on their attention state.

Experiment 3: Each participant completed three successive driving sessions, each session with one of the three alert types; all participants received all alert types exactly once. The three alert types were Staged, Imminent–External Threat, and Imminent–No External Threat. During the sessions, participants had free exposure to a non-driving-related task (i.e., use of their own cell phone or the provided tablet as they felt was appropriate) and were presented with a message stating that they must take control of the vehicle.

Method of testing

Interviews, Questionnaires, Simulation, Test track


Test fleet, participants and environment

Number and make of vehicles

Experiment 1: This experiment used a 2009 Chevrolet Malibu equipped with a prototype L2 automated driving system. As part of the automated driving system, several Human-Machine Interface (HMI) components were installed. This vehicle was modified to include ACC and lane centering, along with a flexible driver interface and researcher’s control console. The purpose of the researcher’s console was to allow the in-vehicle experimenter to trigger various displays and to change the operation of the automated driving system, which included simulating erroneous behavior and equipment failures. The vehicle was also instrumented with a data collection and recording device. The data recorder was connected to the automated driving system and the vehicle Controller Area Network (CAN).

Experiment 2: This experiment used a 2010 model year Cadillac SRX equipped with a prototype L2 automated driving system. As part of the automated driving system, several HMI components were installed. These included an instrument panel binnacle-mounted screen providing information on the automated driving system, and two steering wheel buttons to control the automation: one ACC button and one button for the lane-centering system, a prototype automated vehicle system. The vehicle was instrumented with Virginia Tech Transportation Institute’s (VTTI) Data Acquisition System (DAS). The DAS was connected to the prototype automated driving system via Ethernet.

Experiment 3: This experiment used a 2012 Lexus RX450h. This L2 vehicle was equipped with a prototype automated driving system that can simulate L3 driving on a test track. As part of the prototype system, several HMI components were installed. These included an instrument panel binnacle-mounted screen providing information on the automated driving system and two steering wheel buttons to control the automation: one on button on the left side of the wheel and one off button on the right side of the wheel. The vehicle was instrumented with VTTI’s DAS. The DAS was connected to the automated driving system via Ethernet.

Description and number of participants/drivers

Experiment 1: Data were collected from 35 participants; however, 10 participants were considered invalid (i.e., session cancellation due to adverse weather, track closures, or technical issues associated with the prototype vehicle). The analysis presented in this chapter represents data from 25 participants (16 males, 9 females). The mean age of participants was 44.3 years old (standard deviation [S.D.] = 19.24), with ages ranging from 18 to 72 years old.

Experiment 2: Data were collected from 56 participants (28 males, 28 females) with a mean age of 41 years old (S.D. = 16.3), ranging from 18 to 72 years old.

Experiment 3: Data were collected from 37 participants; however, 12 participants were considered invalid (i.e., session cancellation due to adverse weather, technical issues associated with the DAS or the prototype vehicle). The analysis presented in this chapter consists of data from 25 participants (17 males, 8 females). The mean age of the participants was 38.8 years old (S.D. = 13.77), with ages ranging from 18 to 69 years old.

Tested environment and facilities

Test track simulating a mixed traffic highway driving condition.

Legal and ethical aspects

None noted.

Duration of testing

Experiment 1: One 90-minute driving session, ~ 175 minutes for the entire experimental session Experiment 2: Three 1-hour driving sessions, ~ 310 minutes for the entire experimental session Experiment 3: Three 30-minute driving sessions, ~200 minutes for the entire experimental session

Input parameters and assumptions of simulation tests

N/A

Data

Logging
  • CAN
  • HMI
  • In-vehicle cameras
  • System internal data


Key Performance Indicators (KPIs)
  • Time to react to alert (Experiments 1, 2, and 3)
    • Time to react to event (Experiment 2)
    • Time to react to prompt (Experiment 2)
  • Time to regain control (Experiments 1, 2, and 3)
  • Performance (Experiments 1, 2, and 3)
  • Method used to regain control/ cancel automation (Experiment 3)
  • Time to release control (Experiments 1, 2, and 3)
    • Time to activate automation
    • Time to release control of steering
  • Time to resume non-driving task (Experiments 1, 2, and 3)
  • Monitoring rate and non-driving related glances (Experiments 2 and 3)
Situational data available

Experiment 1: Key variables collected included status of the automation (e.g., off, on and actively controlling, failure mode), vehicle speed, lane position, and flags indicating the presentation of messages and system failures. In addition, the following video views were collected: Operator’s face; Over-the-shoulder (OTS) view; Forward roadway; Rear; Foot (pedal area); HMI; Exterior left rear; and Exterior right rear.

Experiment 2: The variables collected by the DAS included status of the automation, vehicle speed, and lane position. In addition, the following video views were collected: Operator’s face; OTS view; Forward roadway; Exterior left rear; Foot (pedal area); and HMI.

Experiment 3: The variables collected by the DAS included throttle/brake input and automation state. In addition, the following video views were collected: Operator’s face; Over the shoulder; Forward roadway; Exterior left rear; Foot (pedal area); and HMI.

Subjective data collected

Experiment 1: 10 in-vehicle trust scales; after-experience trust scale; and an open-ended interview

Experiment 2: 21 in-vehicle trust scales; 3 post-session trust scales and 3 post-session satisfaction scales; and an open-ended interview.

Experiment 3: 12 in-vehicle trust scales; after-experience trust scale; and an open-ended interview.

Results

Issues that affected the impact assessment

Experiment 1: Data were collected from 35 participants; however, 10 participants were considered invalid (i.e., session cancellation due to adverse weather, track closures, or technical issues associated with the prototype vehicle).

Experiment 2: None known.

Experiment 3: Data were collected from 37 participants; however, 12 participants were considered invalid (i.e., session cancellation due to adverse weather, technical issues associated with the DAS or the prototype vehicle).

Results

Experiment 1:

  • The alert modality significantly affected the time taken to regain control of the vehicle. Participants regained control of the vehicle significantly faster when the visual alert included a haptic component compared to when the alert was just visual.
  • Participants regained control sooner when alerted by an Imminent visual alert comprising a red light-emitting diode compared to a Cautionary visual alert comprising a yellow LED.
  • Participants regained control of the vehicle just as fast to the Imminent visual and haptic alert when it corresponded with the unexpected lane drift as when it was presented without the unexpected lane drift. However, most participants reacted to the alert before the vehicle had moved significantly in the lane, making the experience very similar to an alert generated in the absence of the lane drift.
  • Overall, participants reported that they greatly trusted the partial automation before, during, and after experiencing it.

Experiment 2:

  • Prompting was successful in getting participants to monitor the road. The results suggest that a 2-second prompt encourages operators to monitor the driving environment more than a 7-second prompt when operating a vehicle equipped with an L2 system.
    • The 7-second prompts increased participants’ attention to the road after they were presented. This increase was also sustained over time. However, because they were only issued when participants exhibited 7 consecutive seconds of inattention, they were presented only when participants were extremely inattentive to the driving environment.
    • The 2-second prompts were not found to increase participants’ attention to the driving environment after they were presented. However, they did lead to the highest amount of attentiveness over the course of the experiment. This may have occurred because the 2-second prompts were presented more frequently than the 7-second prompts, and were presented before participants became extremely inattentive to the driving environment.
  • The visual plus haptic Imminent alert was effective at getting participants to regain control of the vehicle when an unexpected lane drift occurred.
    • Participants took 2.4 s to regain control of the vehicle in response to an unexpected lane drift if they received an alert. (Note that the alert and the lane drift began simultaneously.)
    • Many participants had to be instructed to regain control of the vehicle when they did not receive an alert. Those regaining control without an alert or an instruction from the experimenter to regain control took 4.4 s to regain control.
  • Participants reported that they greatly trusted the partial automation before, during, and after experiencing it. However, participants that experienced a lane keeping performance issue without an alert lost some trust in the automation.

Experiment 3:

  • In the absence of an external threat, participants were faster at regaining control of the vehicle when they received an Imminent alert compared to when they received a Staged alert.
    • The Imminent alert presented direct instructions to take control now, while the information phase of the Staged alert presented a message instructing operators to prepare for manual control along with a countdown timer.
    • The long response time may not be attributable to participants ignoring the informational phase of the Staged alert, but rather that they were following the instructions given by the HMI.
    • It is also possible that because participants had seen a video demonstrating all four phases of the Staged alert, they might not have felt an urgency to take control right away.
    • All participants receiving the Staged alert regained control before the alert reached its third phase.
  • Participants were equally as fast at regaining control of the L3 automated vehicle after an Imminent alert in the presence of an external threat (a box) as they were in the absence of an external threat.
  • Trust was rated very high prior to the experiment. In addition, trust significantly increased over time.
  • After an alert, the time taken to release control of the driving task once the automation was engaged was found to be affected by the severity of the preceding alert and scenario, even though participants still reported that they had high trust.

Summary: This study shows that different HMI elements can have a large impact on how operators interact with partially automated vehicles. Overall, participants greatly trusted the capabilities of the automated systems. Participants were also observed prioritizing non-driving activities over the operation of the vehicle and disregarding take-over requests(TORs) when they were presented. Future research could explore these issues to help optimize the automation’s HMI. The driver engagement patterns observed in this study provide data and evidence that could support the future development of human factors design principles for L2 and L3 partially automated vehicles.

Publications

http://www.nhtsa.gov/DOT/NHTSA/NVS/Crash%20Avoidance/Technical%20Publications/2015/812182_HumanFactorsEval-L2L3-AutomDrivingConcepts.pdf

Other things to report

Method of testingInterviews +, Questionnaires +, Simulation + and Test track +
Purpose of testingAssessment of driver behaviour (human, vehicle) / road user behaviour + and Assessment of user (driver, traveller, etc.) acceptance, usability, take up, etc. +