l5kit.evaluation.metrics module

class l5kit.evaluation.metrics.CollisionType(value)

Bases: enum.IntEnum

This enum defines the three types of collisions: front, rear and side.

FRONT = 0
REAR = 1
SIDE = 2
l5kit.evaluation.metrics.average_displacement_error_mean(ground_truth: numpy.ndarray, pred: numpy.ndarray, confidences: numpy.ndarray, avails: numpy.ndarray) → numpy.ndarray

Calls _average_displacement_error() to get the mean average displacement error.

Parameters

  • ground_truth (np.ndarray) – array of shape (time)x(2D coords)

  • pred (np.ndarray) – array of shape (modes)x(time)x(2D coords)

  • confidences (np.ndarray) – array of shape (modes) with a confidence for each mode in each sample

  • avails (np.ndarray) – array of shape (time) with the availability for each gt timestep

Returns

mean average displacement error (ADE), a single float number

Return type

np.ndarray

l5kit.evaluation.metrics.average_displacement_error_oracle(ground_truth: numpy.ndarray, pred: numpy.ndarray, confidences: numpy.ndarray, avails: numpy.ndarray) → numpy.ndarray

Calls _average_displacement_error() to get the oracle average displacement error.

Parameters

  • ground_truth (np.ndarray) – array of shape (time)x(2D coords)

  • pred (np.ndarray) – array of shape (modes)x(time)x(2D coords)

  • confidences (np.ndarray) – array of shape (modes) with a confidence for each mode in each sample

  • avails (np.ndarray) – array of shape (time) with the availability for each gt timestep

Returns

oracle average displacement error (ADE), a single float number

Return type

np.ndarray

l5kit.evaluation.metrics.detect_collision(pred_centroid: numpy.ndarray, pred_yaw: numpy.ndarray, pred_extent: numpy.ndarray, target_agents: numpy.ndarray) → Optional[Tuple[l5kit.evaluation.metrics.CollisionType, str]]

Computes whether a collision occured between ego and any another agent. Also computes the type of collision: rear, front, or side. For this, we compute the intersection of ego’s four sides with a target agent and measure the length of this intersection. A collision is classified into a class, if the corresponding length is maximal, i.e. a front collision exhibits the longest intersection with egos front edge.

Note

please note that this funciton will stop upon finding the first colision, so it won’t return all collisions but only the first one found.

Parameters

  • pred_centroid – predicted centroid

  • pred_yaw – predicted yaw

  • pred_extent – predicted extent

  • target_agents – target agents

Returns

None if not collision was found, and a tuple with the collision type and the agent track_id

l5kit.evaluation.metrics.distance_to_reference_trajectory(pred_centroid: torch.Tensor, ref_traj: torch.Tensor) → torch.Tensor

Computes the distance from the predicted centroid to the closest waypoint in the reference trajectory.

Parameters

  • pred_centroid (torch.Tensor, float) – predicted centroid tensor, size: [batch_size, 2]

  • ref_traj (torch.Tensor, float) – reference trajectory tensor, size: [batch_size, num_timestamps, 2]

Returns

closest distance between the predicted centroid and the reference trajectory, size: [batch_size,]

Return type

torch.Tensor, float

l5kit.evaluation.metrics.final_displacement_error_mean(ground_truth: numpy.ndarray, pred: numpy.ndarray, confidences: numpy.ndarray, avails: numpy.ndarray) → numpy.ndarray

Calls _final_displacement_error() to get the mean average displacement error.

Parameters

  • ground_truth (np.ndarray) – array of shape (time)x(2D coords)

  • pred (np.ndarray) – array of shape (modes)x(time)x(2D coords)

  • confidences (np.ndarray) – array of shape (modes) with a confidence for each mode in each sample

  • avails (np.ndarray) – array of shape (time) with the availability for each gt timestep

Returns

mean final displacement error (FDE), a single float number

Return type

np.ndarray

l5kit.evaluation.metrics.final_displacement_error_oracle(ground_truth: numpy.ndarray, pred: numpy.ndarray, confidences: numpy.ndarray, avails: numpy.ndarray) → numpy.ndarray

Calls _final_displacement_error() to get the oracle average displacement error.

Parameters

  • ground_truth (np.ndarray) – array of shape (time)x(2D coords)

  • pred (np.ndarray) – array of shape (modes)x(time)x(2D coords)

  • confidences (np.ndarray) – array of shape (modes) with a confidence for each mode in each sample

  • avails (np.ndarray) – array of shape (time) with the availability for each gt timestep

Returns

oracle final displacement error (FDE), a single float number

Return type

np.ndarray

l5kit.evaluation.metrics.neg_multi_log_likelihood(ground_truth: numpy.ndarray, pred: numpy.ndarray, confidences: numpy.ndarray, avails: numpy.ndarray) → numpy.ndarray

Compute a negative log-likelihood for the multi-modal scenario. log-sum-exp trick is used here to avoid underflow and overflow, For more information about it see: https://en.wikipedia.org/wiki/LogSumExp#log-sum-exp_trick_for_log-domain_calculations https://timvieira.github.io/blog/post/2014/02/11/exp-normalize-trick/ https://leimao.github.io/blog/LogSumExp/

For more details about used loss function and reformulation, please see https://github.com/woven-planet/l5kit/blob/master/competition.md.

Parameters

  • ground_truth (np.ndarray) – array of shape (timesteps)x(2D coords)

  • pred (np.ndarray) – array of shape (modes)x(timesteps)x(2D coords)

  • confidences (np.ndarray) – array of shape (modes) with a confidence for each mode in each sample

  • avails (np.ndarray) – array of shape (timesteps) with the availability for each gt timesteps

Returns

negative log-likelihood for this example, a single float number

Return type

np.ndarray

l5kit.evaluation.metrics.prob_true_mode(ground_truth: numpy.ndarray, pred: numpy.ndarray, confidences: numpy.ndarray, avails: numpy.ndarray) → numpy.ndarray

Return the probability of the true mode

Parameters

  • ground_truth (np.ndarray) – array of shape (timesteps)x(2D coords)

  • pred (np.ndarray) – array of shape (modes)x(timesteps)x(2D coords)

  • confidences (np.ndarray) – array of shape (modes) with a confidence for each mode in each sample

  • avails (np.ndarray) – array of shape (timesteps) with the availability for each gt timesteps

Returns

a (modes) numpy array

Return type

np.ndarray

l5kit.evaluation.metrics.rmse(ground_truth: numpy.ndarray, pred: numpy.ndarray, confidences: numpy.ndarray, avails: numpy.ndarray) → numpy.ndarray

Return the root mean squared error, computed using the stable nll

Parameters

  • ground_truth (np.ndarray) – array of shape (timesteps)x(2D coords)

  • pred (np.ndarray) – array of shape (modes)x(timesteps)x(2D coords)

  • confidences (np.ndarray) – array of shape (modes) with a confidence for each mode in each sample

  • avails (np.ndarray) – array of shape (timesteps) with the availability for each gt timesteps

Returns

negative log-likelihood for this example, a single float number

Return type

np.ndarray

l5kit.evaluation.metrics.time_displace(ground_truth: numpy.ndarray, pred: numpy.ndarray, confidences: numpy.ndarray, avails: numpy.ndarray) → numpy.ndarray

Return the displacement at timesteps T

Parameters

  • ground_truth (np.ndarray) – array of shape (timesteps)x(2D coords)

  • pred (np.ndarray) – array of shape (modes)x(timesteps)x(2D coords)

  • confidences (np.ndarray) – array of shape (modes) with a confidence for each mode in each sample

  • avails (np.ndarray) – array of shape (timesteps) with the availability for each gt timesteps

Returns

a (timesteps) numpy array

Return type

np.ndarray