l5kit.cle.metrics module

class l5kit.cle.metrics.CollisionFrontMetric

Bases: l5kit.cle.metrics.CollisionMetricBase

Computes the front collision metric.

metric_name = 'collision_front'

class l5kit.cle.metrics.CollisionMetricBase(collision_type: l5kit.evaluation.metrics.CollisionType)

Bases: abc.ABC

This is the abstract base class for the collision metric.

Parameters

collision_type – the type of collision to compute

compute(simulation_output: l5kit.simulation.unroll.SimulationOutputCLE) → torch.Tensor

Compute the metric on all frames of the scene.

Parameters

simulation_output – the output from the closed-loop simulation

Returns

collision per frame (a 1D array with the same size of the number of frames, where 1 means a colision, 0 otherwise)

class l5kit.cle.metrics.CollisionRearMetric

Bases: l5kit.cle.metrics.CollisionMetricBase

Computes the rear collision metric.

metric_name = 'collision_rear'

class l5kit.cle.metrics.CollisionSideMetric

Bases: l5kit.cle.metrics.CollisionMetricBase

Computes the side collision metric.

metric_name = 'collision_side'

class l5kit.cle.metrics.DisplacementErrorL2Metric

Bases: l5kit.cle.metrics.DisplacementErrorMetric

Displacement error calculated with euclidean distance.

metric_name: str = 'displacement_error_l2'

class l5kit.cle.metrics.DisplacementErrorMetric(error_function: Callable[[torch.Tensor, torch.Tensor], torch.Tensor])

Bases: l5kit.cle.metrics.SupportsMetricCompute

Displacement error computes the elementwise distance from the simulated trajectory and the observed trajectory.

Parameters

error_function – error function to compute distance

compute(simulation_output: l5kit.simulation.unroll.SimulationOutputCLE) → torch.Tensor

Compute the metric on all frames of the scene.

Parameters

simulation_output – the output from the closed-loop simulation

Returns

distance per frame [Shape: N, where N = timesteps]

metric_name: str = 'displacement_error'

class l5kit.cle.metrics.DistanceToRefTrajectoryMetric(scene_fraction: float = 0.8)

Bases: l5kit.cle.metrics.SupportsMetricCompute

Distance to reference trajectory metric. This metric will compute the distance from the predicted centroid to the closest waypoint in the reference trajectory.

Note

Please note that this metric is different than the displacement error because it is taking into consideration the entire reference trajectory at each point of the simulated trajectory.

Parameters

scene_fraction – fraction of the simulated scene used to evaluate against the reference trajectory. This fraction should be between 0.0 and 1.0.

compute(simulation_output: l5kit.simulation.unroll.SimulationOutputCLE) → torch.Tensor

Compute the metric on all frames of the scene.

Parameters

simulation_output – the output from the closed-loop simulation

Returns

distance to reference trajectory per frame [Shape: N, where N = timesteps]

metric_name: str = 'distance_to_reference_trajectory'

class l5kit.cle.metrics.ReplayDrivenMilesMetric

Bases: object

This metric will compute the driven miles per frame for the observed trajectory, the one in the log replay (as opposed to the one simulated).

METER_TO_MILES = 0.000621371

compute(simulation_output: l5kit.simulation.unroll.SimulationOutputCLE) → torch.Tensor

Compute the metric on all frames of the scene.

Parameters

simulation_output – the output from the closed-loop simulation

Returns

driven miles per each frame

metric_name = 'replay_driven_miles'

class l5kit.cle.metrics.SimulatedDrivenMilesMetric

Bases: object

This metric will compute the driven miles per frame for the simulated trajectory (as opposed to the one in the log replay).

METER_TO_MILES = 0.000621371

compute(simulation_output: l5kit.simulation.unroll.SimulationOutputCLE) → torch.Tensor

Compute the metric on all frames of the scene.

Parameters

simulation_output – the output from the closed-loop simulation

Returns

driven miles per each frame

metric_name = 'simulated_driven_miles'

class l5kit.cle.metrics.SimulatedVsRecordedEgoSpeedMetric(*args, **kwargs)

Bases: l5kit.cle.metrics.SupportsMetricCompute

This metric computes the speed delta between recorded and simulated ego. When simulated ego is traveling faster than recorded ego, this metric is > 0. When simulated ego is traveling slower than recorded ego, this metric is < 0. We can use this metric in conjunction with a RangeValidator to identify cases where simulated ego is consistently traveling much faster (or much slower) than recorded ego.

compute(simulation_output: l5kit.simulation.unroll.SimulationOutputCLE) → torch.Tensor

The compute method sould return the result of the metric computed at every frame of the scene.

Parameters

simulation_output – the output from the closed-loop simulation

Returns

a tensor with the result of the metric per frame

metric_name: str = 'simulated_minus_recorded_ego_speed'

class l5kit.cle.metrics.SupportsMetricCompute(*args, **kwargs)

Bases: Protocol

Protocol supporting the computation method for metrics.

abstract compute(simulation_output: l5kit.simulation.unroll.SimulationOutputCLE) → torch.Tensor

The compute method sould return the result of the metric computed at every frame of the scene.

Parameters

simulation_output – the output from the closed-loop simulation

Returns

a tensor with the result of the metric per frame

metric_name: str

class l5kit.cle.metrics.YawErrorCAMetric

Bases: l5kit.cle.metrics.YawErrorMetric

Yaw error calculated with closest angle.

metric_name: str = 'yaw_error_closest_angle'

class l5kit.cle.metrics.YawErrorMetric(error_function: Callable[[torch.Tensor, torch.Tensor], torch.Tensor] = <function closest_angle_error>)

Bases: l5kit.cle.metrics.SupportsMetricCompute

Yaw error computes the difference between the simulated trajectory yaw and the observed trajectory yaw.

Parameters

error_function – error function to compute distance

compute(simulation_output: l5kit.simulation.unroll.SimulationOutputCLE) → torch.Tensor

Compute the metric on all frames of the scene.

Parameters

simulation_output – the output from the closed-loop simulation

Returns

distance per frame [Shape: N, where N = timesteps]

metric_name: str = 'yaw_error'