l5kit.cle.composite_metrics module¶

class l5kit.cle.composite_metrics.CompositeMetricAggregator(reduce_mode: l5kit.cle.composite_metrics.ReduceMode = ReduceMode.SUM)¶

Bases: l5kit.cle.composite_metrics.SupportsCompositeMetricAggregation

Aggregates composite metric results locally and across multiple distributed nodes.

Parameters: reduce_mode – the reduce operation to be applied both into local scenes and also across distributed nodes.

aggregate(scene_composite_metric_results: Dict[int, Dict[str, float]]) → Dict[str, Any]¶

This method will aggregate scenes locally and then will do the reduction step to aggregate data across distributed nodes.

Parameters: scene_composite_metric_results – results from composite metric outputs per scene
Returns: result indexed by the composite metric name.

aggregate_scenes(scene_composite_metric_results: Dict[int, Dict[str, float]]) → Dict[str, Any]¶

Aggregate the scenes locally on each node. This method will just sum the number of invalid scenes across all scenes in the node.

Parameters: scene_composite_metric_results – results from composite metric outputs per scene
Returns: a dictionary with the composite metric name as keys and the reduction of the composite metrics

class l5kit.cle.composite_metrics.DrivenMilesCompositeMetric(composite_metric_name: str, driven_miles_metric: Type[l5kit.cle.metrics.SupportsMetricCompute] = <class 'l5kit.cle.metrics.SimulatedDrivenMilesMetric'>)¶

Bases: l5kit.cle.composite_metrics.SupportsCompositeMetricCompute

Composite metric to accumulate the total driven miles.

Parameters

composite_metric_name – name of the composite metric
driven_miles_metric – the metric that should be used to accumulate its values, defaults to the simulated driven miles

composite_metric_name: str¶: Name of the composite metric

compute(metric_results: Dict[str, torch.Tensor], validation_results: Dict[str, l5kit.cle.validators.ValidatorOutput], simulation_output: l5kit.simulation.unroll.SimulationOutputCLE) → float¶

Method that supports the computation of the composite metric. This metric should return a single float per scene.

Parameters

metric_results – results of the metrics required
validation_results – results from the validators required
simulation_output – output from the closed-loop simulator

Tensor metric_results

[N], where N is the number of frames in the scene

Returns

a float result

requires_metric: List[str]¶: List of names for metrics this composite metric depends on

requires_validator: List[str]¶: List of validators that this composite metric depends on

class l5kit.cle.composite_metrics.PassedDrivenMilesCompositeMetric(composite_metric_name: str, intervention_validators: List[str], driven_miles_metric: Type[l5kit.cle.metrics.SupportsMetricCompute] = <class 'l5kit.cle.metrics.SimulatedDrivenMilesMetric'>, ignore_entire_scene: bool = False)¶

Bases: l5kit.cle.composite_metrics.SupportsCompositeMetricCompute

This composite metric will compute the “passed driven miles”, which represents the sum of driven miles up to a first intervention frame that is detected based on the list of specified validators.

Parameters

intervention_validators – an intervention is a defined by any failed frame from any of the specified validators.
driven_miles_metric – the metric that should be used as driven miles (defaults to simulated driven miles), but can also use the driven miles by the log replay.
ignore_entire_scene – if the entire driven miles from the scene should be ignored when there was an intervention.

composite_metric_name: str¶: Name of the composite metric

Computes the driven miles until the first intervention is found (first failed frame).

Parameters

metric_results – results of the metrics required
validation_results – results from the validators required
simulation_output – output from the closed-loop simulator

Tensor metric_results

[N], where N is the number of frames in the scene

Returns

passed driven miles

requires_metric: List[str]¶: List of names for metrics this composite metric depends on

requires_validator: List[str]¶: List of validators that this composite metric depends on

class l5kit.cle.composite_metrics.ReduceMode(value)¶

Bases: enum.IntEnum

Reduction modes supported by the composite metric aggregator.

SUM = 0¶: Summation of results locally and across nodes

class l5kit.cle.composite_metrics.SupportsCompositeMetricAggregation(*args, **kwargs)¶

Bases: Protocol

Protocol supporting the composite metric aggregation. This aggregator is responsible for aggregating results from the the composite metrics and also doing a reduction across multiple distributed nodes.

aggregate(scene_composite_metric_results: Dict[int, Dict[str, float]]) → Dict[str, Any]¶

This method will aggregate scenes locally and then will do the reduction step to aggregate data across distributed nodes.

Parameters: scene_composite_metric_results – results from composite metric outputs per scene
Returns: any result indexed by the composite metric name.

class l5kit.cle.composite_metrics.SupportsCompositeMetricCompute(*args, **kwargs)¶

Bases: Protocol

Protocol supporting the computation method for metrics.

composite_metric_name: str¶: Name of the composite metric

abstract compute(metric_results: Dict[str, torch.Tensor], validation_results: Dict[str, l5kit.cle.validators.ValidatorOutput], simulation_output: l5kit.simulation.unroll.SimulationOutputCLE) → float¶

Method that supports the computation of the composite metric. This metric should return a single float per scene.

Parameters

metric_results – results of the metrics required
validation_results – results from the validators required
simulation_output – output from the closed-loop simulator

Tensor metric_results

[N], where N is the number of frames in the scene

Returns

a float result

requires_metric: List[str]¶: List of names for metrics this composite metric depends on

requires_validator: List[str]¶: List of validators that this composite metric depends on