l5kit.cle.metric_set module¶

class l5kit.cle.metric_set.BaseMetricSet¶

Bases: abc.ABC

Base class interface for the metric sets.

abstract evaluate(sim_outputs: List[l5kit.simulation.unroll.SimulationOutputCLE]) → None¶

Run the CLE (Closed-loop Evaluator) on simulated scenes.

abstract get_results() → Dict[str, Any]¶: Perform all required aggregations and returns a dictionary composed by all results.

class l5kit.cle.metric_set.L5MetricSet(metric_prefix: str = 'L5')¶

This class is responsible for computing a set of metric parametrization for the L5Kit.

Parameters: metric_prefix – this is a prefix that will identify the metric set being used.

aggregate_failed_frames() → Dict[str, Any]¶: This method will aggregate the failed scenes and will return a dictionary indexed by the validator name associated with a list with FailedFrame items containing the scene_id and the frame index that triggered the validator.

build_composite_metrics() → List[l5kit.cle.composite_metrics.SupportsCompositeMetricCompute]¶: Return a list of composite metrics that should be computed. Composite metrics are metrics that depend upon metrics and validator results.

abstract build_metrics() → List[l5kit.cle.metrics.SupportsMetricCompute]¶: Returns a list of metrics that will be computed.

build_validators() → List[l5kit.cle.validators.SupportsMetricValidate]¶: Returns a list of validators that will operate on the computed metrics.

Run the CLE (Closed-loop Evaluator) on simulated scenes.

get_results() → Dict[str, Any]¶: Perform all required aggregations and returns a dictionary composed by all results.

get_validator_interventions() → List[str]¶: Returns a list of validators that are considered an intervention.

reset() → None¶: Reset the current state of the CLE (Closed-loop Evaluator).