l5kit.cle.validators module

class l5kit.cle.validators.DurationMode(value)

Bases: enum.IntEnum

For more information about duration mode, see RangeValidator docs.

CONTINUOUS = 1
TOTAL = 0
class l5kit.cle.validators.FailedFrame(scene_id: int, frame_index: int)

Bases: tuple

A named-tuple composed of the scene if and the frame index on that scene that caused the validator to fail.

frame_index: int

Alias for field number 1

scene_id: int

Alias for field number 0

class l5kit.cle.validators.RangeValidator(validator_name: str, metric: Type[l5kit.cle.metrics.SupportsMetricCompute], min_value: Optional[float] = None, max_value: Optional[float] = None, violation_duration_s: float = 0.0, duration_mode: l5kit.cle.validators.DurationMode = DurationMode.TOTAL)

Bases: l5kit.cle.validators.SupportsMetricValidate

Validates a metric based on specified range. The range validator will check: min_value < metric_value < max_value. It will also check if the metric violated a maximum duration (see duration_mode parameter notes).

Parameters
  • validator_name – name of the validator

  • metric – metric class used to validate

  • min_value – minimum value allowed (inclusive)

  • max_value – maximum value allowed (inclusive)

  • violation_duration_s – maximum allowed duration in seconds where the metric can be violated

  • duration_mode – the duration mode can be “total” or “continuous”. In “total” mode, all violations are summed per scene, and if they exceed the violation_duration_s parameter, then it will not pass the validation. In “continuous” mode, the violation_duration_s parameter must be exceded on a continuous violation of the metric in time (without a gap of non-violation). This parameter is ignored if violation_duration_s is zero.

static cumsum_with_reset(timestamp_diff: torch.Tensor, validation_mask: torch.Tensor) torch.Tensor

Cumulative sum (cumsum) that takes into consideration a valid mask. If the valid mask is False on a timestamp, it will reset the accumulator.

Parameters
  • timestamp_diff – timestamps differentiated (1D array)

  • validation_mask – a boolean mask with valid/invalid timestamps (1D array)

Returns

cumulative sum (1D array)

requires_metric: List[str]
validate(metric_results: Dict[str, torch.Tensor], simulation_output: l5kit.simulation.unroll.SimulationOutputCLE) l5kit.cle.validators.ValidatorOutput

Apply the validator on the results of the metric computation.

Parameters
  • metric_results – all metric results

  • simulation_output – output from the closed-loop simulator

Returns

True if validator passed, False otherwise

validator_name: str
class l5kit.cle.validators.SupportsMetricValidate(*args, **kwargs)

Bases: Protocol

Protocol supporting the validation for metrics. The evaluation plan has two main components: metrics and validators. Metrics are completely independent, but validators not, as they depend on metrics, therefore the validator needs to carry a list of metrics it requires to compute, otherwise, the evaluation plan is not consistent, and this is checked by the evaluation plan.

requires_metric: List[str]
abstract validate(metric_results: Dict[str, torch.Tensor], simulation_output: l5kit.simulation.unroll.SimulationOutputCLE) l5kit.cle.validators.ValidatorOutput

Apply the validator on the metric results.

Parameters
  • metric_results – results from all computed metrics

  • simulation_output – output from the closed-loop simulator

Returns

True if validator passed, False otherwise

validator_name: str
class l5kit.cle.validators.SupportsValidationAggregation(*args, **kwargs)

Bases: Protocol

Protocol supporting the validation aggregation. This aggregator is responsible for aggregating results from the the validators and also doing a reduction step across multiple distributed nodes.

aggregate(scene_validation_results: Dict[int, Dict[str, l5kit.cle.validators.ValidatorOutput]]) Dict[str, Any]

This method will aggregate scenes locally and then will do the reduction step to aggregate data across distributed nodes.

Parameters

scene_validation_results – results from validator outputs per scene

Returns

any result (it can be a composite object with scenes and frames or just a float value) indexed by the validator name.

class l5kit.cle.validators.ValidationCountingAggregator(failed_frames: bool = False)

Bases: l5kit.cle.validators.SupportsValidationAggregation

This aggregator will count (sum) the amount of invalid scenes or optionally the amount of failed frames on each scene.

Parameters

failed_frames – if True, it will count the number of frames failed instead of scenes failed.

aggregate(scene_validation_results: Dict[int, Dict[str, l5kit.cle.validators.ValidatorOutput]]) Dict[str, Any]

This method will aggregate scenes locally.

Parameters

scene_validation_results – results from validator outputs per scene

Returns

any result (it can be a composite object with scenes and frames or just a float value) indexed by the validator name.

aggregate_scenes(scene_validation_results: Dict[int, Dict[str, l5kit.cle.validators.ValidatorOutput]]) Dict[str, Any]

Aggregate the scenes locally on each node. This method will just sum the number of invalid scenes across all scenes in the node.

Parameters

scene_validation_results – results from validator outputs per scene

Returns

a dictionary with the validation metric name as keys and the sum of invalid scenes

class l5kit.cle.validators.ValidationFailedFramesAggregator

Bases: object

This aggregator will aggregate all failed frames (and scenes).

aggregate(scene_validation_results: Dict[int, Dict[str, l5kit.cle.validators.ValidatorOutput]]) Dict[str, Any]

This method will aggregate scenes locally and then will do the reduction step to aggregate data across distributed nodes.

Parameters

scene_validation_results – results from validator outputs per scene

Returns

any result (it can be a composite object with scenes and frames or just a float value) indexed by the validator name.

aggregate_scenes(scene_validation_results: Dict[int, Dict[str, l5kit.cle.validators.ValidatorOutput]]) Dict[str, Any]

This method will aggregate the failed scene/frame tuples locally at each node and then build a 1D torch tensor with the unpacked tuples such as [scene, frame, scene frame, (…)], to be able to gather them later on the reduction across different nodes.

Parameters

scene_validation_results – results from validator outputs per scene

Returns

a dictionary, indexed by the validator name, with FailedFrame list containing the scene/frames failed.

class l5kit.cle.validators.ValidatorOutput(is_valid_scene: bool, failed_frames: List[int])

Bases: tuple

Output from validators. Validators should return a boolean telling if the scene is valid or not and a list of failed frames.

failed_frames: List[int]

Alias for field number 1

is_valid_scene: bool

Alias for field number 0