l5kit.cle.validators module¶

class l5kit.cle.validators.DurationMode(value)¶

Bases: enum.IntEnum

For more information about duration mode, see RangeValidator docs.

CONTINUOUS = 1¶

TOTAL = 0¶

class l5kit.cle.validators.FailedFrame(scene_id: int, frame_index: int)¶

Bases: tuple

A named-tuple composed of the scene if and the frame index on that scene that caused the validator to fail.

frame_index: int¶: Alias for field number 1

scene_id: int¶: Alias for field number 0

class l5kit.cle.validators.RangeValidator(validator_name: str, metric: Type[l5kit.cle.metrics.SupportsMetricCompute], min_value: Optional[float] = None, max_value: Optional[float] = None, violation_duration_s: float = 0.0, duration_mode: l5kit.cle.validators.DurationMode = DurationMode.TOTAL)¶

Bases: l5kit.cle.validators.SupportsMetricValidate

Validates a metric based on specified range. The range validator will check: min_value < metric_value < max_value. It will also check if the metric violated a maximum duration (see duration_mode parameter notes).

Parameters

validator_name – name of the validator
metric – metric class used to validate
min_value – minimum value allowed (inclusive)
max_value – maximum value allowed (inclusive)
violation_duration_s – maximum allowed duration in seconds where the metric can be violated
duration_mode – the duration mode can be “total” or “continuous”. In “total” mode, all violations are summed per scene, and if they exceed the violation_duration_s parameter, then it will not pass the validation. In “continuous” mode, the violation_duration_s parameter must be exceded on a continuous violation of the metric in time (without a gap of non-violation). This parameter is ignored if violation_duration_s is zero.

static cumsum_with_reset(timestamp_diff: torch.Tensor, validation_mask: torch.Tensor) → torch.Tensor¶

Cumulative sum (cumsum) that takes into consideration a valid mask. If the valid mask is False on a timestamp, it will reset the accumulator.

Parameters

timestamp_diff – timestamps differentiated (1D array)
validation_mask – a boolean mask with valid/invalid timestamps (1D array)

Returns

cumulative sum (1D array)

requires_metric: List[str]¶

validate(metric_results: Dict[str, torch.Tensor], simulation_output: l5kit.simulation.unroll.SimulationOutputCLE) → l5kit.cle.validators.ValidatorOutput¶

Apply the validator on the results of the metric computation.

Parameters

metric_results – all metric results
simulation_output – output from the closed-loop simulator

Returns

True if validator passed, False otherwise

validator_name: str¶

class l5kit.cle.validators.SupportsMetricValidate(*args, **kwargs)¶

Bases: Protocol

Protocol supporting the validation for metrics. The evaluation plan has two main components: metrics and validators. Metrics are completely independent, but validators not, as they depend on metrics, therefore the validator needs to carry a list of metrics it requires to compute, otherwise, the evaluation plan is not consistent, and this is checked by the evaluation plan.

requires_metric: List[str]¶

abstract validate(metric_results: Dict[str, torch.Tensor], simulation_output: l5kit.simulation.unroll.SimulationOutputCLE) → l5kit.cle.validators.ValidatorOutput¶

Apply the validator on the metric results.

Parameters

metric_results – results from all computed metrics
simulation_output – output from the closed-loop simulator

Returns

True if validator passed, False otherwise

validator_name: str¶

class l5kit.cle.validators.SupportsValidationAggregation(*args, **kwargs)¶

Bases: Protocol

Protocol supporting the validation aggregation. This aggregator is responsible for aggregating results from the the validators and also doing a reduction step across multiple distributed nodes.

aggregate(scene_validation_results: Dict[int, Dict[str, l5kit.cle.validators.ValidatorOutput]]) → Dict[str, Any]¶

This method will aggregate scenes locally and then will do the reduction step to aggregate data across distributed nodes.

Parameters: scene_validation_results – results from validator outputs per scene
Returns: any result (it can be a composite object with scenes and frames or just a float value) indexed by the validator name.

class l5kit.cle.validators.ValidationCountingAggregator(failed_frames: bool = False)¶

Bases: l5kit.cle.validators.SupportsValidationAggregation

This aggregator will count (sum) the amount of invalid scenes or optionally the amount of failed frames on each scene.

Parameters: failed_frames – if True, it will count the number of frames failed instead of scenes failed.

aggregate(scene_validation_results: Dict[int, Dict[str, l5kit.cle.validators.ValidatorOutput]]) → Dict[str, Any]¶

This method will aggregate scenes locally.

Parameters: scene_validation_results – results from validator outputs per scene
Returns: any result (it can be a composite object with scenes and frames or just a float value) indexed by the validator name.

aggregate_scenes(scene_validation_results: Dict[int, Dict[str, l5kit.cle.validators.ValidatorOutput]]) → Dict[str, Any]¶

Aggregate the scenes locally on each node. This method will just sum the number of invalid scenes across all scenes in the node.

Parameters: scene_validation_results – results from validator outputs per scene
Returns: a dictionary with the validation metric name as keys and the sum of invalid scenes

class l5kit.cle.validators.ValidationFailedFramesAggregator¶

Bases: object

This aggregator will aggregate all failed frames (and scenes).

aggregate(scene_validation_results: Dict[int, Dict[str, l5kit.cle.validators.ValidatorOutput]]) → Dict[str, Any]¶

This method will aggregate scenes locally and then will do the reduction step to aggregate data across distributed nodes.

Parameters: scene_validation_results – results from validator outputs per scene
Returns: any result (it can be a composite object with scenes and frames or just a float value) indexed by the validator name.

aggregate_scenes(scene_validation_results: Dict[int, Dict[str, l5kit.cle.validators.ValidatorOutput]]) → Dict[str, Any]¶

This method will aggregate the failed scene/frame tuples locally at each node and then build a 1D torch tensor with the unpacked tuples such as [scene, frame, scene frame, (…)], to be able to gather them later on the reduction across different nodes.

Parameters: scene_validation_results – results from validator outputs per scene
Returns: a dictionary, indexed by the validator name, with FailedFrame list containing the scene/frames failed.

class l5kit.cle.validators.ValidatorOutput(is_valid_scene: bool, failed_frames: List[int])¶

Bases: tuple

Output from validators. Validators should return a boolean telling if the scene is valid or not and a list of failed frames.

failed_frames: List[int]¶: Alias for field number 1

is_valid_scene: bool¶: Alias for field number 0