l5kit.evaluation package

l5kit.evaluation.compute_metrics_csv(ground_truth_path: str, inference_output_path: str, metrics: List[Callable[[numpy.ndarray, numpy.ndarray, numpy.ndarray, numpy.ndarray], numpy.ndarray]], max_modes: int = 3) → dict

Compute a set of metrics between ground truth and prediction csv files

Parameters

  • ground_truth_path (str) – Path to the ground truth csv file.

  • inference_output_path (str) – Path to the csv file containing network output.

  • metrics (List[Callable]) – a list of callable to be applied to the elements retrieved from the 2

  • files (csv) –

  • max_modes (int) – maximum number of predicted modes

Returns

keys are metrics name, values is the average metric computed over the elements

Return type

dict

l5kit.evaluation.create_chopped_dataset(zarr_path: str, th_agent_prob: float, num_frames_to_copy: int, num_frames_gt: int, min_frame_future: int) → str

Create a chopped version of the zarr that can be used as a test set. This function was used to generate the test set for the competition so that the future GT is not in the data.

Store:

  • a dataset where each scene has been chopped at num_frames_to_copy frames;

  • a mask for agents for those final frames based on the original mask and a threshold on the future_frames;

  • the GT csv for those agents

For the competition, only the first two (dataset and mask) will be available in the notebooks

Parameters

  • zarr_path (str) – input zarr path to be chopped

  • th_agent_prob (float) – threshold over agents probabilities used in select_agents function

  • num_frames_to_copy (int) – number of frames to copy from the beginning of each scene, others will be discarded

  • min_frame_future (int) – minimum number of frames that must be available in the future for an agent

  • num_frames_gt (int) – number of future predictions to store in the GT file

Returns

the parent folder of the new datam

Return type

str

l5kit.evaluation.export_zarr_to_csv(zarr_dataset: l5kit.data.zarr_dataset.ChunkedDataset, csv_file_path: str, future_num_frames: int, filter_agents_threshold: float, step_time: float = 0.1, agents_mask: Optional[numpy.array] = None) → None

Produces a csv file containing the ground truth from a zarr file.

Parameters

  • zarr_dataset (np.ndarray) – The open zarr dataset.

  • csv_file_path (str) – File path to write a CSV to.

  • future_num_frames (int) – Amount of future displacements we want.

  • filter_agents_threshold (float) – Value between 0 and 1 to use as cutoff value for agent filtering

  • agents_mask (instead of computing the) – a boolean mask of shape (len(zarr_dataset.agents)) which will be used

  • agents_mask –

l5kit.evaluation.read_gt_csv(csv_path: str) → Iterator[dict]

Generator function that returns a line at a time from the csv file as a dict

Parameters

csv_path (str) – path of the csv to read

Returns

dict keys are the csv header fieldnames

Return type

Iterator[dict]

l5kit.evaluation.read_pred_csv(csv_path: str, max_modes: int = 3) → Iterator[dict]

Generator function that returns a line at the time from the csv file as a dict

Parameters

  • csv_path (str) – path of the csv to read

  • max_modes (int) – maximum number of predicted modes

Returns

dict keys are the csv header fieldnames

Return type

Iterator[dict]

l5kit.evaluation.write_gt_csv(csv_path: str, timestamps: numpy.ndarray, track_ids: numpy.ndarray, coords: numpy.ndarray, avails: numpy.ndarray) → None

Encode the ground truth into a csv file

Parameters

  • csv_path (str) – path to the csv to write

  • timestamps (np.ndarray) – (num_example,) frame timestamps

  • track_ids (np.ndarray) – (num_example,) agent ids

  • coords (np.ndarray) – (num_example x future_len x num_coords) meters displacements

  • avails (np.ndarray) – (num_example x future_len) array with value 0 (discard in eval) or 1 (keep in eval)

Returns:

l5kit.evaluation.write_pred_csv(csv_path: str, timestamps: numpy.ndarray, track_ids: numpy.ndarray, coords: numpy.ndarray, confs: Optional[numpy.ndarray] = None, max_modes: int = 3) → None

Encode the predictions into a csv file. Coords can have an additional axis for multi-mode. We handle up to max_modes modes. For the uni-modal case (i.e. all predictions have just a single mode), coords should not have the additional axis and confs should be set to None. In this case, a single mode with confidence 1 will be written.

Parameters

  • csv_path (str) – path to the csv to write

  • timestamps (np.ndarray) – (num_example,) frame timestamps

  • track_ids (np.ndarray) – (num_example,) agent ids

  • coords (np.ndarray) – (num_example x (modes) x future_len x num_coords) meters displacements

  • confs (Optional[np.ndarray]) – (num_example x modes) confidence of each modes in each example.

  • 1 (Rows should sum to) –

  • max_modes (int) – maximum number of predicted modes

Returns:

Submodules