l5kit.data package

class l5kit.data.ChunkedDataset(path: str, key: str = '')

Bases: object

ChunkedDataset is a dataset that lives on disk in compressed chunks, it has easy to use data loading and writing interfaces that involves making numpy-like slices. Currently only .zarr directory stores are supported (i.e. the data will live in a folder on your local filesystem called <something>.zarr).

get_scene_dataset(scene_index: int) → l5kit.data.zarr_dataset.ChunkedDataset

Get a new ChunkedDataset of a single scene. This dataset lives in memory (as np.ndarray)

Parameters

scene_index – the scene index

Returns

a dataset with a single scene inside

initialize(mode: str = 'w', num_scenes: int = 0, num_frames: int = 0, num_agents: int = 0, num_tl_faces: int = 0) → l5kit.data.zarr_dataset.ChunkedDataset

Initializes a new zarr dataset, creating the underlying arrays.

Parameters

• mode – Mode to open dataset in, should be something that supports writing. (default: {“w”})

• num_scenes – pre-allocate this number of scenes

• num_frames – pre-allocate this number of frames

• num_agents – pre-allocate this number of agents

• num_tl_faces – pre-allocate this number of traffic lights

open(mode: str = 'r', cached: bool = True, cache_size_bytes: int = 1000000000) → l5kit.data.zarr_dataset.ChunkedDataset

Opens a zarr dataset from disk from the path supplied in the constructor.

Parameters

• mode – Mode to open dataset in, default to read-only (default: {“r”})

• cached – Whether to cache files read from disk using a LRU cache. (default: {True})

• cache_size_bytes – Size of cache in bytes (default: {1e9} (1GB))

class l5kit.data.DataManager

Bases: abc.ABC

abstract require(key: str) → str

class l5kit.data.LocalDataManager(local_data_folder: Optional[Union[str, pathlib.Path]] = None)

Bases: l5kit.data.local_data_manager.DataManager

LocalDataManager allows you to require data to be present in the subpath of a specific folder.

Example: Your data folder is set to "/tmp/my-data-folder", and you call local_data_manager.require("path/to/image.jpg"), it would check if "/tmp/my-data-folder/path/to/image.jpg" exists, and if so return that complete path ("/tmp/my-data-folder/path/to/image.jpg"), otherwise it raises an error.

In order of precedence, the local data folder is set by

1. Passing in the path to the constructor of LocalDataManager

2. Setting the L5KIT_DATA_FOLDER environment variable.

require(key: str) → str

Require checks whether the file with the given key is present in the local data folder, if it is not it raises an error. Returns the path to the file otherwise.

Parameters

key (str) – Path from the data folder where the file or folder should be present.

Returns

str – Filepath including the data folder where required key is present.

class l5kit.data.MapAPI(protobuf_map_path: str, world_to_ecef: numpy.ndarray)

Bases: object

static from_cfg(data_manager: l5kit.data.local_data_manager.DataManager, cfg: dict) → l5kit.data.map_api.MapAPI

Build a MapAPI object starting from a config file and a data manager

Parameters

• data_manager – a data manager object ot resolve paths

• cfg – the config dict

Returns

a MapAPI object

get_bounds() → dict

For each elements of interest returns bounds [[min_x, min_y],[max_x, max_y]] and proto ids Coords are computed by the MapAPI and, as such, are in the world ref system.

Returns

keys are classes of elements, values are dict with bounds and ids keys

Return type

dict

get_color_for_face(face_id: str) → str

Utility function. It calls is_traffic_face_color for a set of colors until it gets an answer. If no color is found, then face_id is not the id of a traffic light face (and we raise ValueError).

Parameters

face_id (str) – the element id

Returns

the color as string for this traffic face

Return type

str

get_crosswalk_coords(element_id: str) → dict

Get XYZ coordinates in world ref system for a crosswalk given its id lru_cached for O(1) access

Parameters

element_id (str) – crosswalk element id

Returns

a dict with the polygon coordinates as an (Nx3) XYZ array

Return type

dict

get_lane_as_interpolation(element_id: str, step: float, method: l5kit.data.map_api.InterpolationMethod) → dict

Perform an interpolation of the left and right lanes and compute the midlane. See interpolate for details about the different interpolation methods

Parameters

• element_id (str) – lane id

• step (float) – step param for the method

• method (InterpolationMethod) – one of the accepted methods

Returns

same as get_lane_coords but overwrite xyz values for the lanes

Return type

dict

get_lane_coords(element_id: str) → dict

Get XYZ coordinates in world ref system for a lane given its id lru_cached for O(1) access

Parameters

element_id (str) – lane element id

Returns

a dict with the two boundaries coordinates as (Nx3) XYZ arrays

Return type

dict

get_lane_traffic_control_ids(element_id: str) → set

get_tl_feature_for_lane(lane_id: str, active_tl_face_to_color: dict) → int

Get traffic light feature for a lane given its active tl faces and a constant priority map.

static id_as_str(element_id: road_network_pb2.GlobalId) → str

Get the element id as a string. Elements ids are stored as a variable len sequence of bytes in the protobuf

Parameters

element_id (GlobalId) – the GlobalId in the protobuf

Returns

the id as a str

Return type

str

static interpolate(xyz: numpy.ndarray, step: float, method: l5kit.data.map_api.InterpolationMethod) → numpy.ndarray

Interpolate points based on cumulative distances from the first one. Two modes are available: INTER_METER: interpolate using step as a meter value over cumulative distances (variable len result) INTER_ENSURE_LEN: interpolate using a variable step such that we always get step values :param xyz: XYZ coords :type xyz: np.ndarray :param step: param for the interpolation :type step: float :param method: method to use to interpolate :type method: InterpolationMethod

Returns

the new interpolated coordinates

Return type

np.ndarray

static is_crosswalk(element: road_network_pb2.MapElement) → bool

Check whether an element is a valid crosswalk

Parameters

element (MapElement) – a proto element

Returns

True if the element is a valid crosswalk

Return type

bool

static is_lane(element: road_network_pb2.MapElement) → bool

Check whether an element is a valid lane

Parameters

element (MapElement) – a proto element

Returns

True if the element is a valid lane

Return type

bool

is_traffic_face(element_id: str) → bool

Check if the element is a traffic light face (of any color)

Parameters

element_id (str) – the id (utf-8 encode) of the element

Returns

True if the element is a traffic light face, False otherwise

is_traffic_face_color(element_id: str, color: str) → bool

Check if the element is a traffic light face of the given color

Parameters

• element_id (str) – the id (utf-8 encode) of the element

• color (str) – the color to check

Returns

True if the element is a traffic light face with the given color

is_traffic_light(element_id: str) → bool

Check if the element is a traffic light :param element_id: the id (utf-8 encode) of the element :type element_id: str

Returns

True if the element is a traffic light

unpack_deltas_cm(dx: Sequence[int], dy: Sequence[int], dz: Sequence[int], frame: road_network_pb2.GeoFrame) → numpy.ndarray

Get coords in world reference system (local ENU->ECEF->world). See the protobuf annotations for additional information about how coordinates are stored

Parameters

• dx (Sequence[int]) – X displacement in centimeters in local ENU

• dy (Sequence[int]) – Y displacement in centimeters in local ENU

• dz (Sequence[int]) – Z displacement in centimeters in local ENU

• frame (GeoFrame) – geo-location information for the local ENU. It contains lat and long origin of the frame

Returns

array of shape (Nx3) with XYZ coordinates in world ref system

Return type

np.ndarray

l5kit.data.filter_agents_by_distance(agents: numpy.ndarray, centroid: numpy.ndarray, max_distance: float) → numpy.ndarray

Filter agents by distance, cut to max_distance and sort the result :param agents: array of agents :type agents: np.ndarray :param centroid: centroid towards which compute distance :type centroid: np.ndarray :param max_distance: max distance to cut off :type max_distance: float

Returns

agents sorted and cut to max_distance

Return type

np.ndarray

l5kit.data.filter_agents_by_frames(frames: numpy.ndarray, agents: numpy.ndarray) → List[numpy.ndarray]

Get a list of agents array, one array per frame. Note that “agent_index_interval” is used to filter agents, so you should take care of re-setting it if you have previously sliced agents.

Parameters

• frames (np.ndarray) – an array of frames

• agents (np.ndarray) – an array of agents

Returns

List[np.ndarray] with the agents divided by frame

l5kit.data.filter_agents_by_labels(agents: numpy.ndarray, threshold: float = 0.5) → numpy.ndarray

Filters an agents array, keeping those agents that meet the threshold.

Parameters

agents (np.ndarray) – Agents array

Keyword Arguments

threshold (float) – probability threshold for filtering (default: {0.5})

Returns

np.ndarray – A subset of input agents array.

l5kit.data.filter_agents_by_track_id(agents: numpy.ndarray, track_id: int) → numpy.ndarray

Return all agent object (np.ndarray) of a given track_id.

Parameters

• agents (np.ndarray) – agents array. NOTE: do NOT pass a zarr to this function, it can’t handle boolean indexing

• track_id (int) – agent track id to select

Returns

np.ndarray – Selected agent.

l5kit.data.filter_tl_faces_by_frames(frames: numpy.ndarray, tl_faces: numpy.ndarray) → List[numpy.ndarray]

Get a list of traffic light faces arrays, one array per frame. This functions mimics filter_agents_by_frames for traffic light faces

Parameters

• frames (np.ndarray) – an array of frames

• tl_faces (np.ndarray) – an array of traffic light faces

Returns

List[np.ndarray] with the traffic light faces divided by frame

l5kit.data.filter_tl_faces_by_status(tl_faces: numpy.ndarray, status: str) → numpy.ndarray

Filter tl_faces and keep only active faces :param tl_faces: array of traffic faces :type tl_faces: np.ndarray :param status: status we want to keep TODO refactor for enum :type status: str

Returns

traffic light faces array with only faces with that status

Return type

np.ndarray

l5kit.data.get_agents_slice_from_frames(frame_a: numpy.ndarray, frame_b: Optional[numpy.ndarray] = None) → slice

Get a slice for indexing agents giving a start and end frame

Parameters

• frame_a (np.ndarray) – the starting frame

• frame_b (Optional[np.ndarray]) – the ending frame. If None, then frame_a end will be used

Returns

a slice object starting from the first agent in frame_a to the last one in frame_b

Return type

slice

l5kit.data.get_combined_scenes(scenes: numpy.ndarray) → numpy.ndarray

Takes as input an np.ndarray or zarr array with scenes, and combines scenes that follow up eachother perfectly (i.e. from a single recording by the same host). Returns an np.ndarray of combined scenes.

Arguments:

scenes (np.ndarray): scenes

Returns:

np.ndarray – combined scenes where followup scenes have been merged.

l5kit.data.get_frames_slice_from_scenes(scene_a: numpy.ndarray, scene_b: Optional[numpy.ndarray] = None) → slice

Get a slice for indexing frames giving a start and end scene

Parameters

• scene_a (np.ndarray) – the starting scene

• scene_b (Optional[np.ndarray]) – the ending scene. If None, then scene_a end will be used

Returns

a slice object starting from the first frame in scene_a to the last one in scene_b

Return type

slice

l5kit.data.get_tl_faces_slice_from_frames(frame_a: numpy.ndarray, frame_b: Optional[numpy.ndarray] = None) → slice

Get a slice for indexing traffic light faces giving a start and end frame

Parameters

• frame_a (np.ndarray) – the starting frame

• frame_b (Optional[np.ndarray]) – the ending frame. If None, then frame_a end will be used

Returns

a slice object starting from the first tl_face in frame_a to the last one in frame_b

Return type

slice

l5kit.data.zarr_concat(input_zarrs: List[str], output_zarr: str) → None

Concat many zarr into a single one. Takes care of updating indices for frames and agents.

Parameters

• input_zarrs (List[str]) – a list of paths to input zarrs

• output_zarr (str) – the path to the output zarr

Returns:

Subpackages

• l5kit.data.proto package
  • Submodules
    • l5kit.data.proto.road_network_pb2 module

Submodules

• l5kit.data.combine module
• l5kit.data.filter module
• l5kit.data.labels module
• l5kit.data.local_data_manager module
• l5kit.data.map_api module
• l5kit.data.zarr_dataset module
• l5kit.data.zarr_utils module