l5kit.prediction.vectorized.safepathnet module

class l5kit.prediction.vectorized.safepathnet.MultimodalDecoder(dim_in: int = 128, projection_dim: int = 256, nhead: int = 8, dim_feedforward: int = 1024, num_layers: int = 3, num_decode_layers: int = 3, dropout: float = 0.1, future_num_frames: int = 45, agent_future_num_frames: int = 30, num_outputs: int = 3, num_mlp_layers: int = 3, num_trajectories: int = 10, agent_num_trajectories: int = 5, max_num_agents: int = 30)

Bases: torch.nn.modules.module.Module

The multimodal global graph model, based on the transformer encoder-decoder model.

forward(polys_embed: torch.Tensor, types: torch.Tensor, invalid_mask: torch.Tensor) → Tuple[torch.Tensor, torch.Tensor]

Forward of the model. Returns: - the agent predictions, shape:

[batch_size, num_agents, agent_num_trajectories, agent_num_timesteps, num_outputs]

• the associated trajectory probabilities, shape:

  [batch_size, num_agents, agent_num_trajectories]

Parameters

• polys_embed – input tensor representing the polylines embeddings shape: [batch_size, num_polylines, dim_in] (float)

• types – float tensor representing the polylines type embedding shape: [batch_size, num_polylines, dim_in] (float)

• invalid_mask – bool tensor representing the mask of polylines with no valid information shape: [batch_size, num_polylines] (bool)

training: bool

class l5kit.prediction.vectorized.safepathnet.MultimodalTransformerGlobalGraph(transformer_global_graph: l5kit.prediction.vectorized.safepathnet.TransformerGlobalGraph, num_trajectories: int, agent_num_trajectories: int = 1, max_num_agents: int = 0)

Bases: torch.nn.modules.module.Module

This class extends the transformer global graph to the multimodal case (prediction of multiple trajectories). Each agent embedding is repeated (agent_)num_trajectories times and a set of (agent_)num_trajectories learned embeddings is added to the cloned ego features to obtain diverse queries.

forward(polys_embed: torch.Tensor, invalid_mask: torch.Tensor, types: torch.Tensor) → torch.Tensor

Clones the agent features (self.num_trajectories - 1) times before calling the parent forward method.

The model returns agent embeddings with shape [batch_size, num_agents, dim_in]

Parameters

• polys_embed – input tensor representing the polylines embeddings shape: [batch_size, num_polylines, dim_in] (float)

• invalid_mask – bool tensor representing the mask of polylines with no valid information shape: [batch_size, num_polylines] (bool)

• types – float tensor representing the polylines type embedding shape: [batch_size, num_polylines, dim_in] (float)

repeat_agent_features(features: torch.Tensor) → torch.Tensor

Repeat agents features self.num_trajectories times

repeat_ego_features(features: torch.Tensor) → torch.Tensor

Repeat ego features self.num_trajectories times

repeat_features(features: torch.Tensor, num_repeats: int) → torch.Tensor

Repeat input features num_repeats times on dimension 1 in a deterministic way

(avoiding the usage of repeat_interleave).

Parameters

• features – Input features to be repeated

• num_repeats – Number of times features will be repeated on dimension 1

training: bool

class l5kit.prediction.vectorized.safepathnet.TrajectoryMatcher(cost_prob_coeff: float = 1)

Bases: torch.nn.modules.module.Module

Picks the closest trajectory based on provided costs per trajectory and trajectory logits.

forward(trajectory_costs: torch.Tensor, trajectory_logits: torch.Tensor) → torch.Tensor

Returns the indices corresponding to the lowest-cost trajectories

Parameters

• trajectory_costs – cost of each trajectory (e.g. loss per traj.). Shape [batch_size, num_trajectories]

• trajectory_logits – logits of the trajectory distribution. Shape [batch_size, num_trajectories]

Returns

indices of the trajectories with the lowest cost. Shape [batch_size]

training: bool

class l5kit.prediction.vectorized.safepathnet.TransformerGlobalGraph(dim_in: int = 128, projection_dim: int = 256, nhead: int = 8, dim_feedforward: int = 1024, num_layers: int = 3, num_decode_layers: int = 3, dropout: float = 0.1)

Bases: torch.nn.modules.module.Module

This transformer encoder-decoder module implements self attention across polyline embeddings. This represents a global graph that generates attention between each polyline towards other polyline. Then, the decoder queries the global graph features with the given queries.

forward(polys_embed: torch.Tensor, polys_embed_decoder: torch.Tensor, invalid_mask: torch.Tensor, tgt_invalid_mask: Optional[torch.Tensor], types: torch.Tensor) → torch.Tensor

Forward pass for the transformer model based on polyline and type embeddings, the polyline invalid

mask, the polyline decoder queries. The attention keys are the sum of polyline and type embedding.

Parameters

• polys_embed – input tensor representing the polylines embeddings shape: [batch_size, num_polylines, dim_in] (float)

• polys_embed_decoder – input tensor representing the transformer decoder queries shape: [batch_size, num_polylines, dim_in] (float)

• invalid_mask – bool tensor representing the mask of polylines with no valid information shape: [batch_size, num_polylines] (bool)

• tgt_invalid_mask – bool tensor representing the mask of decoded polylines with no valid information shape: [batch_size, num_output_polylines] (bool)

• types – float tensor representing the polyline type embeddings shape: [batch_size, num_polylines] (float)

training: bool