Scenario Description
There exists a lot real-world autonomous driving datasets, such as Waymo and nuPlan. Those datasets contain rich information about road map, traffic lights, traffic participants and their motions. Those data can be imported to the MetaDrive so that we can reconstruct an interactive environments and replay (or even alter!) those scenarios recorded in the real world.
To achieve that, we have a separated project called ScenarioNet (https://metadriverse.github.io/scenarionet/) that converts various sources of AD data into a unified data structure, manage those data, and import those data into MetaDrive for creating interactive environments.
As you can see in the next figure of ScenarioNet, the key functionalities developed in ScenarioNet are:
Unified Scenario Description
Scenario Data Management
Real-world Scenario Simulation
MetaDrive implements the unified scenario description in metadrive/scenario/scenario_description.py and ScenarioNet implements the data management toolkits.
For completeness, we will walk through both the ScenarioNet dataset, which contains a lots of scenarios, and the Scenario Description, a nested dict that stores the information of one scenario.
What is the ScenarioNet dataset?
TL;DR: A established ScenarioNet dataset is a folder containing dataset_summary.pkl and dataset_mapping.pkl (optionally), along with a set of scenario_version_id.pkl files each of which stores the unified scenario description of a scenario.
To be specific, the functions of these files are as follows.
dataset_summary.pklsummarizes the meta information for each scenario. Like the number of objects and moving distance of each object.dataset_mapping.pklcontains the mapping from the scenario ID to its relative path on your machine. When this file is missing, the relative path to look up scenarios will be set to the directory ofdataset_summary.pkl.scenario_version_id.pklstores the serialized scenario description, i.e. a python dict, of a scenario.
Note: All these files are human readable after beding loadded and decoded by pickle
Stored detailed scenario information, scenario_version_id.pkl files are stored either in the same folder of dataset_summary.pkl or in some other folders with the relative path to dataset_summary.pkl registered in dataset_mapping.pkl. The function ofdataset_mapping.pkl is like a soft link or a pointer. Thus, to build new datasets, we only combine a set of relative paths pointing to target scenarios, without the need to copy scenario_version_id.pkl scenario files. In the following content, we call scenario files .pkl file for convenience.
In this section, we demonstrate how to use the utilities from MetaDrive to easily access scenarios. We prepare two demo datasets, splitting from Waymo Open Dataset and nuScenes Dataset. Here we will demonstrate how to load the files. You can also find some useful tutorials in ScenarioNet repo: https://github.com/metadriverse/scenarionet/tree/main/tutorial
from metadrive.engine.asset_loader import AssetLoader
import os
waymo_data = AssetLoader.file_path(AssetLoader.asset_path, "waymo", unix_style=False) # Use the built-in datasets with simulator
os.listdir(waymo_data) # there are 3 waymo scenario files with one 'dataset_summary.pkl' in this example dataset.
['sd_training.tfrecord-00000-of-01000_8a346109094cd5aa.pkl',
'sd_training.tfrecord-00000-of-01000_2a1e44d405a6833f.pkl',
'dataset_summary.pkl',
'sd_training.tfrecord-00000-of-01000_c403d5992cab9e0.pkl']
nuscenes_data = AssetLoader.file_path(AssetLoader.asset_path, "nuscenes", unix_style=False) # Use the built-in datasets with simulator
os.listdir(nuscenes_data) # there are 10 nuscenes scenario file with a 'dataset_summary.pkl' and a 'dataset_summary.pkl'
['nuscenes_6',
'nuscenes_7',
'nuscenes_2',
'nuscenes_5',
'dataset_mapping.pkl',
'nuscenes_0',
'dataset_summary.pkl',
'nuscenes_1',
'nuscenes_3',
'nuscenes_4']
You can find that for the Waymo dataset, as we put all scenarios in the same folder, we don’t need the dataset_mapping.pkl that routes scenario ID to corresponding .pkl file. In the nuScenes dataset, we have both dataset_mapping.pkl and dataset_summary.pkl because we have a hierarchy file structure to store the scenarios.
How to read a ScenarioNet dataset?
Read the dataset summary
The first step to read a ScenarioNet dataset is to call the utility function read_dataset_summary from MetaDrive.
This function will not read all the data and exhausts your memory as it only read the dataset_summary.pkl and optionally the dataset_mapping.pkl. The dataset_summary contains the summaries (a set of stats) of each scenario in the dataset and the dataset_mapping contains the relative paths of the .pkl files of the scenarios relative to the folder that hosts dataset_summary.pkl file.
The output of the read_dataset_summary is a tuple of three elements:
the summary dict mapping from scenario ID to its metadata,
the list of all scenarios IDs, and
a dict mapping from scenario IDs to the folder that hosts their files.
from metadrive.scenario import utils as sd_utils
dataset_path = waymo_data
print(f"Reading the summary file from Waymo data at: {dataset_path}")
waymo_dataset_summary = sd_utils.read_dataset_summary(dataset_path)
print(f"The dataset summary is a {type(waymo_dataset_summary)}, with lengths {len(waymo_dataset_summary)}.")
Reading the summary file from Waymo data at: /home/zhenghao/metadrive/metadrive/assets/waymo
The dataset summary is a <class 'tuple'>, with lengths 3.
waymo_scenario_summary, waymo_scenario_ids, waymo_scenario_files = waymo_dataset_summary
print(f"The scenario summary is a dict with keys: {waymo_scenario_summary.keys()} \nwhere each value of the dict is the summary of a scenario.\n")
The scenario summary is a dict with keys: dict_keys(['sd_training.tfrecord-00000-of-01000_2a1e44d405a6833f.pkl', 'sd_training.tfrecord-00000-of-01000_c403d5992cab9e0.pkl', 'sd_training.tfrecord-00000-of-01000_8a346109094cd5aa.pkl'])
where each value of the dict is the summary of a scenario.
What is the summary dict of a scenario?
As you can see, the first return value from the utils will return a dict waymo_scenario_summary summarizing all scenarios in the dataset.
Each item in this dict represents the summary of one scenario.
Now let’s dive into the detailed structure and see what is included in the summary of one scenario.
example_scenario_summary = waymo_scenario_summary['sd_training.tfrecord-00000-of-01000_2a1e44d405a6833f.pkl']
print(f"The summary of a scenario is a dict with keys: {example_scenario_summary.keys()}")
The summary of a scenario is a dict with keys: dict_keys(['coordinate', 'ts', 'metadrive_processed', 'sdc_id', 'dataset', 'scenario_id', 'source_file', 'track_length', 'current_time_index', 'sdc_track_index', 'objects_of_interest', 'tracks_to_predict', 'object_summary', 'number_summary'])
# A string indicating the coordinate system of the scenario.
print(example_scenario_summary["coordinate"])
waymo
# A list containing the wall time of each time step
print(example_scenario_summary["ts"])
[0. 0.09998 0.19996 0.29995 0.39994 0.49991 0.59994 0.69993 0.79995
0.89997 0.99999 1.09998 1.2 1.30002 1.40001 1.5 1.59998 1.69997
1.79994 1.89994 1.99996 2.09995 2.19998 2.3 2.40003 2.50002 2.60003
2.70002 2.80004 2.90004 3.00001 3.09995 3.19989 3.2998 3.3997 3.49961
3.59951 3.69942 3.79937 3.89928 3.99918 4.09914 4.19905 4.29897 4.39888
4.49875 4.59866 4.69856 4.79844 4.89836 4.99823 5.09814 5.19802 5.2979
5.3978 5.49771 5.59762 5.69753 5.79748 5.89743 5.99738 6.09736 6.19736
6.29735 6.39737 6.4974 6.59746 6.69749 6.79759 6.89762 6.99768 7.09773
7.1978 7.2979 7.39799 7.49809 7.59822 7.69832 7.79839 7.89849 7.99855
8.09861 8.19868 8.29874 8.39879 8.49882 8.59888 8.6989 8.79893 8.89896
8.99898]
# A boolean indicator saying whether the scenario is exported from MetaDrive.
# Note: we have a replay and record systems in MetaDrive that will generate
# scenarios with `metadrive_processed=True`.
print(example_scenario_summary["metadrive_processed"])
False
# The object ID of the self-driving car (if exists)
print(example_scenario_summary["sdc_id"])
1629
# Where the scenario comes from
print(example_scenario_summary["dataset"])
waymo
# The ID of the scenario
print(example_scenario_summary["scenario_id"])
2a1e44d405a6833f
# The number of each type of objects. This field is very useful when filtering the datasets.
print(example_scenario_summary["number_summary"])
{'num_objects': 211, 'object_types': {'CYCLIST', 'PEDESTRIAN', 'VEHICLE'}, 'num_objects_each_type': {'VEHICLE': 184, 'PEDESTRIAN': 25, 'CYCLIST': 2}, 'num_moving_objects': 69, 'num_moving_objects_each_type': defaultdict(<class 'int'>, {'VEHICLE': 52, 'PEDESTRIAN': 15, 'CYCLIST': 2}), 'num_traffic_lights': 8, 'num_traffic_light_types': {'LANE_STATE_STOP', 'LANE_STATE_UNKNOWN'}, 'num_traffic_light_each_step': {'LANE_STATE_UNKNOWN': 164, 'LANE_STATE_STOP': 564}, 'num_map_features': 358, 'map_height_diff': 2.4652252197265625}
# The summary for each object, including information like the moving distance, type, valid time steps.
print(example_scenario_summary["object_summary"])
{'325': {'type': 'VEHICLE', 'object_id': '325', 'track_length': 91, 'moving_distance': 2.7487233877182007, 'valid_length': 3, 'continuous_valid_length': 2}, '327': {'type': 'VEHICLE', 'object_id': '327', 'track_length': 91, 'moving_distance': 5.598679721355438, 'valid_length': 15, 'continuous_valid_length': 15}, '332': {'type': 'VEHICLE', 'object_id': '332', 'track_length': 91, 'moving_distance': 103.90164065361023, 'valid_length': 91, 'continuous_valid_length': 91}, '333': {'type': 'VEHICLE', 'object_id': '333', 'track_length': 91, 'moving_distance': 111.29066771268845, 'valid_length': 91, 'continuous_valid_length': 91}, '336': {'type': 'VEHICLE', 'object_id': '336', 'track_length': 91, 'moving_distance': 5.664326503872871, 'valid_length': 23, 'continuous_valid_length': 21}, '337': {'type': 'VEHICLE', 'object_id': '337', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 43, 'continuous_valid_length': 43}, '339': {'type': 'VEHICLE', 'object_id': '339', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 55, 'continuous_valid_length': 55}, '341': {'type': 'VEHICLE', 'object_id': '341', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 34, 'continuous_valid_length': 34}, '342': {'type': 'VEHICLE', 'object_id': '342', 'track_length': 91, 'moving_distance': 113.51009303331375, 'valid_length': 91, 'continuous_valid_length': 91}, '349': {'type': 'VEHICLE', 'object_id': '349', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 2, 'continuous_valid_length': 2}, '352': {'type': 'VEHICLE', 'object_id': '352', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 9, 'continuous_valid_length': 5}, '353': {'type': 'VEHICLE', 'object_id': '353', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 1, 'continuous_valid_length': 1}, '354': {'type': 'VEHICLE', 'object_id': '354', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 66, 'continuous_valid_length': 66}, '365': {'type': 'VEHICLE', 'object_id': '365', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 3, 'continuous_valid_length': 2}, '368': {'type': 'VEHICLE', 'object_id': '368', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 5, 'continuous_valid_length': 5}, '371': {'type': 'VEHICLE', 'object_id': '371', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 31, 'continuous_valid_length': 2}, '375': {'type': 'VEHICLE', 'object_id': '375', 'track_length': 91, 'moving_distance': 1.276842713356018, 'valid_length': 4, 'continuous_valid_length': 4}, '376': {'type': 'VEHICLE', 'object_id': '376', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 29, 'continuous_valid_length': 3}, '378': {'type': 'VEHICLE', 'object_id': '378', 'track_length': 91, 'moving_distance': 0.8214730098843575, 'valid_length': 29, 'continuous_valid_length': 12}, '381': {'type': 'VEHICLE', 'object_id': '381', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 26, 'continuous_valid_length': 14}, '385': {'type': 'VEHICLE', 'object_id': '385', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 1, 'continuous_valid_length': 1}, '387': {'type': 'VEHICLE', 'object_id': '387', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 24, 'continuous_valid_length': 8}, '396': {'type': 'VEHICLE', 'object_id': '396', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 76, 'continuous_valid_length': 76}, '400': {'type': 'VEHICLE', 'object_id': '400', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 6, 'continuous_valid_length': 4}, '417': {'type': 'VEHICLE', 'object_id': '417', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 32, 'continuous_valid_length': 1}, '419': {'type': 'VEHICLE', 'object_id': '419', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 11, 'continuous_valid_length': 11}, '421': {'type': 'VEHICLE', 'object_id': '421', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 10, 'continuous_valid_length': 10}, '422': {'type': 'VEHICLE', 'object_id': '422', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 10, 'continuous_valid_length': 10}, '428': {'type': 'VEHICLE', 'object_id': '428', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 22, 'continuous_valid_length': 17}, '431': {'type': 'VEHICLE', 'object_id': '431', 'track_length': 91, 'moving_distance': 1.1140524987131357, 'valid_length': 55, 'continuous_valid_length': 55}, '432': {'type': 'VEHICLE', 'object_id': '432', 'track_length': 91, 'moving_distance': 0.7980888915481046, 'valid_length': 51, 'continuous_valid_length': 51}, '437': {'type': 'VEHICLE', 'object_id': '437', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 7, 'continuous_valid_length': 3}, '446': {'type': 'VEHICLE', 'object_id': '446', 'track_length': 91, 'moving_distance': 0.6166307427920401, 'valid_length': 19, 'continuous_valid_length': 19}, '451': {'type': 'VEHICLE', 'object_id': '451', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 60, 'continuous_valid_length': 59}, '452': {'type': 'VEHICLE', 'object_id': '452', 'track_length': 91, 'moving_distance': 6.6010882165282965, 'valid_length': 52, 'continuous_valid_length': 51}, '455': {'type': 'VEHICLE', 'object_id': '455', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 1, 'continuous_valid_length': 1}, '457': {'type': 'VEHICLE', 'object_id': '457', 'track_length': 91, 'moving_distance': 0.823718479485251, 'valid_length': 91, 'continuous_valid_length': 91}, '458': {'type': 'VEHICLE', 'object_id': '458', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 70, 'continuous_valid_length': 67}, '460': {'type': 'VEHICLE', 'object_id': '460', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 4, 'continuous_valid_length': 4}, '464': {'type': 'VEHICLE', 'object_id': '464', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 3, 'continuous_valid_length': 2}, '466': {'type': 'VEHICLE', 'object_id': '466', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 1, 'continuous_valid_length': 1}, '469': {'type': 'VEHICLE', 'object_id': '469', 'track_length': 91, 'moving_distance': 7.551295776618645, 'valid_length': 67, 'continuous_valid_length': 67}, '471': {'type': 'VEHICLE', 'object_id': '471', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 8, 'continuous_valid_length': 4}, '472': {'type': 'VEHICLE', 'object_id': '472', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 5, 'continuous_valid_length': 5}, '475': {'type': 'VEHICLE', 'object_id': '475', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 4, 'continuous_valid_length': 4}, '480': {'type': 'VEHICLE', 'object_id': '480', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 4, 'continuous_valid_length': 2}, '484': {'type': 'VEHICLE', 'object_id': '484', 'track_length': 91, 'moving_distance': 9.533195000141859, 'valid_length': 39, 'continuous_valid_length': 1}, '485': {'type': 'VEHICLE', 'object_id': '485', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 10, 'continuous_valid_length': 10}, '487': {'type': 'VEHICLE', 'object_id': '487', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 55, 'continuous_valid_length': 16}, '490': {'type': 'VEHICLE', 'object_id': '490', 'track_length': 91, 'moving_distance': 34.23079764936119, 'valid_length': 91, 'continuous_valid_length': 91}, '493': {'type': 'VEHICLE', 'object_id': '493', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 7, 'continuous_valid_length': 7}, '495': {'type': 'VEHICLE', 'object_id': '495', 'track_length': 91, 'moving_distance': 14.006807163357735, 'valid_length': 44, 'continuous_valid_length': 44}, '496': {'type': 'VEHICLE', 'object_id': '496', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 68, 'continuous_valid_length': 33}, '497': {'type': 'VEHICLE', 'object_id': '497', 'track_length': 91, 'moving_distance': 15.832195341587067, 'valid_length': 22, 'continuous_valid_length': 22}, '500': {'type': 'VEHICLE', 'object_id': '500', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 89, 'continuous_valid_length': 27}, '501': {'type': 'VEHICLE', 'object_id': '501', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 10, 'continuous_valid_length': 9}, '502': {'type': 'VEHICLE', 'object_id': '502', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 11, 'continuous_valid_length': 11}, '504': {'type': 'VEHICLE', 'object_id': '504', 'track_length': 91, 'moving_distance': 6.018707137554884, 'valid_length': 80, 'continuous_valid_length': 80}, '505': {'type': 'VEHICLE', 'object_id': '505', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 11, 'continuous_valid_length': 1}, '506': {'type': 'VEHICLE', 'object_id': '506', 'track_length': 91, 'moving_distance': 7.8919847551733255, 'valid_length': 49, 'continuous_valid_length': 49}, '507': {'type': 'VEHICLE', 'object_id': '507', 'track_length': 91, 'moving_distance': 13.17229554709047, 'valid_length': 91, 'continuous_valid_length': 91}, '508': {'type': 'VEHICLE', 'object_id': '508', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 18, 'continuous_valid_length': 8}, '509': {'type': 'VEHICLE', 'object_id': '509', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 7, 'continuous_valid_length': 1}, '511': {'type': 'VEHICLE', 'object_id': '511', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 19, 'continuous_valid_length': 1}, '515': {'type': 'VEHICLE', 'object_id': '515', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 44, 'continuous_valid_length': 22}, '516': {'type': 'VEHICLE', 'object_id': '516', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 21, 'continuous_valid_length': 8}, '517': {'type': 'VEHICLE', 'object_id': '517', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 9, 'continuous_valid_length': 1}, '518': {'type': 'VEHICLE', 'object_id': '518', 'track_length': 91, 'moving_distance': 5.325449822470546, 'valid_length': 91, 'continuous_valid_length': 91}, '519': {'type': 'VEHICLE', 'object_id': '519', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 18, 'continuous_valid_length': 10}, '526': {'type': 'VEHICLE', 'object_id': '526', 'track_length': 91, 'moving_distance': 16.80795606970787, 'valid_length': 27, 'continuous_valid_length': 26}, '527': {'type': 'VEHICLE', 'object_id': '527', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 13, 'continuous_valid_length': 2}, '530': {'type': 'VEHICLE', 'object_id': '530', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 86, 'continuous_valid_length': 21}, '531': {'type': 'VEHICLE', 'object_id': '531', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 88, 'continuous_valid_length': 88}, '532': {'type': 'VEHICLE', 'object_id': '532', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 11, 'continuous_valid_length': 11}, '534': {'type': 'VEHICLE', 'object_id': '534', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 88, 'continuous_valid_length': 39}, '536': {'type': 'VEHICLE', 'object_id': '536', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 8, 'continuous_valid_length': 8}, '537': {'type': 'VEHICLE', 'object_id': '537', 'track_length': 91, 'moving_distance': 0.4888385673984885, 'valid_length': 5, 'continuous_valid_length': 5}, '540': {'type': 'VEHICLE', 'object_id': '540', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 85, 'continuous_valid_length': 85}, '542': {'type': 'VEHICLE', 'object_id': '542', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 13, 'continuous_valid_length': 1}, '543': {'type': 'VEHICLE', 'object_id': '543', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 17, 'continuous_valid_length': 1}, '544': {'type': 'VEHICLE', 'object_id': '544', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 16, 'continuous_valid_length': 1}, '545': {'type': 'VEHICLE', 'object_id': '545', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 12, 'continuous_valid_length': 1}, '546': {'type': 'VEHICLE', 'object_id': '546', 'track_length': 91, 'moving_distance': 0.6660598255693913, 'valid_length': 5, 'continuous_valid_length': 5}, '547': {'type': 'VEHICLE', 'object_id': '547', 'track_length': 91, 'moving_distance': 1.0082283169031143, 'valid_length': 5, 'continuous_valid_length': 5}, '548': {'type': 'VEHICLE', 'object_id': '548', 'track_length': 91, 'moving_distance': 1.9234367492608726, 'valid_length': 80, 'continuous_valid_length': 80}, '549': {'type': 'VEHICLE', 'object_id': '549', 'track_length': 91, 'moving_distance': 26.11587554216385, 'valid_length': 37, 'continuous_valid_length': 36}, '550': {'type': 'VEHICLE', 'object_id': '550', 'track_length': 91, 'moving_distance': 1.9516869257204235, 'valid_length': 78, 'continuous_valid_length': 1}, '555': {'type': 'VEHICLE', 'object_id': '555', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 78, 'continuous_valid_length': 78}, '556': {'type': 'VEHICLE', 'object_id': '556', 'track_length': 91, 'moving_distance': 31.758361959829926, 'valid_length': 78, 'continuous_valid_length': 78}, '557': {'type': 'VEHICLE', 'object_id': '557', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 41, 'continuous_valid_length': 1}, '560': {'type': 'VEHICLE', 'object_id': '560', 'track_length': 91, 'moving_distance': 0.7265744209289551, 'valid_length': 5, 'continuous_valid_length': 5}, '562': {'type': 'VEHICLE', 'object_id': '562', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 13, 'continuous_valid_length': 9}, '566': {'type': 'VEHICLE', 'object_id': '566', 'track_length': 91, 'moving_distance': 9.712436974048615, 'valid_length': 14, 'continuous_valid_length': 2}, '568': {'type': 'VEHICLE', 'object_id': '568', 'track_length': 91, 'moving_distance': 1.843812669860199, 'valid_length': 73, 'continuous_valid_length': 73}, '569': {'type': 'VEHICLE', 'object_id': '569', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 73, 'continuous_valid_length': 73}, '570': {'type': 'VEHICLE', 'object_id': '570', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 19, 'continuous_valid_length': 1}, '571': {'type': 'VEHICLE', 'object_id': '571', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 12, 'continuous_valid_length': 1}, '572': {'type': 'VEHICLE', 'object_id': '572', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 19, 'continuous_valid_length': 2}, '574': {'type': 'VEHICLE', 'object_id': '574', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 24, 'continuous_valid_length': 9}, '576': {'type': 'VEHICLE', 'object_id': '576', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 10, 'continuous_valid_length': 10}, '578': {'type': 'VEHICLE', 'object_id': '578', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 68, 'continuous_valid_length': 15}, '579': {'type': 'VEHICLE', 'object_id': '579', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 10, 'continuous_valid_length': 10}, '580': {'type': 'VEHICLE', 'object_id': '580', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 68, 'continuous_valid_length': 21}, '581': {'type': 'VEHICLE', 'object_id': '581', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 22, 'continuous_valid_length': 21}, '582': {'type': 'VEHICLE', 'object_id': '582', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 73, 'continuous_valid_length': 73}, '583': {'type': 'VEHICLE', 'object_id': '583', 'track_length': 91, 'moving_distance': 5.7331065577454865, 'valid_length': 68, 'continuous_valid_length': 1}, '584': {'type': 'VEHICLE', 'object_id': '584', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 66, 'continuous_valid_length': 18}, '585': {'type': 'VEHICLE', 'object_id': '585', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 23, 'continuous_valid_length': 23}, '587': {'type': 'VEHICLE', 'object_id': '587', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 67, 'continuous_valid_length': 67}, '591': {'type': 'VEHICLE', 'object_id': '591', 'track_length': 91, 'moving_distance': 37.11683337390423, 'valid_length': 66, 'continuous_valid_length': 66}, '592': {'type': 'VEHICLE', 'object_id': '592', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 66, 'continuous_valid_length': 66}, '593': {'type': 'VEHICLE', 'object_id': '593', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 64, 'continuous_valid_length': 19}, '594': {'type': 'VEHICLE', 'object_id': '594', 'track_length': 91, 'moving_distance': 15.813339234795421, 'valid_length': 66, 'continuous_valid_length': 66}, '596': {'type': 'VEHICLE', 'object_id': '596', 'track_length': 91, 'moving_distance': 11.588347688317299, 'valid_length': 66, 'continuous_valid_length': 66}, '597': {'type': 'VEHICLE', 'object_id': '597', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 68, 'continuous_valid_length': 2}, '598': {'type': 'VEHICLE', 'object_id': '598', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 66, 'continuous_valid_length': 23}, '599': {'type': 'VEHICLE', 'object_id': '599', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 65, 'continuous_valid_length': 22}, '601': {'type': 'VEHICLE', 'object_id': '601', 'track_length': 91, 'moving_distance': 6.672057034331374, 'valid_length': 64, 'continuous_valid_length': 64}, '602': {'type': 'VEHICLE', 'object_id': '602', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 18, 'continuous_valid_length': 4}, '603': {'type': 'VEHICLE', 'object_id': '603', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 23, 'continuous_valid_length': 16}, '604': {'type': 'VEHICLE', 'object_id': '604', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 64, 'continuous_valid_length': 64}, '606': {'type': 'VEHICLE', 'object_id': '606', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 63, 'continuous_valid_length': 1}, '607': {'type': 'VEHICLE', 'object_id': '607', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 62, 'continuous_valid_length': 62}, '608': {'type': 'VEHICLE', 'object_id': '608', 'track_length': 91, 'moving_distance': 1.405983716249466, 'valid_length': 6, 'continuous_valid_length': 6}, '610': {'type': 'VEHICLE', 'object_id': '610', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 62, 'continuous_valid_length': 1}, '612': {'type': 'VEHICLE', 'object_id': '612', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 12, 'continuous_valid_length': 12}, '613': {'type': 'VEHICLE', 'object_id': '613', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 60, 'continuous_valid_length': 60}, '615': {'type': 'VEHICLE', 'object_id': '615', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 58, 'continuous_valid_length': 58}, '616': {'type': 'VEHICLE', 'object_id': '616', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 60, 'continuous_valid_length': 60}, '617': {'type': 'VEHICLE', 'object_id': '617', 'track_length': 91, 'moving_distance': 6.042413581628352, 'valid_length': 52, 'continuous_valid_length': 1}, '621': {'type': 'VEHICLE', 'object_id': '621', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 60, 'continuous_valid_length': 8}, '622': {'type': 'VEHICLE', 'object_id': '622', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 57, 'continuous_valid_length': 57}, '624': {'type': 'VEHICLE', 'object_id': '624', 'track_length': 91, 'moving_distance': 2.07014262676239, 'valid_length': 8, 'continuous_valid_length': 8}, '627': {'type': 'VEHICLE', 'object_id': '627', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 53, 'continuous_valid_length': 53}, '628': {'type': 'VEHICLE', 'object_id': '628', 'track_length': 91, 'moving_distance': 0.25654466077685356, 'valid_length': 5, 'continuous_valid_length': 5}, '629': {'type': 'VEHICLE', 'object_id': '629', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 52, 'continuous_valid_length': 52}, '631': {'type': 'VEHICLE', 'object_id': '631', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 51, 'continuous_valid_length': 51}, '632': {'type': 'VEHICLE', 'object_id': '632', 'track_length': 91, 'moving_distance': 15.686851650476456, 'valid_length': 48, 'continuous_valid_length': 4}, '634': {'type': 'VEHICLE', 'object_id': '634', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 48, 'continuous_valid_length': 48}, '635': {'type': 'VEHICLE', 'object_id': '635', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 29, 'continuous_valid_length': 3}, '637': {'type': 'VEHICLE', 'object_id': '637', 'track_length': 91, 'moving_distance': 5.095191866159439, 'valid_length': 46, 'continuous_valid_length': 46}, '639': {'type': 'VEHICLE', 'object_id': '639', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 48, 'continuous_valid_length': 48}, '640': {'type': 'VEHICLE', 'object_id': '640', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 49, 'continuous_valid_length': 4}, '642': {'type': 'VEHICLE', 'object_id': '642', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 48, 'continuous_valid_length': 48}, '644': {'type': 'VEHICLE', 'object_id': '644', 'track_length': 91, 'moving_distance': 1.7224541562609375, 'valid_length': 39, 'continuous_valid_length': 2}, '645': {'type': 'VEHICLE', 'object_id': '645', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 42, 'continuous_valid_length': 42}, '646': {'type': 'VEHICLE', 'object_id': '646', 'track_length': 91, 'moving_distance': 4.590390021679923, 'valid_length': 41, 'continuous_valid_length': 1}, '647': {'type': 'VEHICLE', 'object_id': '647', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 19, 'continuous_valid_length': 6}, '648': {'type': 'VEHICLE', 'object_id': '648', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 46, 'continuous_valid_length': 46}, '649': {'type': 'VEHICLE', 'object_id': '649', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 41, 'continuous_valid_length': 41}, '650': {'type': 'VEHICLE', 'object_id': '650', 'track_length': 91, 'moving_distance': 1.6701414708513767, 'valid_length': 38, 'continuous_valid_length': 5}, '651': {'type': 'VEHICLE', 'object_id': '651', 'track_length': 91, 'moving_distance': 5.15161857008934, 'valid_length': 38, 'continuous_valid_length': 38}, '652': {'type': 'VEHICLE', 'object_id': '652', 'track_length': 91, 'moving_distance': 14.368283748626709, 'valid_length': 13, 'continuous_valid_length': 13}, '654': {'type': 'VEHICLE', 'object_id': '654', 'track_length': 91, 'moving_distance': 46.69343686103821, 'valid_length': 35, 'continuous_valid_length': 35}, '656': {'type': 'VEHICLE', 'object_id': '656', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 44, 'continuous_valid_length': 44}, '657': {'type': 'VEHICLE', 'object_id': '657', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 43, 'continuous_valid_length': 43}, '658': {'type': 'VEHICLE', 'object_id': '658', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 35, 'continuous_valid_length': 1}, '660': {'type': 'VEHICLE', 'object_id': '660', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 36, 'continuous_valid_length': 23}, '661': {'type': 'VEHICLE', 'object_id': '661', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 37, 'continuous_valid_length': 9}, '662': {'type': 'VEHICLE', 'object_id': '662', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 35, 'continuous_valid_length': 35}, '663': {'type': 'VEHICLE', 'object_id': '663', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 25, 'continuous_valid_length': 25}, '664': {'type': 'VEHICLE', 'object_id': '664', 'track_length': 91, 'moving_distance': 21.729984641075134, 'valid_length': 24, 'continuous_valid_length': 24}, '665': {'type': 'VEHICLE', 'object_id': '665', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 31, 'continuous_valid_length': 31}, '666': {'type': 'VEHICLE', 'object_id': '666', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 19, 'continuous_valid_length': 18}, '667': {'type': 'VEHICLE', 'object_id': '667', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 30, 'continuous_valid_length': 1}, '668': {'type': 'VEHICLE', 'object_id': '668', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 20, 'continuous_valid_length': 5}, '669': {'type': 'VEHICLE', 'object_id': '669', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 29, 'continuous_valid_length': 29}, '670': {'type': 'VEHICLE', 'object_id': '670', 'track_length': 91, 'moving_distance': 12.148452699184418, 'valid_length': 17, 'continuous_valid_length': 13}, '671': {'type': 'VEHICLE', 'object_id': '671', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 21, 'continuous_valid_length': 21}, '673': {'type': 'VEHICLE', 'object_id': '673', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 26, 'continuous_valid_length': 26}, '675': {'type': 'VEHICLE', 'object_id': '675', 'track_length': 91, 'moving_distance': 1.6272050365805626, 'valid_length': 15, 'continuous_valid_length': 15}, '676': {'type': 'VEHICLE', 'object_id': '676', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 22, 'continuous_valid_length': 6}, '677': {'type': 'VEHICLE', 'object_id': '677', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 24, 'continuous_valid_length': 24}, '678': {'type': 'VEHICLE', 'object_id': '678', 'track_length': 91, 'moving_distance': 13.123132586479187, 'valid_length': 13, 'continuous_valid_length': 13}, '679': {'type': 'VEHICLE', 'object_id': '679', 'track_length': 91, 'moving_distance': 10.786753237247467, 'valid_length': 13, 'continuous_valid_length': 13}, '680': {'type': 'VEHICLE', 'object_id': '680', 'track_length': 91, 'moving_distance': 7.589548408985138, 'valid_length': 13, 'continuous_valid_length': 13}, '683': {'type': 'VEHICLE', 'object_id': '683', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 18, 'continuous_valid_length': 1}, '684': {'type': 'VEHICLE', 'object_id': '684', 'track_length': 91, 'moving_distance': 12.004751324653625, 'valid_length': 9, 'continuous_valid_length': 8}, '685': {'type': 'VEHICLE', 'object_id': '685', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 12, 'continuous_valid_length': 1}, '687': {'type': 'VEHICLE', 'object_id': '687', 'track_length': 91, 'moving_distance': 6.919632136821747, 'valid_length': 8, 'continuous_valid_length': 8}, '690': {'type': 'VEHICLE', 'object_id': '690', 'track_length': 91, 'moving_distance': 1.319422960281372, 'valid_length': 4, 'continuous_valid_length': 4}, '691': {'type': 'VEHICLE', 'object_id': '691', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 6, 'continuous_valid_length': 6}, '692': {'type': 'VEHICLE', 'object_id': '692', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 1, 'continuous_valid_length': 1}, '752': {'type': 'PEDESTRIAN', 'object_id': '752', 'track_length': 91, 'moving_distance': 9.990137495100498, 'valid_length': 70, 'continuous_valid_length': 70}, '757': {'type': 'PEDESTRIAN', 'object_id': '757', 'track_length': 91, 'moving_distance': 0.9828233160078526, 'valid_length': 11, 'continuous_valid_length': 1}, '761': {'type': 'PEDESTRIAN', 'object_id': '761', 'track_length': 91, 'moving_distance': 0.5501821283251047, 'valid_length': 7, 'continuous_valid_length': 1}, '763': {'type': 'PEDESTRIAN', 'object_id': '763', 'track_length': 91, 'moving_distance': 0.8864371012896299, 'valid_length': 13, 'continuous_valid_length': 5}, '775': {'type': 'PEDESTRIAN', 'object_id': '775', 'track_length': 91, 'moving_distance': 1.2418494001030922, 'valid_length': 11, 'continuous_valid_length': 11}, '778': {'type': 'PEDESTRIAN', 'object_id': '778', 'track_length': 91, 'moving_distance': 4.065424693748355, 'valid_length': 22, 'continuous_valid_length': 2}, '779': {'type': 'PEDESTRIAN', 'object_id': '779', 'track_length': 91, 'moving_distance': 1.8515599071979523, 'valid_length': 17, 'continuous_valid_length': 16}, '786': {'type': 'PEDESTRIAN', 'object_id': '786', 'track_length': 91, 'moving_distance': 1.2681039795279503, 'valid_length': 11, 'continuous_valid_length': 1}, '788': {'type': 'PEDESTRIAN', 'object_id': '788', 'track_length': 91, 'moving_distance': 1.7190424352884293, 'valid_length': 12, 'continuous_valid_length': 12}, '789': {'type': 'PEDESTRIAN', 'object_id': '789', 'track_length': 91, 'moving_distance': 0.802222803235054, 'valid_length': 9, 'continuous_valid_length': 9}, '795': {'type': 'PEDESTRIAN', 'object_id': '795', 'track_length': 91, 'moving_distance': 3.484301283955574, 'valid_length': 26, 'continuous_valid_length': 26}, '796': {'type': 'PEDESTRIAN', 'object_id': '796', 'track_length': 91, 'moving_distance': 1.7422855999320745, 'valid_length': 24, 'continuous_valid_length': 24}, '798': {'type': 'PEDESTRIAN', 'object_id': '798', 'track_length': 91, 'moving_distance': 1.9151662737131119, 'valid_length': 20, 'continuous_valid_length': 20}, '800': {'type': 'PEDESTRIAN', 'object_id': '800', 'track_length': 91, 'moving_distance': 1.0499958880245686, 'valid_length': 16, 'continuous_valid_length': 10}, '802': {'type': 'PEDESTRIAN', 'object_id': '802', 'track_length': 91, 'moving_distance': 0.0, 'valid_length': 1, 'continuous_valid_length': 1}, '803': {'type': 'PEDESTRIAN', 'object_id': '803', 'track_length': 91, 'moving_distance': 1.7290306314826012, 'valid_length': 17, 'continuous_valid_length': 17}, '807': {'type': 'PEDESTRIAN', 'object_id': '807', 'track_length': 91, 'moving_distance': 1.7332774251699448, 'valid_length': 13, 'continuous_valid_length': 13}, '808': {'type': 'PEDESTRIAN', 'object_id': '808', 'track_length': 91, 'moving_distance': 0.7331183645874262, 'valid_length': 7, 'continuous_valid_length': 7}, '809': {'type': 'PEDESTRIAN', 'object_id': '809', 'track_length': 91, 'moving_distance': 1.4187113791704178, 'valid_length': 15, 'continuous_valid_length': 15}, '810': {'type': 'PEDESTRIAN', 'object_id': '810', 'track_length': 91, 'moving_distance': 4.130044959485531, 'valid_length': 10, 'continuous_valid_length': 5}, '811': {'type': 'PEDESTRIAN', 'object_id': '811', 'track_length': 91, 'moving_distance': 0.6255718544125557, 'valid_length': 4, 'continuous_valid_length': 4}, '812': {'type': 'PEDESTRIAN', 'object_id': '812', 'track_length': 91, 'moving_distance': 1.3105508536100388, 'valid_length': 14, 'continuous_valid_length': 14}, '818': {'type': 'PEDESTRIAN', 'object_id': '818', 'track_length': 91, 'moving_distance': 0.37125103222206235, 'valid_length': 9, 'continuous_valid_length': 3}, '825': {'type': 'PEDESTRIAN', 'object_id': '825', 'track_length': 91, 'moving_distance': 0.36107008159160614, 'valid_length': 4, 'continuous_valid_length': 4}, '827': {'type': 'PEDESTRIAN', 'object_id': '827', 'track_length': 91, 'moving_distance': 0.015506293624639511, 'valid_length': 2, 'continuous_valid_length': 2}, '1624': {'type': 'CYCLIST', 'object_id': '1624', 'track_length': 91, 'moving_distance': 19.005091778934002, 'valid_length': 80, 'continuous_valid_length': 61}, '1625': {'type': 'CYCLIST', 'object_id': '1625', 'track_length': 91, 'moving_distance': 2.818502277135849, 'valid_length': 11, 'continuous_valid_length': 11}, '1629': {'type': 'VEHICLE', 'object_id': '1629', 'track_length': 91, 'moving_distance': 84.20105910301208, 'valid_length': 91, 'continuous_valid_length': 91}}
# You might also find that the summaries of scenarios from different datasets might vary.
print(f"Keys of the scenario from Waymo: {example_scenario_summary.keys()}. \n")
print("Keys of the scenario from nuScene: ", sd_utils.read_dataset_summary(nuscenes_data)[0]['sd_nuscenes_v1.0-mini_scene-0061.pkl'].keys())
Keys of the scenario from Waymo: dict_keys(['coordinate', 'ts', 'metadrive_processed', 'sdc_id', 'dataset', 'scenario_id', 'source_file', 'track_length', 'current_time_index', 'sdc_track_index', 'objects_of_interest', 'tracks_to_predict', 'object_summary', 'number_summary']).
Keys of the scenario from nuScene: dict_keys(['dataset', 'metadrive_processed', 'map', 'date', 'coordinate', 'scenario_token', 'id', 'scenario_id', 'sample_rate', 'ts', 'sdc_id', 'object_summary', 'number_summary'])
What is the other two returns from read_dataset_summary?
Recall that sd_utils.read_dataset_summary(dataset_path) will return a tuple of three elements.
The second element is a list containing the .pkl file names of all scenarios.
The third element is a dict mapping the .pkl file names to their relative path from dataset_summary.pkl file.
Case 1: All .pkl files are in the same folder as dataset_summary.pkl
_, waymo_scenario_ids, waymo_scenario_files = sd_utils.read_dataset_summary(waymo_data)
print("Scenarios ID: ", waymo_scenario_ids)
# As those .pkl files are in the same folder of `dataset_summary.pkl`, there is no relative path from pkl to summary file.
print("The mapping: ", waymo_scenario_files)
Scenarios ID: ['sd_training.tfrecord-00000-of-01000_2a1e44d405a6833f.pkl', 'sd_training.tfrecord-00000-of-01000_c403d5992cab9e0.pkl', 'sd_training.tfrecord-00000-of-01000_8a346109094cd5aa.pkl']
The mapping: {'sd_training.tfrecord-00000-of-01000_2a1e44d405a6833f.pkl': '', 'sd_training.tfrecord-00000-of-01000_c403d5992cab9e0.pkl': '', 'sd_training.tfrecord-00000-of-01000_8a346109094cd5aa.pkl': ''}
Case 2: The .pkl files are not in the same folder as dataset_summary.pkl
print("Reading from: ", nuscenes_data)
_, nuscenes_scenario_ids, nuscenes_scenario_files = sd_utils.read_dataset_summary(nuscenes_data)
print("Scenarios ID: ", nuscenes_scenario_ids)
# As those .pkl files for nuscenes dataset is stored in separate subfolder from the folder containing `dataset_summary.pkl`,
# there are the relative paths.
#
# For example, the relative path of `sd_nuscenes_v1.0-mini_scene-0061.pkl` can be created by appending:
# nuscenes_scenario_files[`sd_nuscenes_v1.0-mini_scene-0061.pkl`] + '/' + 'sd_nuscenes_v1.0-mini_scene-0061.pkl'
print("The mapping: ", nuscenes_scenario_files)
Reading from: /home/zhenghao/metadrive/metadrive/assets/nuscenes
Scenarios ID: ['sd_nuscenes_v1.0-mini_scene-0061.pkl', 'sd_nuscenes_v1.0-mini_scene-0103.pkl', 'sd_nuscenes_v1.0-mini_scene-0553.pkl', 'sd_nuscenes_v1.0-mini_scene-0655.pkl', 'sd_nuscenes_v1.0-mini_scene-0757.pkl', 'sd_nuscenes_v1.0-mini_scene-0796.pkl', 'sd_nuscenes_v1.0-mini_scene-0916.pkl', 'sd_nuscenes_v1.0-mini_scene-1077.pkl', 'sd_nuscenes_v1.0-mini_scene-1094.pkl', 'sd_nuscenes_v1.0-mini_scene-1100.pkl']
The mapping: {'sd_nuscenes_v1.0-mini_scene-0061.pkl': 'nuscenes_0', 'sd_nuscenes_v1.0-mini_scene-0103.pkl': 'nuscenes_1', 'sd_nuscenes_v1.0-mini_scene-0553.pkl': 'nuscenes_2', 'sd_nuscenes_v1.0-mini_scene-0655.pkl': 'nuscenes_3', 'sd_nuscenes_v1.0-mini_scene-0757.pkl': 'nuscenes_4', 'sd_nuscenes_v1.0-mini_scene-0796.pkl': 'nuscenes_5', 'sd_nuscenes_v1.0-mini_scene-0916.pkl': 'nuscenes_6', 'sd_nuscenes_v1.0-mini_scene-1077.pkl': 'nuscenes_7', 'sd_nuscenes_v1.0-mini_scene-1094.pkl': 'nuscenes_7', 'sd_nuscenes_v1.0-mini_scene-1100.pkl': 'nuscenes_7'}
What is the Scenario Description? ⭐
TL;DR: Scenario Description is a nested python dict that stores the information of one AD scenario.
By defining the unified data structure, from the input end, we can define a set of toolkits to convert each dataset into the dataset in the form of Scenario Description.
Then we can store each scenario into a .pkl file in the local file system and manage those files with ScenarioNet.
From the output end, no matter where the source data is from, we can reconstruct interactive environment via MetaDrive and conduct downstream tasks like RL training by reading from the dataset containing those .pkl files.
Here is an example of what the Scenario Description looks like:
scenario = {
# ===== Meta data about the scenario =====
# string. The name of the scenario
"id": "Waymo-001",
# string. The version of data format.
"version": "MetaDrive v0.3.0.1",
# int. The length of all trajectory and state arrays (T).
"length": 200,
# ===== Meta data ===
"metadata": {
# np.ndarray in (T, ). The time stamp of each time step.
"ts": np.array([0.0, 0.1, 0.2, ...], dtype=np.float32),
# bool. Whether the scenario is processed and exported by MetaDrive.
# Some operations may be done, such as interpolating the lane to
# make way points uniformly scattered in given interval.
"metadrive_processed": True,
# string. Coordinate system.
"coordinate": "metadrive",
# optional keys
"source_file": "training_20s.tfrecord-00014-of-01000",
"dataset": "waymo",
"scenario_id": "dd0c8c27fdd6ef59", # Used in Waymo dataset
"seed": 512,
"history_metadata": {},
"sdc_id": "172", # A key exists in tracks
},
# ===== Trajectories of active participants, e.g. vehicles, pedestrians =====
# a dict mapping object ID to it's state dict.
"tracks": {
"vehicle1": {
# The type string in metadrive.type.MetaDriveType
"type": "VEHICLE",
# The state dict. All values must have T elements.
"state": {
"position": np.ones([200, 3], dtype=np.float32),
...
},
# The meta data dict. Store useful information about the object. type in metadata could be those from
# different dataset
"metadata": {
"type": "VEHICLE",
"track_length": 200,
"object_id": "vehicle1",
# Optional keys
"agent_name": "default_agent",
"policy_spawn_info": { # Information needed to re-instantiate the policy
"policy_class": ("metadrive.policy.idm_policy", "IDMPolicy"),
"args": ...,
"kwargs": ...,
}
}
},
"pedestrian1": ...
},
# ===== States sequence of dynamics objects, e.g. traffic light =====
# a dict mapping object ID to it's state dict.
"dynamic_map_states": {
"trafficlight1": {
# The type string in metadrive.type.MetaDriveType
"type": "TRAFFIC_LIGHT",
# The state dict. All values must have T elements.
"state": {
"object_state": np.ones([200, ], dtype=int),
...
},
# The meta data dict. Store useful information about the object
"metadata": {
"type": "TRAFFIC_LIGHT",
"track_length": 200,
}
}
# ===== Map features =====
# A dict mapping from map feature ID to a line segment
"map_features": {
"219": {
"type": "LANE_SURFACE_STREET",
"polyline": np.array in [21, 2], # A set of 2D points describing a line segment
# optional, only works for lane
"polygon": np.array in [N, 2] # A set of 2D points representing convexhull
},
"182": ...
...
}
}
# Note: Example is from the docstring of metadrive/scenario/scenario_description.py
Read a scenario description in .pkl with MetaDrive utils
We want to avoid using any advanced tools to ensure the maximal compatibility. Here let me demonstrate how to read a .pkl file and instantiate a Scenario Description from it.
We will use read_scenario to read a given .pkl file, and it will return a ScenarioDescrption instance. A ScenarioDescrption is a wrapper of native python dict with some useful utility functions.
# Get the dataset path
dataset_path = waymo_data
print("Dataset path: ", dataset_path)
# Get the scenario .pkl file name
_, scenario_ids, dataset_mapping = sd_utils.read_dataset_summary(dataset_path)
# Just pick the first scenario
scenario_pkl_file = scenario_ids[0]
# Get the relative path to the .pkl file
print("The pkl file relative path: ", dataset_mapping[scenario_pkl_file]) # An empty path
# Get the absolute path to the .pkl file
abs_path_to_pkl_file = os.path.join(dataset_path, dataset_mapping[scenario_pkl_file], scenario_pkl_file)
print("The pkl file absolute path: ", abs_path_to_pkl_file)
# Call utility function in MD and get the Scenario Description object
scenario = sd_utils.read_scenario_data(abs_path_to_pkl_file)
print(f"\nThe raw data type after reading the .pkl file is {type(scenario)}")
Dataset path: /home/zhenghao/metadrive/metadrive/assets/waymo
The pkl file relative path:
The pkl file absolute path: /home/zhenghao/metadrive/metadrive/assets/waymo/sd_training.tfrecord-00000-of-01000_2a1e44d405a6833f.pkl
The raw data type after reading the .pkl file is <class 'metadrive.scenario.scenario_description.ScenarioDescription'>
print(f"The keys in a ScenarioDescription are: {scenario.keys()}")
The keys in a Scenario Description are: dict_keys(['id', 'version', 'length', 'tracks', 'dynamic_map_states', 'map_features', 'metadata'])
# Now you can play with the data in scenario as it's like a python dict:
print(f"The SDC vehicle name is: {scenario['metadata']['sdc_id']}")
sdc_id = scenario['metadata']['sdc_id']
print(f"The SDC vehicle type is: {scenario['tracks'][sdc_id]['type']}")
print(f"The SDC vehicle position at last time step is: {scenario['tracks'][sdc_id]['state']['position'][-1]}")
print(f"There are traffic lights: {scenario['dynamic_map_states'].keys()}")
print(f"The traffic light 316's state is: {scenario['dynamic_map_states']['316']['state']['object_state']}")
The SDC vehicle name is: 1629
The SDC vehicle type is: VEHICLE
The SDC vehicle position at last time step is: [ 2976.9563 -3251.819 186.91092]
There are traffic lights: dict_keys(['226', '227', '272', '316', '317', '321', '354', '355'])
The traffic light 316's state is: ['LANE_STATE_UNKNOWN', 'LANE_STATE_UNKNOWN', 'LANE_STATE_UNKNOWN', 'LANE_STATE_UNKNOWN', 'LANE_STATE_UNKNOWN', 'LANE_STATE_UNKNOWN', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP', 'LANE_STATE_STOP']
Useful utility functions from ScenarioDescription
Note that these functions are class functions.
# Check the self-consistency of a scenario and whether the required fields are filled.
scenario.sanity_check?
scenario.sanity_check(scenario) # If no error is raised, the check passes.
Signature:
scenario.sanity_check(
scenario_dict,
check_self_type=False,
valid_check=False,
)
Docstring:
Check if the input scenario dict is self-consistent and has filled required fields.
The required high-level fields include tracks, dynamic_map_states, metadata, map_features.
For each object, the tracks[obj_id] should at least contain type, state, metadata.
For each object, the tracks[obj_id]['state'] should at least contain position, heading.
For each lane in map_features, map_feature[map_feat_id] should at least contain polyline.
For metadata, it should at least contain metadrive_processed, coordinate and timestep.
We have more checks to ensure the consistency of the data.
Args:
scenario_dict: the input dict.
check_self_type: if True, assert the input dict is a native Python dict.
valid_check: if True, we will assert the values for a given timestep are zeros if valid=False at that
timestep.
File: ~/metadrive/metadrive/scenario/scenario_description.py
Type: method
# Convert the ScenarioDescription to a python dict.
scenario_dict = scenario.to_dict()
print(f"The keys in the converted dict are: {scenario_dict.keys()}")
The keys in the converted dict are: dict_keys(['id', 'version', 'length', 'tracks', 'dynamic_map_states', 'map_features', 'metadata'])
# Return the object info dict of the SDC.
sdc_track = scenario.get_sdc_track()
print("The 'tracks' field stores the information of an object, including its type, state and metadata. "
f"The object info dict contains these keys: {sdc_track.keys()}\n")
print("We demo the state vectors stored in the SDC's info dict: \n", {k: v.shape for k, v in sdc_track['state'].items()})
print("\nBut note that different type of objects might have different state vectors.")
The 'tracks' field stores the information of an object, including its type, state and metadata. The object info dict contains these keys: dict_keys(['type', 'state', 'metadata'])
We demo the state vectors stored in the SDC's info dict:
{'position': (91, 3), 'length': (91,), 'width': (91,), 'height': (91,), 'heading': (91,), 'velocity': (91, 2), 'valid': (91,)}
But note that different type of objects might have different state vectors.
# Compute the moving distance of SDC.
sd_utils.ScenarioDescription.sdc_moving_dist(scenario)
84.20105910301208
# Verify if the .pkl file is valid.
sd_utils.ScenarioDescription.is_scenario_file(abs_path_to_pkl_file)
True
# Get the summary of an object. Here we use the SDC.
sd_utils.ScenarioDescription.get_object_summary(
object_dict=scenario.get_sdc_track(),
object_id=scenario['metadata']['sdc_id']
)
{'type': 'VEHICLE',
'object_id': '1629',
'track_length': 91,
'moving_distance': 84.20105910301208,
'valid_length': 91,
'continuous_valid_length': 91}
# Get the stats of all objects in a scenario.
sd_utils.ScenarioDescription.get_number_summary(
scenario=scenario
)
{'num_objects': 211,
'object_types': {'CYCLIST', 'PEDESTRIAN', 'VEHICLE'},
'num_objects_each_type': {'VEHICLE': 184, 'PEDESTRIAN': 25, 'CYCLIST': 2},
'num_moving_objects': 69,
'num_moving_objects_each_type': defaultdict(int,
{'VEHICLE': 52, 'PEDESTRIAN': 15, 'CYCLIST': 2}),
'num_traffic_lights': 8,
'num_traffic_light_types': {'LANE_STATE_STOP', 'LANE_STATE_UNKNOWN'},
'num_traffic_light_each_step': {'LANE_STATE_UNKNOWN': 164,
'LANE_STATE_STOP': 564},
'num_map_features': 358,
'map_height_diff': 2.4652252197265625}
# Get the number of all objects in a scenario.
sd_utils.ScenarioDescription.get_num_objects(
scenario=scenario
)
211
# Get the number of all moving objects in a scenario.
sd_utils.ScenarioDescription.get_num_moving_objects(
scenario=scenario
)
69
How to filter Scenarios?
To ensure the maximal compatibility and simplicity, we use the native file systems to store all .pkl files, so you can use native Python loop to read and filter scenarios.
We have defined some high-level commands to conduct filtering in ScenarioNet: https://scenarionet.readthedocs.io/en/latest/operations.html#filter
To give more flexibility, here we provide the full example of how you can build a ScenarioNet dataset by filtering existing data.
In this example, we try to filter the scenario if there exists traffic lights in the scene.
Note that if you replace “waymo” in AssetLoader by “nuscenes”, you will find that the new dataset does not include any scenario as none of the scenarios in the example nuscenes dataset contains traffic light.
import os
import pickle
from pathlib import Path
import tqdm
import metadrive.scenario.utils as sd_utils
from metadrive.engine.asset_loader import AssetLoader
from metadrive.scenario.scenario_description import ScenarioDescription as SD
# === Specify the path to the original and the new dataset ===
waymo_data = AssetLoader.file_path(AssetLoader.asset_path, "waymo", unix_style=False)
dataset_path = waymo_data
new_dataset_path = "./filtered_dataset"
# ============================================================
new_dataset_path = Path(new_dataset_path).resolve()
if not os.path.isdir(new_dataset_path):
os.makedirs(new_dataset_path)
print(f"Reading existing dataset at: {dataset_path}")
print(f"Will save new dataset at: {new_dataset_path}")
print("\n")
# Read the summary
scenario_summaries, scenario_ids, dataset_mapping = sd_utils.read_dataset_summary(dataset_path)
# Define a filter function that return True if the scenario is accepted
def filter(scenario):
# Get the number of traffic light
num_tl = len(scenario[SD.DYNAMIC_MAP_STATES])
# You can also get the number from metadata (recommended)
num_tl_from_metadata = scenario[SD.METADATA][SD.SUMMARY.NUMBER_SUMMARY][SD.SUMMARY.NUM_TRAFFIC_LIGHTS]
assert num_tl_from_metadata == num_tl
print(f"We found {num_tl} traffic lights in scenario {scenario[SD.ID]}")
has_traffic_light = num_tl > 0
return has_traffic_light
# Iterate over all scenarios
remove_scenario = []
new_mapping = {}
for file_name in tqdm.tqdm(scenario_summaries.keys(), desc="Filter Scenarios"):
abs_path = Path(dataset_path) / dataset_mapping[file_name] / file_name
abs_path_to_file_dir = Path(dataset_path) / dataset_mapping[file_name]
# Call utility function in MD and get the ScenarioDescription object
scenario = sd_utils.read_scenario_data(abs_path)
# Filter
accepted = filter(scenario)
print(f"Processing: {file_name}. This scenario is {'accepted' if accepted else 'rejected'}.")
if not accepted:
remove_scenario.append(file_name)
# Translate the relative path in mapping to the new location
new_mapping[file_name] = os.path.relpath(abs_path_to_file_dir, new_dataset_path)
print("\n")
for file in remove_scenario:
scenario_summaries.pop(file)
new_mapping.pop(file)
summary_file_path = new_dataset_path / SD.DATASET.SUMMARY_FILE
with open(summary_file_path, "wb") as file:
pickle.dump(scenario_summaries, file)
print(f"Summary file is saved at: {summary_file_path}")
mapping_file_path = new_dataset_path / SD.DATASET.MAPPING_FILE
with open(mapping_file_path, "wb") as file:
pickle.dump(new_mapping, file)
print(f"Mapping file is saved at: {summary_file_path}")
# Verify
_, _, new_mapping = sd_utils.read_dataset_summary(new_dataset_path)
print(f"\nLet's verify if the new dataset is valid."
f"\n{len(new_mapping)} scenarios are in the new dataset."
f"\nThe new mapping:\n{new_mapping}")
Reading existing dataset at: /home/zhenghao/metadrive/metadrive/assets/waymo
Will save new dataset at: /home/zhenghao/metadrive/documentation/source/filtered_dataset
Filter Scenarios: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 421.41it/s]
We found 8 traffic lights in scenario 2a1e44d405a6833f
Processing: sd_training.tfrecord-00000-of-01000_2a1e44d405a6833f.pkl. This scenario is accepted.
We found 8 traffic lights in scenario c403d5992cab9e0
Processing: sd_training.tfrecord-00000-of-01000_c403d5992cab9e0.pkl. This scenario is accepted.
We found 19 traffic lights in scenario 8a346109094cd5aa
Processing: sd_training.tfrecord-00000-of-01000_8a346109094cd5aa.pkl. This scenario is accepted.
Summary file is saved at: /home/zhenghao/metadrive/documentation/source/filtered_dataset/dataset_summary.pkl
Mapping file is saved at: /home/zhenghao/metadrive/documentation/source/filtered_dataset/dataset_summary.pkl
Let's verify if the new dataset is valid.
3 scenarios are in the new dataset.
The new mapping:
{'sd_training.tfrecord-00000-of-01000_2a1e44d405a6833f.pkl': '../../../metadrive/assets/waymo', 'sd_training.tfrecord-00000-of-01000_c403d5992cab9e0.pkl': '../../../metadrive/assets/waymo', 'sd_training.tfrecord-00000-of-01000_8a346109094cd5aa.pkl': '../../../metadrive/assets/waymo'}