# Deepdrive The easiest way to experiment with self-driving AI ## Simulator requirements * Linux * Python 3.6+ * 10GB disk space * 8GB RAM ## Optional - baseline agent requirements * CUDA capable GPU (tested and developed on 970, 1070, and 1060's) * 1.7 <= Tensorflow < 2.0 [See Tensorflow install tips](/#tensorflow-install-tips) ## Install 1. [Create a Miniconda env](/docs/miniconda) 2. Clone ``` git clone https://github.com/deepdrive/deepdrive cd deepdrive ``` > Optional - Activate the Python conda env or virtualenv with Tensorflow installed, then > > Note: If you use Anaconda (we recommend Miniconda) - only use `pip install` in your deepdrive conda environment, never `conda install` 1. Install ``` python install.py # Do not run as sudo! Use Miniconda or virtualenv to install without sudo. ``` #### Cloud [Cloud setup instructions](/docs/tutorial/cloud/cloud) ### Examples #### Forward-agent ``` python example.py ``` #### Synchronous forward-agent ``` python example_sync.py ``` * [Remote agent example](https://github.com/deepdrive/forward-agent) - operates over the network using the [deepdrive remote api](https://github.com/deepdrive/deepdrive-api) #### Mnet2 baseline agent ``` python main.py --mnet2-baseline --experiment my-baseline-test ``` #### Built-in C++ [FSM](https://github.com/deepdrive/deepdrive-sim/tree/c2d26a38692f1db61d48986263b20721ab136fe3/Plugins/DeepDrivePlugin/Source/DeepDrivePlugin/Private/Simulation/Agent/Controllers/LocalAI/States) / [PID](https://github.com/deepdrive/deepdrive-sim/blob/v3/Plugins/DeepDrivePlugin/Source/DeepDrivePlugin/Private/Simulation/Agent/Controllers/DeepDriveAgentSteeringController.cpp) agent that can overtake in the canyons map ``` python main.py --path-follower --experiment my-path-follower-test ``` #### Record training data for imitation learning / behavioral cloning ``` python main.py --record --jitter-actions --sync ``` Note that we recorded the baseline dataset in sync mode which is much slower than async mode. Async mode *probably* is fine to record in, we just haven't got around to trying it out for v3. Optional: Convert to HDF5 files to tfrecords (for training MNET2) ``` python main.py --hdf5-2-tfrecord ``` #### Train on recorded data ``` python main.py --train [--agent dagger|dagger_mobilenet_v2|bootstrapped_ppo2] --recording-dir ``` #### Train on our dataset Grab the [dataset](/#dataset) ``` python main.py --train --recording-dir [--agent dagger|dagger_mobilenet_v2|bootstrapped_ppo2] ``` #### Tensorboard ``` tensorboard --logdir="/tensorflow" ``` Where `` below is by default in `$HOME/Deepdrive` and can be configured in `$HOME/.deepdrive/deepdrive_dir` #### Running unit tests ``` pytest tests/unit_tests/test_sanity.py ``` ### Key binds * Esc - Pause (Quit in Unreal Editor) * Enter - Pause with no menu * P - Pause in Unreal Editor * 1 - Chase cam * 2 - Orbit (side) cam * 3 - Hood cam * 4 - Free cam (use WASD to fly) * Space - Handbrake * Alt+Tab - Control other windows / Show mouse * \` - Unreal console - do things like `stat FPS` * M - Drive the car with the keyboard WASD - be sure sync is off - Also known issue: Only works in path-follower mode right now * Ctrl-number - Change sun position - works for 1 => 7 * B - Show vehicle bounding boxes * N - Show vehicle collision boxes * Page Up - Next vehicle * Page Down - Prev vehicle ## Observation data All values returned in the observation keep Unreal conventions, specifically * All distances are in centimeters per Unreal's default data type * All rotations / angular values are in the order of roll, pitch, yaw in degrees * x,y,z is forward, right, up ``` { 'acceleration': array([-264.26913452, -227.578125 , 105.16122437]), 'angular_acceleration': array([210980.234375, 105423.765625, 38187.28125 ]), 'angular_velocity': array([2.59908962, 3.8214705 , 1.87282801]), 'brake': 0.0, 'camera_count': 1, 'cameras': [{ 'aspect_ratio': 1.0, 'capture_height': 227, 'capture_width': 227, 'depth_data': array([0.9995 , 0.9995 , 0.9995 , ..., 0.005146, 0.005146, 0.005146], dtype=float16), 'horizontal_field_of_view': 1.7654, 'id': 1, 'image': array([[[ 40., 78., 110.] ..., dtype=float32), 'image_data': array([0.283 , 0.557 , 0.82, ..., 0.02321, 0.02574, 0.02599], dtype=float16), 'image_raw': array([[[144, 195, 233]..., dtype=uint8), 'type': 0 }], 'capture_timestamp': 4132.511303506, 'dimension': array([514.99609375, 514.99609375, 91.1796875 ]), # Vehicle dimensions 'distance_along_route': 70658.828125, # centimeters of progress made along route to destination 'distance_to_center_of_lane': 1038.8463134765625, # centimeters to center of lane 'world': { 'vehicle_positions': [ [ -15800.8193359375, 38030.23828125, 19894.62890625], [ -13854.9384765625, 39296.91015625, 20041.6484375], [ -10323.2744140625, 39767.69921875, 20409.265625], [ -6528.05810546875, 38875.75390625, 21034.83984375], [ 4577.29150390625, 36155.37890625, 22704.166015625]]}, 'distance_to_next_agent': 326125.625, # Next agent in our lane 'distance_to_next_opposing_agent': -1.0, # Next agent in opposite lane 'distance_to_prev_agent': 30758.2734375, # Next agent in our lane 'forward_vector': array([-0.8840133 , -0.4375411 , -0.16455328]), 'gym_action': [0, 1, 0, 0, True], 'gym_done': False, 'gym_reward': -2.4653405387152016, 'handbrake': 0, 'is_game_driving': 0, 'is_passing': 0, 'is_resetting': 205, 'lap_number': 0, 'last_collision': { 'collidee_velocity': array([0., 0., 0.]), 'collision_location': 'rear_right_fender', 'collision_normal': array([0., 0., 0.]), 'time_since_last_collision': 0.0, 'time_stamp': 4105.741911045, 'time_utc': 1562958070}, 'position': array([-10163.55371094, 17115.17382812, 22500.29492188]), 'right_vector': array([-0.8840133 , -0.4375411 , -0.16455328]), 'rotation': array([ 0.10010731, -0.16530512, -2.68199444]), 'route_length': 273551.21875, 'episode_return': { 'avg_kph': 0, 'closest_vehicle_cm': 15812.662932649602, 'closest_vehicle_cm_while_at_least_4kph': 15812.662932649602, 'cm_along_route': 18730.72265625, 'collided_with_vehicle': False, 'collided_with_non_actor': True, 'end_time': '1969-12-31T16:00:00-08:00', 'episode_time': 11.5, 'gforce_penalty': 90.68476390028613, 'got_stuck': False, 'lane_deviation_penalty': 255.7695629358121, 'max_gforce': 0.8785649610557551, 'max_kph': 138.7572978515625, 'max_lane_deviation_cm': 1038.8463134765625, 'num_steps': 0, 'prev_progress_pct': 6.70844576752594, 'progress_pct': 6.8472451856879175, 'progress_reward': 0.0, 'route_length_cm': 273551.21875, 'speed_reward': 371.6081579415893, 'start_time': '2019-07-12T12:00:59.003417-07:00', 'time_penalty': 0.0, 'total': 25.15383110549117, 'wrong_way': False}, 'speed': 3854.369384765625, 'steering': 0.0, 'throttle': 1.0, 'up_vector': array([-0.8840133 , -0.4375411 , -0.16455328]), 'velocity': array([-3404.32958984, -1700.12841797, -613.90289307]), 'view_mode': 'normal', } ``` Additional observation data can be exposed without compiling C++ or Blueprints by accessing the Unreal API with [UnrealEnginePython](https://docs.deepdrive.io/v/v3/docs/tutorial/uepy/uepy). ## Benchmark Agents are automatically graded via [Botleague](https://deepdrive.voyage.auto/leaderboard) ## Dataset 100GB (8.2 hours of driving) of camera, depth, steering, throttle, and brake of an 'oracle' path following agent. We rotate between three different cameras: normal, wide, and semi-truck - with random camera intrisic/extrinsic perturbations at the beginning of each episode (lap). This boosted performance on the benchmark by 3x. We also use DAgger to collect course correction data as in previous versions of Deepdrive. 1. Get the [AWS CLI](https://github.com/aws/aws-cli) 2. Ensure you have 104GB of free space 3. Download our dataset of mixed Windows (Unreal PIE + Unreal packaged) and Linux + variable camera and corrective action recordings (generated with `--record`) ``` cd aws s3 sync s3://deepdrive/data/baseline_tfrecords . ``` or for the legacy HDF5 files for training AlexNet ``` aws s3 sync s3://deepdrive/data/baseline . ``` If you'd like to check out our Tensorboard training session, you can download the 1GB [tfevents files here](https://d1y4edi1yk5yok.cloudfront.net/tensorflow/mnet2_baseline_training_and_eval.zip), unzip, and run ``` tensorboard --logdir ``` and checkout [this view](http://localhost:6006/#scalars&_smoothingWeight=0.935\&runSelectionState=eyIyMDE4LTA3LTE5X18wNS01My0yN1BNIjp0cnVlLCIyMDE4LTA3LTE5X18wNS01MC01NFBNIjp0cnVlfQ%3D%3D&_ignoreYOutliers=false\&tagFilter=error) , which graphs wall time. ## Architecture ![Deepdrive Architecture](/files/-LtpYyY6cmldudqPGoxA) ## Frame rate issues on Linux If you experience low frame rates on Linux, you may need to install NVIDIA’s display drivers including their OpenGL drivers. We recommend installing these with CUDA which bundles the version you will need to run the baseline agent. Also, make sure to [plugin your laptop](https://help.ubuntu.com/community/PowerManagement/ReducedPower). If CUDA is installed, skip to testing [OpenGL](/#opengl). ## Tensorflow install tips * Make sure to install the CUDA / cuDNN major and minor version the Tensorflow instructions specify. i.e. CUDA 9.0 / cuDNN 7.3 for Tensorflow 1.12.0. These will likely be older than the latest version NVIDIA offers. You can see all [CUDA releases here](https://developer.nvidia.com/cuda-toolkit-archive). * Use the packaged install, i.e. deb\[local] on Ubuntu, referred to in [this guide](http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html) * If you are feeling dangerous and use the runfile method, be sure to follow [NVIDIA’s instructions](http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html) on how to disable the Nouveau drivers if you're on Ubuntu. * On Windows, use standard (non-CUDA packaged) display drivers which meet the min required. When installing CUDA, do a custom install and uncheck the display driver install. ## OpenGL `glxinfo | grep OpenGL` should return something like: ``` OpenGL vendor string: NVIDIA Corporation OpenGL renderer string: GeForce GTX 980/PCIe/SSE2 OpenGL core profile version string: 4.5.0 NVIDIA 384.90 OpenGL core profile shading language version string: 4.50 NVIDIA OpenGL core profile context flags: (none) OpenGL core profile profile mask: core profile OpenGL core profile extensions: OpenGL version string: 4.5.0 NVIDIA 384.90 OpenGL shading language version string: 4.50 NVIDIA OpenGL context flags: (none) OpenGL profile mask: (none) OpenGL extensions: OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 384.90 OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20 OpenGL ES profile extensions: ``` You may need to disable secure boot in your BIOS in order for NVIDIA’s OpenGL and tools like nvidia-smi to work. This is not Deepdrive specific, but rather a general requirement of Ubuntu’s NVIDIA drivers. ## Development To run tests in PyCharm, go to File | Settings | Tools | Python Integrated Tools and change the default test runner to `pytest`. Also, disable SciView per [this answer](https://stackoverflow.com/a/48421532/134077). # vendor # openai # README Extracted from OpenAI [baselines @ 1f8a03](https://github.com/openai/baselines/tree/1f8a03f3a62367526f20215188fb5ea4b9ec27e0) # PPO2 * Original paper: * Baselines blog post: * `python -m baselines.ppo2.run_atari` runs the algorithm for 40M frames = 10M timesteps on an Atari game. See help (`-h`) for more options. * `python -m baselines.ppo2.run_mujoco` runs the algorithm for 1M frames on a Mujoco environment. # A2C * Original paper: * Baselines blog post: * `python -m baselines.a2c.run_atari` runs the algorithm for 40M frames = 10M timesteps on an Atari game. See help (`-h`) for more options. # tensorflow # Surgically extracted MobileNetV2 from tensorflow/models @ 84da970ee43c04fbd53a1db3c824ea32cec8936b Datasets kept for running imagenet example TensorRT kept because meh, maaaaybeee we'll use it? # TensorFlow Research Models This folder contains machine learning models implemented by researchers in [TensorFlow](https://tensorflow.org). The models are maintained by their respective authors. To propose a model for inclusion, please submit a pull request. Currently, the models are compatible with TensorFlow 1.0 or later. If you are running TensorFlow 0.12 or earlier, please [upgrade your installation](https://www.tensorflow.org/install). ## Models * [adversarial\_crypto](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/adversarial_crypto/README.md): protecting communications with adversarial neural cryptography. * [adversarial\_text](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/adversarial_text/README.md): semi-supervised sequence learning with adversarial training. * [attention\_ocr](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/attention_ocr/README.md): a model for real-world image text extraction. * [audioset](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/audioset/README.md): Models and supporting code for use with [AudioSet](http://g.co/audioset). * [autoencoder](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/autoencoder/README.md): various autoencoders. * [brain\_coder](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/brain_coder/README.md): Program synthesis with reinforcement learning. * [cognitive\_mapping\_and\_planning](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/cognitive_mapping_and_planning/README.md): implementation of a spatial memory based mapping and planning architecture for visual navigation. * [compression](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/compression/README.md): compressing and decompressing images using a pre-trained Residual GRU network. * [deeplab](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/deeplab/README.md): deep labelling for semantic image segmentation. * [delf](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/delf/README.md): deep local features for image matching and retrieval. * [differential\_privacy](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/differential_privacy/README.md): differential privacy for training data. * [domain\_adaptation](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/domain_adaptation/README.md): domain separation networks. * [gan](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/gan/README.md): generative adversarial networks. * [im2txt](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/im2txt/README.md): image-to-text neural network for image captioning. * [inception](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/inception/README.md): deep convolutional networks for computer vision. * [learning\_to\_remember\_rare\_events](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/learning_to_remember_rare_events/README.md): a large-scale life-long memory module for use in deep learning. * [learning\_unsupervised\_learning](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/learning_unsupervised_learning/README.md): a meta-learned unsupervised learning update rule. * [lexnet\_nc](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/lexnet_nc/README.md): a distributed model for noun compound relationship classification. * [lfads](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/lfads/README.md): sequential variational autoencoder for analyzing neuroscience data. * [lm\_1b](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/lm_1b/README.md): language modeling on the one billion word benchmark. * [maskgan](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/maskgan/README.md): text generation with GANs. * [namignizer](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/namignizer/README.md): recognize and generate names. * [neural\_gpu](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/neural_gpu/README.md): highly parallel neural computer. * [neural\_programmer](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/neural_programmer/README.md): neural network augmented with logic and mathematic operations. * [next\_frame\_prediction](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/next_frame_prediction/README.md): probabilistic future frame synthesis via cross convolutional networks. * [object\_detection](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/object_detection/README.md): localizing and identifying multiple objects in a single image. * [pcl\_rl](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/pcl_rl/README.md): code for several reinforcement learning algorithms, including Path Consistency Learning. * [ptn](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/ptn/README.md): perspective transformer nets for 3D object reconstruction. * [qa\_kg](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/qa_kg/README.md): module networks for question answering on knowledge graphs. * [real\_nvp](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/real_nvp/README.md): density estimation using real-valued non-volume preserving (real NVP) transformations. * [rebar](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/rebar/README.md): low-variance, unbiased gradient estimates for discrete latent variable models. * [resnet](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/resnet/README.md): deep and wide residual networks. * [skip\_thoughts](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/skip_thoughts/README.md): recurrent neural network sentence-to-vector encoder. * [slim](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/slim/README.md): image classification models in TF-Slim. * [street](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/street/README.md): identify the name of a street (in France) from an image using a Deep RNN. * [swivel](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/swivel/README.md): the Swivel algorithm for generating word embeddings. * [syntaxnet](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/syntaxnet/README.md): neural models of natural language syntax. * [tcn](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/tcn/README.md): Self-supervised representation learning from multi-view video. * [textsum](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/textsum/README.md): sequence-to-sequence with attention model for text summarization. * [transformer](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/transformer/README.md): spatial transformer network, which allows the spatial manipulation of data within the network. * [video\_prediction](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/video_prediction/README.md): predicting future video frames with neural advection. # slim # nets # README ## MobileNetV2 This folder contains building code for MobileNetV2, based on [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://arxiv.org/abs/1801.04381) ## Performance ### Latency This is the timing of [MobileNetV1](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/slim/nets/mobilenet_v1.md) vs MobileNetV2 using TF-Lite on the large core of Pixel 1 phone. ![mnet\_v1\_vs\_v2\_pixel1\_latency.png](/files/-LtpYyUzyWZRyXLWqhv7) ### MACs MACs, also sometimes known as MADDs - the number of multiply-accumulates needed to compute an inference on a single image is a common metric to measure the efficiency of the model. Below is the graph comparing V2 vs a few selected networks. The size of each blob represents the number of parameters. Note for [ShuffleNet](https://arxiv.org/abs/1707.01083) there are no published size numbers. We estimate it to be comparable to MobileNetV2 numbers. ![madds\_top1\_accuracy](/files/-LtpYyV0Q-SRAg1zRN93) ## Pretrained models ### Imagenet Checkpoints | Classification Checkpoint | MACs (M) | Parameters (M) | Top 1 Accuracy | Top 5 Accuracy | Mobile CPU (ms) Pixel 1 | | ------------------------------------------------------------------------------------------------------------- | -------- | -------------- | -------------- | -------------- | ------------------------ | | [mobilenet\_v2\_1.4\_224](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.4_224.tgz) | 582 | 6.06 | 75.0 | 92.5 | 138.0 | | [mobilenet\_v2\_1.3\_224](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.3_224.tgz) | 509 | 5.34 | 74.4 | 92.1 | 123.0 | | [mobilenet\_v2\_1.0\_224](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.0_224.tgz) | 300 | 3.47 | 71.8 | 91.0 | 73.8 | | [mobilenet\_v2\_1.0\_192](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.0_192.tgz) | 221 | 3.47 | 70.7 | 90.1 | 55.1 | | [mobilenet\_v2\_1.0\_160](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.0_160.tgz) | 154 | 3.47 | 68.8 | 89.0 | 40.2 | | [mobilenet\_v2\_1.0\_128](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.0_128.tgz) | 99 | 3.47 | 65.3 | 86.9 | 27.6 | | [mobilenet\_v2\_1.0\_96](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.0_96.tgz) | 56 | 3.47 | 60.3 | 83.2 | 17.6 | | [mobilenet\_v2\_0.75\_224](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.75_224.tgz) | 209 | 2.61 | 69.8 | 89.6 | 55.8 | | [mobilenet\_v2\_0.75\_192](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.75_192.tgz) | 153 | 2.61 | 68.7 | 88.9 | 41.6 | | [mobilenet\_v2\_0.75\_160](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.75_160.tgz) | 107 | 2.61 | 66.4 | 87.3 | 30.4 | | [mobilenet\_v2\_0.75\_128](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.75_128.tgz) | 69 | 2.61 | 63.2 | 85.3 | 21.9 | | [mobilenet\_v2\_0.75\_96](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.75_96.tgz) | 39 | 2.61 | 58.8 | 81.6 | 14.2 | | [mobilenet\_v2\_0.5\_224](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.5_224.tgz) | 97 | 1.95 | 65.4 | 86.4 | 28.7 | | [mobilenet\_v2\_0.5\_192](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.5_192.tgz) | 71 | 1.95 | 63.9 | 85.4 | 21.1 | | [mobilenet\_v2\_0.5\_160](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.5_160.tgz) | 50 | 1.95 | 61.0 | 83.2 | 14.9 | | [mobilenet\_v2\_0.5\_128](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.5_128.tgz) | 32 | 1.95 | 57.7 | 80.8 | 9.9 | | [mobilenet\_v2\_0.5\_96](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.5_96.tgz) | 18 | 1.95 | 51.2 | 75.8 | 6.4 | | [mobilenet\_v2\_0.35\_224](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.35_224.tgz) | 59 | 1.66 | 60.3 | 82.9 | 19.7 | | [mobilenet\_v2\_0.35\_192](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.35_192.tgz) | 43 | 1.66 | 58.2 | 81.2 | 14.6 | | [mobilenet\_v2\_0.35\_160](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.35_160.tgz) | 30 | 1.66 | 55.7 | 79.1 | 10.5 | | [mobilenet\_v2\_0.35\_128](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.35_128.tgz) | 20 | 1.66 | 50.8 | 75.0 | 6.9 | | [mobilenet\_v2\_0.35\_96](https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_0.35_96.tgz) | 11 | 1.66 | 45.5 | 70.4 | 4.5 | ## Example See this [ipython notebook](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/vendor/tensorflow/models/research/slim/nets/mobilenet/mobilenet_example.ipynb) or open and run the network directly in [Colaboratory](https://colab.research.google.com/github/tensorflow/models/blob/master/research/slim/nets/mobilenet/mobilenet_example.ipynb). # Running the TensorFlow Official ResNet with TensorRT [TensorRT](https://developer.nvidia.com/tensorrt) is NVIDIA's inference optimizer for deep learning. Briefly, TensorRT rewrites parts of the execution graph to allow for faster prediction times. Here we provide a sample script that can: 1. Convert a TensorFlow SavedModel to a Frozen Graph. 2. Load a Frozen Graph for inference. 3. Time inference loops using the native TensorFlow graph. 4. Time inference loops using FP32, FP16, or INT8 precision modes from TensorRT. We provide some results below, as well as instructions for running this script. ## How to Run This Script ### Step 1: Install Prerequisites 1. [Install TensorFlow.](https://www.tensorflow.org/install/) 2. [Install TensorRT.](http://docs.nvidia.com/deeplearning/sdk/tensorrt-install-guide/index.html) 3. We use the image processing functions from the [Official version of ResNet](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/official/resnet/imagenet_preprocessing.py). Please checkout the Models repository if you haven't already, and add the Official Models to your Python path: ``` git clone https://github.com/tensorflow/models.git export PYTHONPATH="$PYTHONPATH:/path/to/models" ``` ### Step 2: Get a model to test The provided script runs with the [Official version of ResNet trained with ImageNet data](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/official/resnet/README.md), but can be used for other models as well, as long as you have a SavedModel or a Frozen Graph. You can download the ResNetv2-ImageNet [SavedModel](http://download.tensorflow.org/models/official/resnetv2_imagenet_savedmodel.tar.gz) or [Frozen Graph](http://download.tensorflow.org/models/official/resnetv2_imagenet_frozen_graph.pb), or, if you want to train the model yourself, pass `--export_dir` to the Official ResNet [imagenet\_main.py](https://github.com/deepdrive/deepdrive/tree/c64371bd9b652dbaaa84f868a7e2ed025dce2cd9/official/resnet/imagenet_main.py). When running this script, you can pass in a SavedModel directory containing the Protobuf MetaGraphDef and variables directory to `savedmodel_dir`, or pass in a Protobuf frozen graph file directly to `frozen_graph`. If you downloaded the SavedModel linked above, note that you should untar it before passing in to the script. ### Step 3: Get an image to test The script can accept a JPEG image file to use for predictions. If none is provided, random data will be generated. We provide a sample `image.jpg` here which can be passed in with the `--image_file` flag. ### Step 4: Run the model You have TensorFlow, TensorRT, a graph def, and a picture. Now it's time to time. For the full set of possible parameters, you can run `python tensorrt.py --help`. Assuming you used the files provided above, you would run: ``` python tensorrt.py --frozen_graph=resnetv2_imagenet_frozen_graph.pb \ --image_file=image.jpg --native --fp32 --fp16 --int8 --output_dir=/my/output ``` This will print the predictions for each of the precision modes that were run (native, which is the native precision of the model passed in, as well as the TensorRT version of the graph at precisions of fp32, fp16 and int8): ``` INFO:tensorflow:Starting timing. INFO:tensorflow:Timing loop done! Predictions: Precision: native [u'seashore, coast, seacoast, sea-coast', u'promontory, headland, head, foreland', u'breakwater, groin, groyne, mole, bulwark, seawall, jetty', u'lakeside, lakeshore', u'grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus'] Precision: FP32 [u'seashore, coast, seacoast, sea-coast', u'promontory, headland, head, foreland', u'breakwater, groin, groyne, mole, bulwark, seawall, jetty', u'lakeside, lakeshore', u'sandbar, sand bar'] Precision: FP16 [u'seashore, coast, seacoast, sea-coast', u'promontory, headland, head, foreland', u'breakwater, groin, groyne, mole, bulwark, seawall, jetty', u'lakeside, lakeshore', u'sandbar, sand bar'] Precision: INT8 [u'seashore, coast, seacoast, sea-coast', u'promontory, headland, head, foreland', u'breakwater, groin, groyne, mole, bulwark, seawall, jetty', u'grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus', u'lakeside, lakeshore'] ``` The script will generate or append to a file in the output\_dir, `log.txt`, which includes the timing information for each of the models: ``` ========================== network: native_resnetv2_imagenet_frozen_graph.pb, batchsize 128, steps 3 fps median: 930.2, mean: 934.9, uncertainty: 5.9, jitter: 3.3 latency median: 0.13760, mean: 0.13692, 99th_p: 0.13793, 99th_uncertainty: 0.00271 ========================== network: tftrt_fp32_resnetv2_imagenet_frozen_graph.pb, batchsize 128, steps 3 fps median: 1160.2, mean: 1171.8, uncertainty: 18.0, jitter: 17.8 latency median: 0.11033, mean: 0.10928, 99th_p: 0.11146, 99th_uncertainty: 0.00110 ========================== network: tftrt_fp16_resnetv2_imagenet_frozen_graph.pb, batchsize 128, steps 3 fps median: 2007.2, mean: 2007.4, uncertainty: 0.4, jitter: 0.7 latency median: 0.06377, mean: 0.06376, 99th_p: 0.06378, 99th_uncertainty: 0.00001 ========================== network: tftrt_int8_resnetv2_imagenet_frozen_graph.pb, batchsize 128, steps 3 fps median: 2970.2, mean: 2971.4, uncertainty: 109.8, jitter: 279.4 latency median: 0.04309, mean: 0.04320, 99th_p: 0.04595, 99th_uncertainty: 0.00286 ``` The script will also output the GraphDefs used for each of the modes run, for future use and inspection: ``` ls /my/output log.txt tftrt_fp16_resnetv2_imagenet_frozen_graph.pb tftrt_fp32_resnetv2_imagenet_frozen_graph.pb tftrt_int8_calib_resnetv2_imagenet_frozen_graph.pb tftrt_int8_resnetv2_imagenet_frozen_graph.pb ``` ## Troubleshooting and Notes ### GPU/Precision Compatibility Not all GPUs support the ops required for all precisions. For example, P100s cannot currently run INT8 precision. ### Label Offsets Some ResNet models represent 1000 categories, and some represent all 1001, with the 0th category being "background". The models provided are of the latter type. If you are using a different model and find that your predictions seem slightly off, try passing in the `--ids_are_one_indexed` arg, which adjusts the label alignment for models with only 1000 categories. ## Model Links [ResNet-v2-ImageNet Frozen Graph](http://download.tensorflow.org/models/official/resnetv2_imagenet_frozen_graph.pb) [ResNet-v2-ImageNet SavedModel](http://download.tensorflow.org/models/official/resnetv2_imagenet_savedmodel.tar.gz) [ResNet-v1-ImageNet Frozen Graph](http://download.tensorflow.org/models/official/resnetv1_imagenet_frozen_graph.pb) [ResNet-v1-ImageNet SavedModel](http://download.tensorflow.org/models/official/resnetv1_imagenet_savedmodel.tar.gz) # Deepdrive The MIT License Copyright (c) 2019 Deepdrive, Inc () Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ## Kevindale Map The Kevindale map, which is part of the simulator binaries downloaded by this project, is licensed under [this EULA](https://docs.google.com/document/d/1HvAsslRt8wqf3_xOhamfnzC44Ufkwc1PEAAlz80rpxM). # docs # tutorial # cloud # Cloud setup We've tested on Paperspace's ML-in-a-Box Linux public template with a P6000 which already has Tensorflow installed and just requires To set it up choose your region, then go to Public Templates, and choose the ML-in-a-Box template ![Paperspace](https://i.imgur.com/ZyltYsM.png) ## Miniconda Then download the 64-bit Linux installer from [Miniconda](https://docs.conda.io/en/latest/miniconda.html) or click [here](https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh) Now run the Miniconda installer ``` bash Miniconda3-latest-Linux-x86_64.sh ``` Next, create a conda env for deepdrive ``` bash # to get conda in your shell conda create -n dd python=3.7 conda activate dd ``` ## Clone Now clone our repo ``` git clone https://github.com/deepdrive/deepdrive ``` ## Run And finally run the install ``` cd deepdrive python install.py ``` You can validate everything is working by running ``` python example.py ``` If you run into issues, try starting the sim directly as Unreal may need to install some prerequisetes (i.e. DirectX needs to be installed on the Paperspace Parsec Windows box). The default location of the Unreal sim binary is in your user directory under `Deepdrive/deepdrive-sim*/DeepDrive/Binaries/Linux/DeepDrive`, i.e. ``` /home/YOURUSERNAME/Deepdrive/deepdrive-sim-linux-3.1.20191112232556/DeepDrive/Binaries/Linux/DeepDrive ``` You can continue testing things out by running the rest of our examples [here](https://docs.deepdrive.io/#examples) # uepy # UnrealEnginePython Tutorial The following tutorial demonstrates the power of UnrealEnginePython to manipulate objects in the running game with Python, i.e. `import unreal_engine`. For more tutorials on how to use UnrealEnginePython (UEPy) see [their docs](https://github.com/deepdrive/UnrealEnginePython/blob/master/README.md). ## Hello world Here, we'll show how to print to the Unreal Logs, get the ego vehicle (your car), and move the ego vehicle around. Find your sim binaries by opening up `~/Deepdrive` and sorting by date ![Find sim bin](/files/-LtpYzULcI9Pg-lQHXFt) Drill down to the Deepdrive binary ![Drill down](/files/-LtpYzUNOXmNgTSDzuhC) Open a terminal, drag the file into the terminal, and press enter to open the sim in the terminal and see the logs. ![Terminal open sim](/files/-LtpYzUPV_quXkLf1RSY) Once the sim is open and you see the, press `M` to drive the car manually. Within your binary folder, i.e. something like `deepdrive-sim-linux-3.0.20190528000032` open your `DeepDrive/Content/Scripts` in your favorite editor and create a new Python file named `move_car_tutorial.py` and enter: ```python print('hello world') ``` ![Editor open](/files/-LtpYzURxB34Jsq712i4) Now hit the backtick `` ` `` key to open the Unreal Console and enter ``` py.exec move_car_tutorial.py ``` ![Unreal Console Hello World](/files/-LtpYzUTqPtdPUI_w9fu) You should then see "hello world" printed to the logs in the terminal: ![Terminal Hello World](/files/-LtpYzUV4UIKIVc8sX9C) ## Get and set the ego's position Now you know how to run Python within Unreal Engine. Let's do something more interesting! Paste the following into `move_car_tutorial.py` ```python import json from api_methods import Api api = Api() ego = api.get_ego_agent() location = ego.get_actor_location() ego.set_actor_location(location.x, location.y, location.z + 1e4) # +100m ``` Now open the Unreal Console again in the simulator with `` ` `` and hit the up arrow to rerun the previous command which should be ``` py.exec move_car_tutorial.py ``` Happy flying!!! ## Introspecting the Unreal API Often when developing a UEPy script, you won't know the exact name of the object you want to manipulate. Some general methods for introspecting the games internal state are demonstrated with the following script ```python import json from api_methods import Api, best_effort_serialize api = Api() ego = api.get_ego_agent() print('Methods -------------------------------') for m in dir(ego): print(m) print('Properties --------------------------- ') print(json.dumps(best_effort_serialize(ego, levels=5), indent=2)) ``` [Example output](https://gist.githubusercontent.com/crizCraig/b9f9f86dc404a5658a85328e490d585e/raw/111e2e717d06ccd928683a782d7a70009a785a62/gistfile1.txt) Here you can see the wealth of functionality and information UnrealEnginePython provides, including the `get_actor_location` and `set_actor_location` methods we've just demonstrated. Imagine learning the game's state information with the [input remapping trick](https://arxiv.org/abs/1504.00702)! # leaderboard # Submitting to the Deepdrive leaderboard ## Overview First-off we'll get you on the leaderboard with the default bot, so you can get familiar with the submission process. The Deepdrive leaderboard uses [Botleague](https://github.com/botleague/botleague) to evaluate submissions. To place an agent on the leaderboards, you just need to submit a bot.json file to the league repo via pull request. Est. time (5-15 minutes depending on your internet connection) ## Step 1: Fork botleague Login to your GitHub account and fork the [botleague](https://github.com/botleague/botleague) repo with the fork button on the top right. ![fork botleague](https://i.imgur.com/tgesEjc.jpg) ## Step 2: Clone your fork ``` git clone https://github.com//botleague cd botleague ``` ## Step 3: Create a bot.json Create a `bots//forward-agent/bot.json` in your fork with the following JSON. ``` { "docker_tag": "crizcraig/forward-agent", "problems": ["deepdrive/unprotected_left"] } ``` > NOTE: Here `crizcraig/forward-agent` is the default docker image for the forward-agent bot. Later on, when you modify your bot, you will replace this docker tag with a repo you have push access to. ## Step 4: Commit and push your bot.json ``` git commit -am 'forward-agent' git push origin master ``` ## Step 5: Click Pull Request on your repo's page ![click pull request](https://i.imgur.com/DsFddJQ.jpg) ## Step 6: Create your pull request ![create the pull request yay](https://i.imgur.com/C6o2r4V.jpg) ## Step 7: Confirm that botleague has started your evaluation Your pull request status should update to something similar to the following ![pull request status update](https://i.imgur.com/bimSaQW.png) ## Step 8: Wait for your bots evaluation to complete Grab a coffee! This will take 5-10 minutes. ## Step 9: Check that your pull request is merged Once your evaluation is complete, it will be automatically merged as displayed in the image below. You may need to refresh the page. If you see an error, contact , or file a GitHub issue. We are automatically notified of these errors as well. ![click ](https://i.imgur.com/6nffqfl.jpg) ## Step 10: Go to the leaderboard! Finally, check the [leaderboards](https://deepdrive.voyage.auto/leaderboard) to see your bot's score and video ranked among the others. ## Next steps ### Local development of your bot To test your bot locally, it's ideal to run the sim and agent on your local machine as in our [examples](https://docs.deepdrive.io/#examples). You can see what your bot scores locally by passing the [same parameters](https://github.com/deepdrive/deepdrive/blob/f93e1091cdd9e393fd5516eedbf85e19e380773c/botleague/problems/unprotected_left/run.sh#L10) to `main.py` as we do on the evaluation server excluding the `--server` parameter. Next you can run the sim in server mode locally with those [same parameters](https://github.com/deepdrive/deepdrive/blob/f93e1091cdd9e393fd5516eedbf85e19e380773c/botleague/problems/unprotected_left/run.sh#L10) again, but this time, keeping `--server` in the params passed to `main.py`. Now make sure your bot runs as a docker container against the official scenario container. For the case of `unprotected_left`, for example, the docker image would be `deepdriveio/deepdrive:problem_unprotected_left`. You can see how our scenario problem images are built and run [here](https://github.com/deepdrive/deepdrive/tree/e565f52794c1d18904f1b2fc7c79a05e8629ed46/botleague/problems). Then to build your bot container, refer to how our baseline agent bot containers are built [here](https://github.com/deepdrive/deepdrive/tree/e565f52794c1d18904f1b2fc7c79a05e8629ed46/botleague/bots). Finally, to submit your bot, create a pull request as we did above, pointing to your bot's docker image. If the image is the same, you can just add whitespace, or change some comment text to allow for the pull request. # Miniconda Then download the 64-bit Linux installer from [Miniconda](https://docs.conda.io/en/latest/miniconda.html) or click [here](https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh) Now run the Miniconda installer ``` bash Miniconda3-latest-Linux-x86_64.sh ``` Next, create a conda env for deepdrive ``` bash # to get conda in your shell conda create -n dd python=3.7 conda activate dd ``` # README Make sure you build the base image using the Dockerfile in the root of this repo if you want to incorporate new changes.