Submitting to the Deepdrive leaderboard

Overview

First-off we'll get you on the leaderboard with the default bot, so you can get familiar with the submission process. The Deepdrive leaderboard uses Botleague to evaluate submissions. To place an agent on the leaderboards, you just need to submit a bot.json file to the league repo via pull request.

Est. time (5-15 minutes depending on your internet connection)

Step 1: Fork botleague

Login to your GitHub account and fork the botleague repo with the fork button on the top right.

fork botleague

Step 2: Clone your fork

git clone https://github.com/<YOUR-GITHUB-NAME>/botleague
cd botleague

Step 3: Create a bot.json

Create a bots/<YOUR-GITHUB-NAME>/forward-agent/bot.json in your fork with the following JSON.

{ 
  "docker_tag": "crizcraig/forward-agent",
  "problems": ["deepdrive/unprotected_left"] 
}

NOTE: Here crizcraig/forward-agent is the default docker image for the forward-agent bot. Later on, when you modify your bot, you will replace this docker tag with a repo you have push access to.

Step 4: Commit and push your bot.json

git commit -am 'forward-agent'
git push origin master

Step 5: Click Pull Request on your repo's page

click pull request

Step 6: Create your pull request

create the pull request yay

Step 7: Confirm that botleague has started your evaluation

Your pull request status should update to something similar to the following

pull request status update

Step 8: Wait for your bots evaluation to complete

Grab a coffee! This will take 5-10 minutes.

Step 9: Check that your pull request is merged

Once your evaluation is complete, it will be automatically merged as displayed in the image below. You may need to refresh the page. If you see an error, contact [email protected], or file a GitHub issue. We are automatically notified of these errors as well.

click

Step 10: Go to the leaderboard!

Finally, check the leaderboards to see your bot's score and video ranked among the others.

Next steps

Local development of your bot

To test your bot locally, it's ideal to run the sim and agent on your local machine as in our examples. You can see what your bot scores locally by passing the same parameters to main.py as we do on the evaluation server excluding the --server parameter.

Next you can run the sim in server mode locally with those same parameters again, but this time, keeping --server in the params passed to main.py.

Now make sure your bot runs as a docker container against the official scenario container. For the case of unprotected_left, for example, the docker image would be deepdriveio/deepdrive:problem_unprotected_left. You can see how our scenario problem images are built and run here.

Then to build your bot container, refer to how our baseline agent bot containers are built here.

Finally, to submit your bot, create a pull request as we did above, pointing to your bot's docker image. If the image is the same, you can just add whitespace, or change some comment text to allow for the pull request.