Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Trueskill Evaluation for New AIs #21

Merged
merged 7 commits into from
Nov 3, 2021
Merged

Add Trueskill Evaluation for New AIs #21

merged 7 commits into from
Nov 3, 2021

Conversation

vwxyzjn
Copy link
Collaborator

@vwxyzjn vwxyzjn commented Oct 30, 2021

This PR allows the users to evaluate on newer AIs.

Bootstrap trueskills for the initial league

poetry install
cd experiments
poetry run python new_league.py --evals agent_sota.pt randomBiasedAI workerRushAI lightRushAI
=======================
             name         mu     sigma  trueskill
0   agent_sota.pt  39.300444  2.273100  32.481145
1    workerRushAI  31.810887  2.335282  24.805039
2     lightRushAI  25.776148  2.388974  18.609226
3  randomBiasedAI  11.527615  3.700987   0.424655

Add trueskills evals for additional agents

poetry run python new_league.py --evals coacAI
             name         mu     sigma  trueskill
0   agent_sota.pt  39.300444  2.273100  32.481145
1          coacAI  36.132655  1.769652  30.823699
2    workerRushAI  31.810887  2.335282  24.805039
3     lightRushAI  25.776148  2.388974  18.609226
4  randomBiasedAI  11.527615  3.700987   0.424655

Notice how the initial league's (i.e. reference agents) trueskill rating is always fixed, which corresponds to OpenAI Five and Alphastar's methods.

Evaluate Trueskill during training

Finally, there is support for evaluating Trueskill during training. This is achieved by using subprocess to call the league evaluation.

cd experiments
rm -rf dataset.db
rm -rf dataset.csv
poetry run python new_league.py --evals randomBiasedAI workerRushAI lightRushAI coacAI
poetry run python ppo_gridnet.py --prod-mode

@vwxyzjn
Copy link
Collaborator Author

vwxyzjn commented Nov 3, 2021

Synced with @kered9 and @xluox. Merging now.

@vwxyzjn vwxyzjn merged commit b54ac3d into master Nov 3, 2021
@vwxyzjn vwxyzjn deleted the new-league branch November 3, 2021 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant