Support for Vector Envs

[cpnota]

No description provided.

[benblack769]

@cpnota So I worked on this quite a bit, but after almost finishing I figured out that the Gym vector API is completely incompatible with ALL.

The problem is what happens to the observation during reset.

ALL's parallel experiment (and any sane system) expects this:

O1, r1, d1, i1
O2, r2, d2, i2
...
Ot, rt, dt, it
O1, r1, d1, i1

Unfortunately, the gym vector API is not sane, so it does this:

O1, r1, d1, i1
O2, r2, d2, i2
...
O1, rt, dt, it
O2, r2, d2, i2

Since the environments are interweaved and reset at different times (they autoreset when done), this does not work with ALL's notion of a mask on Ot.

There are a few options:

1. Ignore the problem. The agent will not train on the first action, only the 2nd and later actions.
2. I have my own implementations of vector environments in SuperSuit. So in theory, I can just make them compatible with ALL via an argument. But this means that ALL will not have perfect interoperability with gym and stable baseline's vector environments. Not clear that this is a big problem, considering how bad interoperability is between stable baselines and Gym's vector environments right now.

Thoughts?

[cpnota]

Oh I see, instead of giving a terminal "observation", it provides the final reward etc. during the first timestep of the previous episode. In their defense, I see why they did it that way (to avoid masking). But that is sort of frustrating.

Are 1. and 2. incompatible? For most of the current environments, I'm guessing that it won't make really any difference, but it could be nice to have the option. I would probably go with the minimum viable fix for now.

[benblack769]

No, the two options are perfectly compatible. I realized that after sending the last message.

[cpnota]

Closed by #240