Multi-Agent Coordination

Our CSC Team won the Phase 2 Bakeoff Competition of the DARPA COORDINATORS program by a very large (I mean seriously seriously large) margin.

There were 640 scenarios in total: 448 were created by an independent evaluation team and 64 were created by each of the three competitors. The leads for the teams were ISI, CMU and Honeywell. In the 448 scenarios created by the independent evaluation team, the distribution of the team that got the top score was:

We were happy.

The scores from each scenario were normalized and the results were:

The ISI score is not surprising since we got the best score on almost all the scenarios. In fact, our worst normalized score was above 90%. The table above shows how much less than us the other teams actually scored (on average).

In addition, each of the three teams submitted a set of 64 scenarios to show off their capabilities:

• ISI created simple problems focused on knowing what fellow team members have done and if necessary changing activities (usually to the only alternative) based on that knowledge. Each scenario required the Coordinators to solve a collection of these problems to succeed. The normalized scores: ISI:100%, CMU:0%, Honeywell:0%.
• CMU (we believe) used the scenario generator tool used by the independent evaluation team and tuned parameters to settings where they believed they would do well. The normalized scores: ISI: 96.6%, CMU: 80.2%, Honeywell: 36.7%. We got the top score on 46/64 scenarios with CMU winning the others.
• Honeywell (which used an MDP-based approach) created problems that were essentially a collection of small independent single-agent MDPs with high rewards for following the optimal path. Not surprisingly, the normalized scores were: Honeywell: 90.4%, ISI: 32.6%, CMU: 16.9%. Surprisingly, we were able to match the top score in 15/64 problems, winning 11 outright.

Finally, the normalized scores over the entire set of problems was:

Needless to say, we were happy.