Назад в библиотеку

A Turn-based Strategy Game Testbed for Artificial

Intelligence

by Joseph H. Souto

Presented to the Graduate and Research Committee of Lehigh University in Candidacy for the Degree of Master of Science

in Computer Engineering

Lehigh University April 2007

This thesis 1s accepted and approved in partial fulfillment of the requirements for the Master of Science.

Abstract

The DARPA Transfer Learning Project is a program that intends to develop, implement, demonstrate and evaluate theories, architectures, algorithms, methods, and techniques that enable computers to apply knowledge learned for a particular, original set of tasks to achieve superior performance on new, previously unseen tasks. [Transfer Learning, 2007] Specifically, the goal is to develop and evaluate the performance of transfer learning agents in simulated environments.

The Testbed for Integrating and Evaluating Learning Techniques (TIELT) is a software tool developed by the Naval Research Lab that is used to integrate AI systems with gaming simulators, and evaluate how well those systems learn on selected tasks. [Aha, 2007]

Call to Power 2 is a turn-based strategy game based on Civilization. In Call to Power 2, the player develops a civilization and progresses through technological ages from ancient to near-future times. During the game, the player interacts with other rival civilizations and manages the resources and economy of their civilization. It is a good game to use to test intelligent agents because there are many disparate factors that much be gauged at any given time, and decisions must be made based upon these elements. Some of the decisions a player or intelligent agent must make include: where to build new cities, whether to explore or expand the civilization, how to dispose forces to attack enemy units, and which technological advances to seek.

As part of the Transfer Learning Project, intelligent agents will be tested in conjunction with Call to Power 2. To this end, Call to Power 2 needed to be integrated with TIELT to use and evaluate these agents. This integration required effort on two fronts. First, the game world and important game objects had to be modeled in TIELT. Second, an API needed to be developed that both enables simple control of the actions of the computer player in Call to Power 2, as well as allowing an interface with TIELT so that a TIELT agent can execute the functions available in the API. This thesis explains the work entailed in this process and shows an example of the end results.

1. Introduction

Transfer learning is a theory proposed by Thorndike & Woodworth (1901) that proposes that things humans approach new tasks by applying things learned in other similar situations. [Thorndike & Woodworth, 1901] A simple transfer learning example would be that when a human drives a bus for the first time, they will apply things that they have learned about driving when they drove a car.

The DARPA Transfer Learning Project is based on the belief that artificial intelligent agents can be constructed and improved by the same method. By training an intelligent agent one context, in our case a computer game, the agent should then perform better in other similar types of computer games.

One of Lehigh's tasks in the Transfer Learning Project is to enable the testing of intelligent agents in the turn-based strategy game Call to Power 2. Call to Power 2 was chosen for two reasons. First, it is a game that involves many strategic factors that an agent would have to make decisions on. Some of the many decisions a player or intelligent agent must make include: where to build new cities, whether to explore or expand the civilization, how to dispose forces to attack enemy units, and which technological advances to seek. Second, the source code for Call to Power 2 was made open-source by Activision in 2003, and is maintained by an open-source community called Apolyton [Apolyton, 2007]. This allows us to make any modifications to the game necessary to enable integration with the AI tools used by the Transfer Learning Project.

The results of this project will not only be of benefit to the DARPA Transfer Learning Project. The development of the API will be of great benefit to other AI researchers because it provides a simple interface that will allow them to construct and test intelligent agents for use in Call to Power 2 without requiring in-depth knowledge of the details of the Call to Power 2 system architecture.

There were a few difficulties involved with doing this project. First, even though Call to Power 2 has been made open-source, for legal reasons Activision had to remove all documentation from the code. This makes it rather challenging to come to an understanding of the Call to Power 2 system architecture and the design decisions that the developers made along the way. Consequently, it can be very arduous to add additional features to the API as it requires understanding some of the internal functionality of the game in order to interact with it. Additionally, TIELT is a continually evolving system to meet the needs of the Transfer Learning Project. When new versions are released by the Naval Research Lab, the work done to integrate a game with previous versions of TIELT must be converted to the new versions.

This thesis is divided as follows. Section 2 introduces the AI research tool, TIELT. Section 3 describes Call to Power 2 and our API. Section 4 explains the TIELT model of Call to Power 2. Section 5 shows a sample session of Call to Power 2 played with the completed TIELT integration. Lastly, Section 6 lists conclusions and further work possible due to these efforts.

2. TIELT

2.1 What is TIELT?

Tielt is a software tool developed by the Naval Research Lab (NRL) to facilitate

AI researchers in the evaluation of intelligent decision systems in various game engines

(e.g., real-time strategy, discrete strategy, role playing, team sports, first-person shooter).

Emphasis has been placed upon games that are applicable to military simulations, which

is part of why Call to Power 2 was selected in conjunction with the Transfer Learning

Project. [Aha, 2007]

One might ask why is TIELT necessary at all? Given that Call to Power 2 is

open-source, why not simply modify the existing AI engine for the computer player

directly? The answer to these questions is time and integration complexity. Even though

a game may be open-source, it would still be a daunting task to come to a complete

enough understanding of the system architecture to be able to modify the AI code

directly. Additionally, modifying the AI directly would make it difficult to test the

performance of more than one algorithm for an intelligent agent. Since your newly

developed code would be very tightly bound to the architecture of the game, it would

require almost a complete rewrite of the code to test a different type of agent program.

As for complexity, given that the purpose of the transfer learning project is to test

a variety of intelligent agents across a variety of simulators, for each one of x agents you

would need to perform an integration for each one of the y simulators you want to test the

agent on. This requires a total of x*y integrations. TIELT, however has the advantage of

acting as an intermediary between the simulators and the agent programs. Once a

simulator is integrated with TIELT, it does not need to know how to communicate with

any of the agent programs because it TIELT hides the details of the communication from

5

the simulator. Likewise all the agents need to know is how to communicate with TIELT; they do not need to know how the communication is performed with the simulation engine. Therefore, with the aid of TIELT, the number of integrations required to test a suite of intelligent agents is reduced from x*y to x + y. See the figures below.

Integrations necessary without TIELT

Cognitive System-i

Cognitive System_n

igure 1: Worst-case number of integrations. Taken from: [Aha, 2004]

Integrations necessary with TIELT

Cognitive System1

Cognitive Systemn

Figure 2: Best-case number of integrations. Taken from: [Aha, 2004] 2.2 TIELT's Components

By acting as this intermediary between the simulation engine and the cognitive system, the task of evaluating a transfer learning agent becomes broken down into a series of much simpler components. The following diagram shows the resulting system in Figure 3.

Figure 3: TIELT System Architecture. Taken from: [Aha, 2007]

Before the experiments can begin, one must first build the game model for the simulation engine in TIELT. This can be seen at the bottom of figure 3 above in which the user of TIELT creates the game model, game interface model, decision system interface model, etc. These components will be explained in greater detail in Section 2.3.

Once TIELT has been integrated, experiments can be performed. As seen in Figure 3, on the leftmost end of the system, we have the game that we're testing our agent with. The game sends information about the changing state of the game world to the TIELT module. Through TIELT's user interface, the user of TIELT can observe this information about changes to the state of the world. When these messages are received, internal state information within TIELT is then updated according to rules set forth by the

TIELT user. These messages about these changes to the game state are also passed on to the decision system that has been integrated with TIELT, as seen in Figure 3 on the right side. The decision system uses these sensors combined with the current state information within TIELT's knowledge bases to evaluate what is happening within the game world. Once it makes a decision on which action to perform, it sends message(s) with the proposed action(s) back to TIELT. These actions can be used in one of two ways. The TIELT user can observe these proposed actions, and either use the agent's proposed actions as advice system to play the game in a more informed manner. Or the agent can control the actions of the player directly by sending these action messages through TIELT to the simulation engine. The TIELT user can also observe the actions that the decision system has performed in order to monitor the quality of the decisions the agent program has made.

It can be seen therefore, that TIELT makes it very easy to test and compare the performance of a large variety of intelligent agents in a given simulation engine. To test the performance of agent program Y instead of agent program X, all that is required is to have a TIELT integration with each agent program. Then, given a suite of tests in the game environment, all which is required is to run these tests with each different agent program hooked up at the other end of TIELT.

To get a more in-depth view of how games are played in conjunction with TIELT, consider the following example situation for Call to Power 2. Suppose you have a Settler, and want to settle a new city. Assume that you have queried the game engine about whether a city can be build on the tile that unit is currently on (more specifics on how this task is done will be provided later in Section 4). TIELT receives a response message, CityBuildable, indicating that a city can be built on that tile. The Learning Translator TIELT module sees that the game engine has indicated that a city can built on that tile and decides to tell the game engine to have that Settler build a city. In order to perform this action, it creates a message, BuildNewCity, which contains parameters indicating that this Settler should build a city on its tile. After it has been created, the Learning Translator module sends this BuildNewCity message to the Decision System. Once the Decision System has received the BuildNewCity message, it sends it to the Action Translator TIELT module to convert this desire to settle a new city into an output message to be sent to the game engine. This whole process is summarized in Figure 4 below.

Figure 4: TIELT Decision System Architecture. Taken from: [Aha, 2007]

After the decision has been made, and the desired action has been selected, the next step is for the Action Translator TIELT module to convert the action to a message that will tell the simulator what to do. When the Action Translator receives the BuildNewCity decision from the Decision Sytem, it determines that it is associated with a particular operator for interfacing with the simulation engine. In this case, it associates the BuildNewCity decision with an action model called BuildCity and populates the parameters of the BuildCity operator with information from BuildNewCity and the current state of the game as kept in TIELT. The BuildCity action model is shown to the TIELT user, and is then using information from the Game Interface Model module, TIELT associates the BuildCity action model with a message, BUILD CITY, which when populated with the appropriate parameters is sent out on a socket to the game in order to perform the action of building a city. Finally the game receives the BUILDCITY message and has the Settler create a new city. This set of actions is shown in Figure 5 below.

Figure 5: Actions in TIELT. Taken from: [Aha, 2007]

After the BUILD_CITY message has been received by the game engine, the Settler will build a new city. Since the creation of a new city changes the game state, once this action has been completed, the game engine will send a message, CITYSETTLED, back to TIELT indicating where the new city was built. The Game Interface Model will see the CITY SETTLED message and translate it to its corresponding Observation Model, CitySettled. The parameters of this observation model are then used, along with the rules in the Game Model, to determine the manner in which TIELT's model of the game state should be modified when CitySettled has been received. After the current state has been updated by the Model Updater TIELT module, it informs the Controller that a CitySettled event has been perceived, and passes this information on to the user and the agent program. This process can be seen below in Figure 6.

Figure 6: Sensing Game State in TIELT. Taken from: [Aha, 2007]

2.3 Sample TIELT Domain

With all that in mind about the architecture of TIELT, now we can discuss how to

model the rules of a game within TIELT. We will do this by example. We will show how a simple domain is set up by giving a brief overview of how to model a well known game, Chess, in TIELT.

The first step with TIELT is to set up the Environment Model. Here we define a few rules about what type of game we wish to model and how it works.

Field: 2D Board » Time: Turn-I

Accessibility: Inaccessibile

Determinism: ® Deterministic

Transition Knowledge: Incomplete Rules Competitiveness: Single Player

Cooperativeness: Team-Oriented

Figure 7: TIELT Environment Properties

Chess is a game played on a 2D board in which two opposing players take turns making their moves. At any given time, the entire game world is completely visible to either of the players (accessible). The next state of the game is determined entirely by the actions of one of the players (deterministic). All the rules of the game are known (complete). And only one of the two players can win the game (Zero Sum, Non-Team-Oriented).

After defining the rules of the game, the next step is to define the state variables. These are variables which represent data about the current state of the game world. When TIELT perceives a change in the state of the game world, the state variables will be modified in order to reflect this change in the game world. Examples of state variables one would want to use in Chess include a data structure to represent the board, the number of pieces on the board, and a representation of each different type of piece used in the game. These, among other sample state variables are shown in Figure 8 below.

Figure 8: TIELT State Variables

Corollary to specifying the state variables is defining various classes to be used in TIELT. Classes in TIELT are much like C++ classes in that they have members, that can be either primitive data types or objects, and functions which operate upon those member variables. An example of a class used for Chess is the Piece class. The Piece class contains such information as which type of piece something is, what player it belongs to, and where it resides on the board among other information. The member variables of the Piece class are shown in Figure 9.

Classes

Piece

Extends:

pawn	Boolean *	=	false		:1
bishop	Boolean t	-	false		t
knight	Boolean »	=	false		t
rook	Boolean *	=	false		t
queen	Boolean	-	false		t
king	Boolean »	=	false		t
white	Boolean *	-	true		t
position	Location				t
exists	Boolean *	=	false		t
hasMoved	Boolean *	-	false		t
					г

Figure 9: TIELT Classes

The Piece class also has one member function, called getLetter(), which is used to

return a single letter representing the type of piece a Piece object represents. This member function is used by other functions to get a quick reference to the types of pieces being dealt with. Part of the definition of the getLetter() function is shown in Figure 10.

Methods:

Add	Delete
•

getLetterQ : String

Code:

declare 1Lower :	String;	A
if (рамп.)
set lbомег =	"P";
endif;
if (bishop)
set lLower -	rrjyr.
endif;
if (knight)
set 1Lower =	,rn,r.
endif;
if (rook)
set lLower =	ГГ ГР . L f
endif;
if (queen)

Figure 10: TIELT Class Member Definition

After these aspects of the state are defined, the next step is to specify the Action Models. As shown earlier, an Action Model is a TIELT representation of an action or set of actions possible in the game world. One such action model needed for Chess is something to represent moving a piece on the game board. This action model can be defined simply as movePiece(), which takes arguments defining what piece is to be moved, and what location on the board it is to be moved to. Figure 11 shows how the movePiece() message is declared in TIELT.

Figure 11: TIELT Action Model

As seen, the Action Model is defined by its name, the parameters it is called with, what state of the game it can be used in, whether there are any specific preconditions restricting the use of that action ("Conditions"), whether the action will change TIELT's game state ("State Changes"), and if this action will move the game into a new phase.

It will also be helpful to define some helper functions to perform logical operations. These functions will be useful for things like determining if a move can be made, and what location on the board a moved piece will be at as a result of a move. An example of a helper function is shown below in Figure 12.

Name: isLegalRookMove

Return Type: Boolean

Parameters:

	pRook	Piece T		t	A
	lDest	Location		t
		w		+

Code:

declare iBegin : Integer;

declare iEnd : Integer;

declare bsSguare : BoardSguare;

if (eguals(lDest.flie, pRook.position.file))

if (lessThan(lDest.row, pRook.position.row) j set iBegin = lDest.row + 1; set iEnd = pRook.position.row - 1; endif;

if (le ssThan(pRook.position.row, IDest.row) | set iBegin = pRook.position.row + 1; set iEnd = lDest.row - 1; endif;

for nine = iBegin to iEnd

set bsSguare = board[nInc][lDest.file]; if (bsSguare.isOccupied)

return false; endif:

Figure 12: TIELT Helper Function

Here we see an example helper function for determining if a rook's move is valid.

Given as input a piece object referring to a rook, and a location object referring to the

rook's intended destination, isLegalRookMove() will return true if the move is valid for

the rook and false if the move is invalid. Functions like these are used in various places to assist in deciding what to do, as we shall see shortly.

After defining all the necessary Action Models, we will now take a step away from the Environment Model for the moment, and begin setting up the Simulator Interface Model. The first thing to do is define how TIELT will communicate with the game engine. The simplest way to do this is via a TCP/IP connection on a socket. A format specifying precisely how messages and their associated parameters will be sent will also be defined here.

Chess.xml: Action Models x Chess.xml: Observation Models x CTP_verl,xml: Communication x

Method:

Act as a server

In the formatting boxes below, the following substitutions apply: %mn - message name %pn - parameter name %pv - parameter value

Spaces are important, If you include a space in the spec, TIELT will expect it in the message, and the message will not be read properly without it. However, spaces will be tolerated between values in messages even if not present in the spec. In order to put spaces in string values, please specify string delimiters.

This header will be used once per message.

%mn(

This will be repeated for every parameter in the message,

%pv

This will separate parameters,

This closer will be used once per message,

This message separator will be used to determine whether a message is complete,

\r\n

This format will be used for empty messages (with no parameters). A second format cannot be validated without a message separator,

%mn

These string delimiters can be used to enclose strings with arbitrary punctuation,

Quotes may be nested:

Example: foo(1.0,false,"Hello World!")

If necessary, provide formats for transmission of arrays and objects.

Object Format ...

Figure 13: TIELT Communication Properties

As seen above, messages to the game engine have been defined to begin with the name of the message, followed by the set of comma-delimited parameter values in parenthesis. Messages can be defined in many ways, but this format is simple, and intuitively looks similar to function calls in a programming language.

The next step is to define the messages that will be sent to the game engine in order to cause things to happen within the game. To continue with our example, we will define a message to move a piece on the board, which we will call move(). This message will be triggered whenever TIELT invokes the movePiece() action model, and thus it will have access to movePiece's parameters, p and lTo. The definition of move is shown in Figure 14.

= Outgoing Simulator Message Entry

Message Name:	move	Trigger movePiece
Parameters
MOVE	String T		t
	ж		t

Parameter Creation Code

set MOVE = getAlgebraicHotation(p, lTo) ; call tryAlgebraicMove(MOVE); if (eguals(curState.sToMove, "White"))

set curState.sToMove = "Black"; endif;

if (equals(curState.sToMove, "Black"))

set curState.sToMove = "White"; endif;

Validate Drop in Code ?

Cancel

Figure 14: TIELT Outgoing Simulator Message

Here we see that move will send a string to the game engine to represent what piece it wants to move to which location. In order to set up this string, it uses some of the helper functions mentioned earlier in order to convert the TIELT objects for p and lTo into a string type. Then it checks which player made this move in order to update the game state to reflect the player whose turn will be next after this move is made.

After all these outgoing messages have been defined, we must define messages coming in from the game engine to TIELT. These messages are sent from the simulator whenever there is a change in the game world. In the Chess example, one such incoming

message would be a message representing the end of the opposing player's turn, which we will call go(). Upon receipt of the go() message, the game state will be updated to reflect that it is now our turn. The definition of the go() message is shown below in Figure 15.

Figure 15: TIELT Incoming Simulator Message

After all of the incoming messages have been defined, the final step is to complete the environment model by setting up the Observation Models. Observation Models represent our ability to sense changes in the game world. One such observation

we will want to sense is associated with the go() message from Figure 15. As seen in the model change code for go(), it triggers the OpponentMoves observation model. When this happens, we will want to update the game state in order to reflect the fact that it is now our turn. This can be seen in Figure 16 below, in that when OpponentMoves is triggered it begins the phase "MyTurn".

Figure 16: TIELT Observation Model

Once all the sensors are defined by the Observation Models, we have completed our TIELT interface with the desired game engine.

3. Call To Power 2

3.1 What is Call to Power 2?

Call to Power 2 is a turn-based strategy game based on Civilization. In Call to

Power 2, the player develops a civilization and progresses through technological ages from ancient to near-future times. During the game, the player interacts with other rival civilizations and manages the resources and economy of their civilization. Victory is attained by either military conquest of all rival civilizations, or by reaching world peace through diplomatic means. It is a good game to use to test intelligent agents because there are many disparate factors that much be gauged at any given time, and decisions must be made based upon these elements. Where to build new cities, whether to explore or expand the civilization, how to dispose forces to attack enemy units, and which technological advances to seek are but a few of the many decisions to be made by a player or an intelligent agent

3.2 How to Play Call to Power 2

A game of Call to Power 2 begins with the player in control of two Settler units.

Settlers are non-military units that build new cities. At the start of the game, the player chooses where to build to their initial cities. Things to consider when choosing where to place a new city are the type of terrain it is on, the amount of food and commerce that tile can provide, and whether there are any tradable goods in the vicinity of that tile. See Figure 17 below for a view of this situation at the start of the game.

Figure 17: Call to Power 2 Interface

Here we see the portion of the map we start the game in. As seen, the board is divided into square tiles, and is viewed from overhead. Tiles that have not yet been explored are displayed in black. There is one Settler built a city on a plains tile. Within the bounds of that city (dashed white line) is a tradable good, tobacco. The other Settler we started the game with is on a forest tile near another tradable good, beavers. In the bottom left is a minimap displaying the explored portion of the map in relation to the rest of the unexplored tiles. In the bottom right is a control panel allowing access to the information about various aspects of your civilization.

Once a city has been built, units can be produced within that city. Military units are one example of a type of unit that can be built in the city. In the figure above, we can

see that a Warrior is being built in Rome, and will be completed in 9 turns. The production of units in a city is controlled via the Build Manager window for each city. An example of the Build Manager window can be seen below in Figure 18. .

Figure 18: Call to Power 2 Build Manager

The Build Manger shows all the units that can be built at that point (more become available as the civilization advances to new technological ages), and has a build queue which allows multiple units to be built sequentially after the previous unit in the queue has completed.

Units are not the only type of thing that can be built in a city. Another type of thing that can be produced is a city improvement. City improvements do not appear on the map, but are stored internally in the state of each particular city. City improvements provide various types of benefits when built, including increased commerce or production, and additional defense against attacking enemy units. An example of a city improvement can be seen below in Figure 19 where we see that building a Granary would

provide a bonus to the production of food within this city. Another type of thing that cities can produce is a Wonder. Wonders are accomplishments that take a long time to build and provide a bonus to your entire empire upon their completion. Examples of Wonders include the Great Wall of China, which removes all barbarians from your empire, and Galileo's Telescope, which increases the rate at which new science is learned.

Figure 19: Call to Power 2 City Improvements

Many other things can be managed within each particular city. See Figure 20 below for a sample of some of the other many aspects of a city. The details of what each specific thing in Figure 20 represents are beyond the scope of what was done for this project, but they are shown to provide an idea of the amount of micromanagement available in the game. It should be noted that the game manages the growth of a city automatically unless the player tweaks the parameters for its management. Additionally, a "Mayor" can be assigned to the city, as seen in Figure 20, which automatically

optimizes the resources of a city towards a particular goal (e.g.: Production, Science, etc.).

Figure 20: Call to Power 2 City Manager

As the game progresses, the player eventually comes into contact with rival civilizations. Figure 21 shows a screenshot from later in the game after a few military units have been produced and more of the map has been explored. As seen, two cities belonging to a rival civilization have been discovered, Swansea and Cardiff, indicated by the orange number next to them.

Figure 21: Call to Power 2 - Contact with Rival Civilizations

Call to Power 2 offers three methods of obtaining victory: diplomacy, scientific advancement, and military conquest. Diplomacy involves offering resources or scientific advances to the other civilization in exchange for eventually attempting to forge treaties and eventually a military alliance with that civilization. Victory through science entails attaining each scientific advancement in the tech tree, and then building the Solaris Project wonder. Diplomacy and scientific advancement are rather complex to deal with from an agent's perspective, and is thus beyond the scope of this project, so it will not be discussed further. Military conquest involves destroying all existing enemy units and cities with your military units. In the bottom right corner of the figure above you can see

the actions available with the highlighted Warrior unit. Military units have the ability to attack enemy units or enemy cities.

When an attack command is issued, combat with the defending enemy units is initiated. An example of a combat window is shown in Figure 22 below. There we see the highlighted Warrior from the previous picture attacking the city adjacent to it, which is defended by an enemy Hoplite unit. Combat is conducted by each unit striking the other simultaneously and doing an amount of damage within a variable range according to which type of unit it is. Additionally, there are bonuses given to the defender when they are garrisoning a city. Combat continues until either the attacking player decides to retreat, or when one of the player's units is wiped out.

Cities I Units I Diplomacy I Stats I Options I

W ItjiI

Figure 22: Call to Power 2 Combat

3.3 Call to Power 2 API

The previous section showed a few of the many complexities involved in playing

a game of Call to Power 2. Since the entire game extremely complex, we decided that an agent should only be able to invoke a subset of all the possible actions available in the game. It was decided to focus on the development of cities, units, and city improvements, and attaining victory through military conquest.

To that end, the source code for Call to Power 2 had to be modified to allow an

outside program to control the actions of the computer player. The source code for Call

to Power 2 was made open-source by Activision, and is maintained by an online

community called Apolyton. [Apolyton, 2007] Ushhan's Gudevia used this source code

to develop a basic API to control the computer player for his Master's thesis in 2006

[cite: Ushan's thesis]. That API allowed for the use of a few basic control commands,

however it was not enough to allow for automatic control by a TIELT-enabled agent

program. It was not able to provide any information as to whether a given command was

successful or not, and what the results of the action were. For example, you could tell a

Warrior to attack an enemy unit, but you could not determine if the combat was won or

lost. Also, that API had only a very limited ability to communicate with TIELT. Other

miscellaneous modifications were required as well, such as disabling modal dialogue

windows that pop up to display information during the game, so that an agent could play

through an entire session of the game without requiring human intervention. Therefore,

this project required adding to the API to allow a TIELT-enabled agent to play a

complete subset of the game with the functions available through the API, as well as

enabling a complete integration with TIELT so that an agent could both query the status

of aspects of the game and sense any spontaneous changes to the state of the game (e.g.: having one of your units killed on an opponent's turn), and making a few miscellaneous optimizations to assist in the integration process. The functions currently available in the API are as follows:

• MoveArmyTo: Instruct an army to move to a specified location

• Settle: Instruct a Settler to build a city on its current location

• CityBuild: Instructs one of your cities to build a specified type of unit

• CityImprove: Instructs a city to start building a city improvement of the specified

^typ^e.

• AttackEnemyPosWithArmy: Instructs an army to attack an enemy-occupied location.

• AttackCityPosWithArmy: Instructs an army to attack a location where there is an enemy-controlled city.

• ArmyToDefend: Instruct an army to fortify the city at its current location.

• StopDefending: Instructs an army to stop fortifying its current location

• FindEnemyUnit: Searches for enemy units visible from one of your own units.

• FindEnemyCity: Searches for enemy cities visible from one of your own units.

• FindUnexplored: Searches for the closest unexplored territory an army can reach.

• ArmyCanReach: Determines whether or not there is a known path from an army to a location.

• QueryCityBuildable: Queries if a unit can build a city at its current location.

• QueryMoveable: Queries if a unit can move to a given location on the map.

• QueryUnitBuildable: Queries if a unit can be produced by a given city you control

• QueryImprovementBuildable: Queries if an improvement can be built in a given city you control.

• QueryGarrison: Queries if a unit can garrison its current location.

• QueryUngarrison: Queries if a unit can stop garrisoning its current location

• QueryUnitAttackable: Queries if a unit can attack an enemy unit on a given location

• QueryCityAttackable: Queries if a unit can attack an enemy city on a given location

The sensor messages added to inform TIELT of game state changes are as follows:

• An attack on an enemy city has succeeded

• An attack on an enemy unit has succeeded

• One of your units has been destroyed

• One of your cities has been destroyed

• A new city has been built

• A new unit has finished building

• Updates on the position of your units at the start of every turn

• Call To Power 2 has been started

• A game has started

• Your turn has started

• The game has ended

• Game has been lost

• Game has been won

• Call to Power 2 has been closed

Another thing worth mentioning is the abstraction provided by the API. Call to Power 2 uses a complex system of interrelated classes and objects to control different aspects of the game. It can become difficult to come to an understanding of this architecture in order to modify the game, both because of the complexity of the system architecture and because all the comments were removed from the source code by Activision. To simplify this problem, the API defines a few different classes in order to collect aspects of these internal objects into a single set of easy to understand objects. The API provides four different classes, defined as follows: API_Player

Description: This is a class that represents a one of the players in the game Members:

• int milndex: this is a signed 32-bit integer which represents a single player, it refers to an index into a global array of players kept by the game

• Player *player: this is a pointer into Call to Power 2's internal Player class

Methods:

• Constructors:

o API_Player( const int p_iIndex ) o API_Player( const API_Player & p_pOther )

• Destructor:

o ~API_Player(): destructor

• Accessors:

o int GetIndex()

6. Conclusions

6.1 Summary

This thesis has detailed the work require to integrate Call to Power 2 with TIELT. A model of the rules of the game world was first defined within TIELT, and the desired actions and percepts an agent can take to play a complete subset of the game were specified. Then the existing Call to Power 2 API was modified and enhanced in order to both allow an outside agent to play a complete subset of the game and to integrate communications with TIELT. The work done for this project will provide an excellent test bed for future experiments in the Transfer Learning Project. With the completion of this thesis, intelligent agents can now be evaluated in conjunction with Call to Power 2. The next step for this project will be to design, implement, and test an assortment of transfer learning agents with the fully integrated TIELT - Call to Power 2 system.

6.2 Future work

The Call to Power 2 - TIELT Integration has laid the groundwork for a multitude of later projects. As stated, we now have a test bed for evaluating the performance of intelligent agents in conjunction with the DARPA Transfer Learning Project. Our next milestone with this project is to construct agents that perform reasoning upon the current state of the world, as modeled in TIELT, in order to evaluate an assortment of transfer learning algorithms. Additionally, as our API and TIELT specifications are all freely available, any AI researcher or hobbyist can make use of any part of this project for their own endeavors.

Bibliographical References

Aha, D. (2007) Testbed for Integrating and Evaluating Learning Techniques. URL: http://www.tielt.org/presentations/TIELT Project Overview (17 Nov 2004).ppt. Last checked: 4/16/07

Apolyton (2007).

URL: http://apolyton.net/forums/forumdisplay.php?s=&forumid=2131. Last checked: 4/16/07

E. L. Thorndike & R. S. Woodworth (1901). The Influence OF Improvement in one Mental Function upon The Efficiency of other functions (I). Psychological Review, 8, 247-261. URL: http://psychclassics.yorku.ca/Thorndike/Transfer/transfer1.htm. Last checked: 4/16/07

Gudevia, U. (2006). Integrating War Game Simulations with AI Testbeds: Integrating Call To Power 2 with Tielt. Computer Science and Engineering. Lehigh University.

Transfer Learning (2006). URL: http://www.darpa.mil/baa/BAA05-29.html. Last checked: 4/16/07.

Vita

Joseph Henry Souto was born in Morristown, NJ to Joseph and Pamela Souto. He attended Lehigh University from the fall of 2000 through the spring of 2004, receiving a B.S. in Computer Engineering with honors. He returned to graduate school at Lehigh University in the fall of 2005 and will receive his Master's in the spring of 2007.