NeuroEvolving Robotic Operatives

[mind.dll]

Hello there.

I'm self-made game developer from Europe (more specifically, from Latvia) with recent but deep interest in AI aspects of machine learning and especially of RL. I think I understand the basic theory of RL in appropriate level, but I am wondering if there are answers on some questions related to using rtNEAT in games, though:

1. How could be possible to implement RL training in automated fashion (well, not all people who play games wants to train their opponents before actual playing)?

2. How could one implement training in a fashion so that physical representations of agents (game entities in form of 3d models, data they contain, etc.) remains, but only their 'brains' are evolving, changing, and generally acting as supposed by means of NEAT? It is totally valid and appropriate aspect of sufficiently specific game such as NERO, but it wouldn't in, say, traditional RPG (opponents or characters should not vanish after some time by their own, only their 'thinking skills' should change and evolve).

3. How could be implemented AI system of an agent (specifically, NPC or opponent in game) so that it (agent) starts with some skills in it's bag already (it definitely would be considered harmful if all of the game characters were starting with no skills at all, IMO), so that, e.g., there could be some archer with appropriate accuracy in aiming and predicting target's trajectory, but he would have long way ahead to learn close-combat methods or hiding from enemies? Also, IMO, it should be appropriate way for an agent to have their different initial skills at different levels (in the case of the archer, he could have above-average archery skills, average hiding/evading skills, below-average close-combat skills, and not-even-below-average arcane knowledge), so that the agent could achieve equal level of all his skills in different time for each of them.

Maybe answers on some or all of these questions are either well-known and trivial among more experienced AI/ML researchers or there are no real answers at all yet, I only hope my questions would not be considered stupid.

I would highly appreciate any advices or suggestions in case of these questions.

Thanks in advance.
_________________
Black Holes are where God divided by zero.

[Ken]

Some of the issues you raise are easier to implement with rtNEAT than others. Here are some answers:

1. There is no special difficulty in making the training automatic. It just means that the "slider" settings would be hidden from the user. That is, the game developer would ship the game with fitness functions already built in. For example, if you wanted enemies to learn to survive, then penalize them for getting hit or dying. They will evolve automatically to avoid the same mistakes they'be been making.

It should be noted that the idea of automated "learning" in a game makes more sense in my opinion as a way to keep it interesting than as a way to cause agents to become extremely smart over time. The idea is that as soon as a player starts to figure out how enemies can be defeated, the enemies will start changing their tactics. That doesn't mean they are "smarter" in a general sense; rather, they are smarter with respect to the player's current tactic. The player will then realize that his tactic has become ineffective and adjust it to exploit the new agent policy, which will once again change. The overall effect is the game stays interesting and can't be easily exploited.

2. If there is only one individual and you want it to learn on its own from experience, rtNEAT is probably the wrong choice. rtNEAT relies on a population model. Other RL techniques such as value-functions techniques do allow single individuals to learn, and these can potentially also be combined with rtNEAT. (So rtNEAT would evolve an agent which then learns further during its lifetimes using value-function RL) However, value-function RL may be too slow for a real video game. They also have some irritating side-effects such as making random moves once in a while in order to try novel policies. That may work in a research project, but in a game it can be annoying. The overall answer then is that there is no great solution to this problem that I know of.

3. It's easy enough to start with pre-evolved brains and just evolve them further. You can also define brains off of FSM models. We have a chapter in the AI Game Programming Wisdom 3 book on this topic. But the general answer is that it's easy to ship a game pretrained or predesigned genomes and let them evolve from there.

ken
_________________
NERO Concept & rtNEAT AI

[sylverone]

I know this is an old topic and that the original poster may never see this, but I would like to propose an idea concerning question #2.

I'm no programmer (not YET anyway; working on that ), but I think that rather than truly keeping the same body, the robot could "respawn" in the exact same position as it was in before, giving the illusion that it was the exact same enemy. I'm not sure about how you would work with the breeding of genetic information, but I'm pretty sure this could work if someone experienced worked on it.

Perhapse this could be accomplished by actually assigning a small "brain library" to each bot. Have each brain start out identical, then trade each one in for a certain amount of time, letting it evolve a little, then repeat this cycle a few times. Then use NEAT to pull out less sucessful brains and replace them.

Anyway, I'm going to stop since noone may ever read this anyway.
_________________
Digg NERO!

NEROvid: Community Trailer Project! Please pitch in!