In part 1 we uncovered my motivation for writing a Bejeweled Blitz bot - it's not just to get to the top of my leaderboard on Facebook! Just like maxxor, I love prototyping in Python, so that's what I used to implement the bot.

There were two things I needed immediately but didn't know how to do. One was the ability to grab screenshots of the game and process them in some way to decipher the state of the board, and the other was to be able to move the mouse cursor and do clicking and dragging. The image processing side was easy - a quick search yielded PIL, the Python Image Library. This allows me to grab an image of what is currently displayed on screen, and then examine individual pixel values. For moving the mouse cursor, I initially turned to Pygame - I've used it a lot for game prototyping, so I know it well. But the mouse module only allows you to grab the state of the mouse buttons - you can't simulate presses. Then I came across PyWinAuto, for the automation of Windows applications. It can be used to start up a program, and then simulate the key presses and mouse clicks using SendKeys as if there was a real person there.

PyWinAuto seemed very fiddly to use with Firefox. To grab a reference to the window, you have to get it by the browser window title, but as soon as you use the go to another page, the title changes and invalidates your reference to the window - as if you never actually store a handle but always grab it from Windows every time you want to interact with it. I did read somewhere that it behaves better with Microsoft based applications, but by then I had it pretty much interacting with Firefox so I didn't try it with Internet Explorer. The first actions I made PyWinAuto do were: press Ctrl-L to move focus to the address bar, type the letters for the address of the Bejeweld Blitz game and press enter, wait for the game to load and then click the mouse on the play now button. It is highly entertaining watching a computer seemingly have a mind of its own; opening a web browser, typing a long address at high speed and then interacting with a game. In fact, very occasionally it types the web address so fast that it SendKeys or Windows actually misses a key, and the bot tries to go to an invalid address. It makes it seem all the more human!

The basic flow of the bot is based on constantly evaluating all the pieces on the board, and then selecting the best move. Figuring out the state of the board is done using quite basic colour processing: it samples a square of pixels for each location on the board, averages the RGB values, translates into HSL space and then compares the value to a list of known colours. Using HSL was something Beetlefeet suggested, since it positions the glittering power gems and the multiplier gems closer to the colours of the standard gems than in RGB space (where the extra white values of the power gems affect the average badly enough for a power gem to be close to a standard gem of a different colour). The limitation of only using colours is that I have lumped power gems and multipliers in with the standard gems, whereas a more intelligent bot would be able to recognise them for their higher gameplay value, and treat them differently.

Anyway, the next optional step is to wait for gems to stop falling after a move - i.e. wait until all gems are recognised, since gems that are moving are generally not recognisable using the colour averaging method. Then, a brute force algorithm looks through the board in priority order for any single move that will achieve 5, 4 or 3 in a row. A random choice is made for each to choose whether to look horizontally or vertically first. Once a move is found, the mouse cursor is moved to the gem in question, a click is performed, then it is moved to the desired location for the second click, and then it is moved up to the top left of the window, so that it is out of the way and unlikely to affect the image processing algorithm. A game is known to last exactly 60 seconds, so the bot will time this itself and stop performing moves at the end of the game. It waits for a fixed time delay for the score screen to finish animating, and then it saves a screenshot of the score and clicks the play again button.

Since the bot is fully automated, it can be left overnight unattended and the next day I can just trawl through all the screenshots and see how well it has been performing. There are some interesting results, as can be seen in the screenshot above; even this rather unsophisticated bot is able to achieve incredibly high scores given enough time. In part 3, I will examine the results to see if they do support support the theory I suggested in part 1.

Disclaimer: Popcap takes cheating in Bejeweled Blitz very seriously. As a game developer, I respect this and I did not write this bot to affect the game experience for other players. This is why I always ran the bot in an "offline mode" - it does not play on Facebook and never uploads any scores or awarded medals. Any scores on my Facebook profile are the result of me playing without any assistance - hence I never finished a weekly tournament at the top of my leaderboard....