Use Sikulix for Game Automation UI Test
Selenium and Appium can be used to solve most UI automation test so far. Cypress is based on different technology and can only be used in Web automation test. There are some applications, like following game UI, it is diffcult to locate its element thus cannot simulate operation on it.
Think how human beings conduct operation on the game. We see the whole UI at first, then on the UI, our eyes find the specific icon, then move the mouse to click it. If we can use a tool to automatically recognize the UI and find the specific icon on the UI, then conduct mouse click or keyboard operation, we achieve the goal of automation test on the game. Sikulix is such a tool.
Based on OpenCV, Sikulix can automate anything we see on the screen, either from Windows, Mac or Linux–I personally have not tried Linux yet. Next I’d like to give example how to use Sikulix to conduct automation test on a Mac desktop application. We can use either JAVA or Python. I am using JAVA here. I will not repeat the steps you can find from the official document.
Take Screenshot as Benchmark
Our first step is to tell sikulix what kind of image we expect it to recognize, and to what extend the target image shopuld match our benchmark image. By default, it would be 75%. We can adjust this similarity parameter by each image match.
Like the above example (ignore the right white part, it is my desktop background, I removed it), sometime when we make screenshot–recommend to use SikulixIDE, we also need to identify the offset information in case we need to operate on another location of the image instead of the center.
Start and Stop the Application
Typically we use command line to open and close the application in JAVA.
Runtime.getRuntime().exec(appPath);
Runtime.getRuntime().exec("killall " + appName);
Frequent Operatioon
For all kinds of operation, we need to have a Screen instance at first.
Screen screen = new Screen();
Check If an Image Exist on the Screen
public Match isImageExist(String imagePath, int timeout) {
Pattern pattern= new Pattern(imagePath));
Match match = screen.exists(pattern, timeout);
return match;
}
Check If an Image contains another Image
public Match findImageInRegion(String baseImagePath, String targetImagePath) {
Pattern pattern= new Pattern(baseImagePath));
Match match = screen.exists(pattern, 5);
if(match != null) {
Region region = new Region(match.getX(), match.getY(), match.getW(), match.getH());
return region.find(new Pattern(targetImagePath).similar(0.6f));
} else {
throw new Exception("put some meaningful message here");
}
}
Click on an Image
public void click(String imagePath, int x, int y, int timeout) {
Pattern pattern= new Pattern(imagePath).targetOffset(x, y);
Match match = screen.exists(pattern, timeout);
if (match != null) {
screen.click();
} else {
throw new Exception("put some meaningful message here");
}
}
When you see above example, you should undersatnd why we need the offset when taking screenshot.
A big disadvantage of using Sikulix, you may have realized, is to maintain the benchmark images. If there is any change on the UI, we will have to update the benchmark screenshot.