virtualization of controls

When performing desktop automation, sometimes it is encountered that the target control cannot be identified by means of object recognition. At this time, the operation can be completed by virtualizing the entire control, that is, a virtual control is dynamically generated for the entire control at runtime, and the size and position of the virtual control are the same as the actual control. With this control, you can use the virtual control to operate this control.

Give an example to illustrate its practical application scenario. The toolbar of a certain application is self-drawn, and only the entire toolbar can be recognized by using object recognition, and each button of it is a picture containing text. In order to click a button on the toolbar, you can directly call the getVirtual() method after obtaining the object without passing any parameters, and it will return the same virtual control object as this control.

The virtual control provides the clickVirtualText() API, so we click on the text content of the button, assuming that the content on the target button is "open", then after adding the toolbar control, we can use the following code to click on it Controls:

await model.getWindow("ToolBar").getVirtual().clickVirtualText('打开');

This way, it can be used without creating a dummy control for this control in the model.