Virtual Control

A virtual control is a logical control by defining a certain interval of an interface control as a logical one. It can perform the following functions.

• Precise clicking
• Calling text recognition (OCR) methods to get text content or manipulate text areas by means of images.
• Screen capture or perform image comparison.

• More fine-grained control. For example, some applications have self-drawing interfaces where some buttons are non-standard controls that cannot be added to the model by recognizing objects. One way to manipulate it is to calculate the mouse click position. Clicking on it is controlled with parameters of position coordinates. But this method of clicking has the following drawbacks.

  1. The code is poorly readable

  2. When the form is stretchable, the effective click coordinate position changes.

     The above disadvantages can be avoided by using virtual controls.

The following is an example of how to define virtual controls in the Model Manager. For a description of the API for virtual controls, see Virtual Control API。

Defining virtual controls

In the Model Manager, virtual controls can be created in two ways. On the toolbar of the control screenshot page, you can select.

1. "Create Virtual Control" and then draw the virtual control directly on the control screenshot by dragging and dropping the mouse.
2. "Advanced editing of virtual controls", open the virtual control editing window and edit virtual controls in bulk.

The following sample demonstrates the creation of a virtual control. The sample animation application appchooser in QT has the following interface.

The image on the left is the interface when it is opened. When one of the icons is clicked, the icon will zoom in and show in the center.

To automate this application, first spy and add the form control to the model, then right click the Panel object in the model, select "Edit Virtual Control", which will open the Virtual Control dialog:

Then perform the following steps：

1. Add 4 virtual controls，name them as Camera、Glass、Ball、Book
2. Select the corresponding area on the image for these controls
3. Choose where to align. For example, the Book control selects the bottom right alignment, so that when the form is stretched, the virtual control still clicks on the icon in the corresponding position in the lower right corner.
4. Click "OK" to save the definitions for these virtual controls.

After finish editing the virtual controls：

You can see：

• The position of the virtual control in the parent control (Panel)
• Operations on the virtual control, which includes click、dblClick、wheel。

After generating code for the virtual control:

//Node.js code:
model.getVirtual("Camera").click();

Can see that the code is quite readable.

For a description of the API of the virtual control, see Virtual Controls API