Virtual Control API

Virtual control creation

Virtual controls can be created in several ways:

1. Define in the model:

You can create virtual controls under a Windows control object in the object model. For related information, see Virtual Controls

1. Descriptive programming

   Create a virtual control by specifying location information in the code.

2. The getVirtual() method without any parameters

   For more details, see virtualization of controls

   You can call the getVirtual() method of an object directly in the code without any parameters, and it will return a virtual control object. The virtual control area is the same size as the object's corresponding control. This approach is usually used to do image recognition or OCR operations on a control.

   For example, for the Win10 calculator, you can call:

   JavaScript

   let numPad = model.getGeneric("Number pad").getVirtual();
      await numPad.clickVisualText('4');

   This example obtains a virtual control from the numeric panel control of the calculator application and identifies the number "4" on it through OCR to directly click on the number button. Note: For other Windows versions of the Calculator application, the code may be different.

Methods and Properties

Virtual control has the following APIs:

• click
• clickVirtualText
• dblClick
• exists
• highlight
• moveMouse
• pressKeys
• rect
• takeSnapshot
• visualText
• wheel

Among them, the methods click, dblClick, and wheel work the same as other controls. The takeSnapshot method only intercepts the screenshot of the virtual control drawing area.

visualText

visualText method has the following signature:

visualText(): Promise<string>;

It will recognize all the text information on the screenshot image of the control area and convert it into text return.

Here is a example:

const { WinAuto } = require("leanpro.win");
let model = WinAuto.loadModel(__dirname + "test.tmodel");

(async () => {
   let text = await model.getVirtual("Virtual1").visualText();
   console.log('text', text);
})()

This example uses the "Virtual1" virtual control in the model and recognize text from it.

If there are multiple blocks in a region that need to be identified separately, you can define multiple virtual controls and then call visualText() on these controls.

clickVisualText

The clickVisualText method only exists in the Virtual control. It use OCR to find the specified text on the virtual control snapshot and click on the corresponding text.

clickVisualText(text: string, options?: ClickVisualTextOptions): Promise<void>;

among them,

• "text" is a word or phrase that can be found on the target image, the text should be on the same line on the image.
• "options" are optional parameters, including "cache", "x", "y", etc.:
• "cache" sets whether to cache text recognition information, the default value is "false"
• "x", "y" sets the click area, for example "{x: 1, y: 1}" click on the upper left corner of the text area. If x and y equal to the default value 0, click on the center of the text box.

When the cache is set to true, the text and location information recognized by the snapshot will be cached in the control. The next time you call the clickVisualText method and pass in the cache: true, the text position information will be reused to perform the operation without having to do OCR recognition again, which can speed up execution. The following is an example of using cache.

This example uses the computer that comes with Windows as an example to identify the digital panel of the calculator and add it to the model with the object name "Number pad".

(async function () {
let numPad = model.getGeneric("Number pad").getVirtual();
   await numPad.clickVisualText('4', { cache: true });
   await numPad.clickVisualText('5', { cache: true });
   await numPad.clickVisualText('6', { cache: true });
})();

An exception is thrown when the text content is not found on the given control. To troubleshot the character recognition problem, you can first call takeScreenshot method to take a screenshot of the control, then call getTextLocations method on the Ocr class to get all the recognized text and text positions, and then diagnose, as shown in the following example:

(async function () {
    let numPad = model.getGeneric("Number pad").getVirtual();
    let snapshot = await numPad.takeScreenshot();
    let blocks = await Ocr.getTextLocations(snapshot);
    console.log(JSON.stringify(blocks, null, 2))
})();

This example prints out the identification information for the text on all controls, including text and location.

For more information on OCR, see OCR (Optical Character Recognition)