Common API Usage

The following introduces the basic operations and property methods that each control (except virtual controls) has. No matter what kind of control the operated control is (or it will be treated as a custom control Custom), some of these methods are based on simulating user operations, and some are based on the operations completed by the underlying Accessibility API provided by the system.

type definition file

These methods are all written on the base class of the control, you can click the leanpro.win library to view, the definition is as follows:

JavaScript

Python

export interface IWinControl extends IWinContainer {
    click(x?: number, y?: number, mousekey?: MouseKey): Promise<void>;
    dblClick(x?: number, y?: number, mousekey?: MouseKey): Promise<void>;
    moveMouse(x?: number, y?: number): Promise<void>
    wheel(value: number): Promise<void>;
    exists(time: number): Promise<boolean>;
    hScroll(value: number | ScrollAmount): Promise<void>;
    vScroll(value: number | ScrollAmount): Promise<void>;
    property(propertyIds: PropertyIds): Promise<string | boolean | number | Rect>;
    waitProperty(propertyIds: PropertyIds, value: string, timeoutSeconds: number): Promise<boolean>
    drop(x?: number, y?: number): Promise<void>;
    drag(x?: number, y?: number): Promise<void>;
    pressKeys(keys: string, options?: PressKeysOptions | number): Promise<void>;
    takeScreenshot(filePath?: string): Promise<void | string>;
    doDefaultAction(): Promise<void>;

    //properties
    type(): Promise<string>;
    text(): Promise<string>;
    rawText(): Promise<string>;
    name(): Promise<string>;
    hwnd(): Promise<number>;
    x(): Promise<number>;
    y(): Promise<number>;
    height(): Promise<number>;
    width(): Promise<number>;
    enabled(): Promise<boolean>;
    focused(): Promise<boolean>;
    helpText(): Promise<string>;
    labeledText(): Promise<string>;
    value(): Promise<string | number>;
    processId(): Promise<number>;
    rect(): Promise<Rect>;
    allPropertis(): Promise<object>;
    modelImage(options?: {encoding: 'buffer' | 'base64'}): Promise<Buffer | string>  //base64 is the default
    
    //advanced navigate
    firstChild(controlType?: string): Promise<IWinControl>;
    lastChild(controlType?: string): Promise<IWinControl>;
    next(controlType?: string): Promise<IWinControl>;
    previous(controlType?: string): Promise<IWinControl>;
    parent(): Promise<IWinControl>;
}

class WinControl(WinContainer):
	def click(x: Optional[int]=None, y: Optional[int]=None, mousekey: Optional[int]=None) -> None
	def dblClick(x: Optional[int]=None, y: Optional[int]=None, mousekey: Optional[int]=None) -> None
	def moveMouse(x: Optional[int]=None, y: Optional[int]=None, seconds: Optional[int]=None) -> None
	def wheel(value: int) -> None
	def exists(time: Optional[int]=None) -> bool
	def hScroll(value: int) -> None
	def vScroll(value: int) -> None
	def property(propertyds: str) -> Union[str, int, bool, Rect]
	def waitProperty(propertyds: str, value: str, timeoutSeconds: Optional[int]=None) -> bool
	def drop(x: Optional[int]=None, y: Optional[int]=None) -> None
	def drag(x: Optional[int]=None, y: Optional[int]=None) -> None
	def pressKeys(keys: str, opt: Optional[Union[int, PressKeysOptions]]=None) -> None
	def takeScreenshot(filePath: Optional[str]=None) -> Union[str, None]
	def highlight(duration: Optional[int]=None) -> None
	def type() -> str
	def text() -> str
	def name() -> str
	def hwnd() -> int
	def x() -> int
	def y() -> int
	def height() -> int
	def width() -> int
	def enabled() -> bool
	def focused() -> bool
	def helpText() -> str
	def labeledText() -> str
	def value() -> Union[str, int]
	def processId() -> int
	def rect() -> "Rect"
	def firstChild(controlType: Optional[str]=None) -> "WinControl"
	def lastChild(controlType: Optional[str]=None) -> "WinControl"
	def next(controlType: Optional[str]=None) -> "WinControl"
	def previous(controlType: Optional[str]=None) -> "WinControl"
	def parent() -> "WinControl"
	def findControls(*conditions: List[Union[str, Dict]]) -> List[WinControl]
	def modelmage(encoding: Optional[str]=None) -> Union[str, bytearray]
	def modelProperties(all: Optional[bool]=None) -> TypedDict
	def allProperties() -> TypedDict
	def doDefaultAction() -> None
	def rawText() -> str

possible class definitions

The following are some type definitions that may be used when calling public APIs.

Rect class

The object used to describe the shape and position information of the control. In CukeTest, all controls can use a rectangle (Bounding Rectangle) to describe the position and shape information. The Rect object contains the following properties:

x: number, Relative coordinates of horizontal pixels;
y: number, vertical pixels relative to coordinates;
width: number, Horizontal width, in pixels;
height: number, vertical height in pixels;

MouseKey

An enumeration class used for click() and dblClick() of the mouse, you can specify to pass in the corresponding number when clicking the mouse button used.

This enumeration class also contains some keyboard control keys, but they are usually not used.
A similar enumeration class can also be seen in the mouse operation library Mouse.

JavaScript

enum MouseKey {
    LButton = 1, // left mouse button
    RButton = 2, // right click
    MButton = 4, // Middle mouse button/wheel button
    Ctrl = 8, // Keyboard CTRL key
    Shift = 16, // Keyboard SHIFT key
    Alt = 32 // Keyboard ALT key
}

PropertyIds

JavaScript

export enum PropertyIds {
    automationId,
    name,
    text,
    url,
    title,
    handle,
    x,
    y,
    height,
    width,
    enabled,
    focused,
    helpText,
    value,
    labeledText,
    processId,
    rect,
    ...
}

In addition to the properties listed above, each control also maintains some unique property names, you can refer to the return result of allProperties() method.

ControlType

All control types that can be identified in CukeTest can be used to specify the control type in the findControls() method, or call the type() of the control The type of the result returned by the method is also in this enumeration class.

JavaScript

enum ControlType {
    Button,
    Calendar,
    CheckBox,
    ComboBox,
    Custom,
    DataGrid,
    DataItem,
    Document,
    Edit,
    Group,
    Generic,
    Header,
    HeaderItem,
    Hyperlink,
    Image,
    List,
    ListItem,
    MenuItem,
    MenuBar,
    Menu,
    Pane,
    RadioButton,
    ScrollBar,
    Slider,
    Spinner,
    Tab,
    TabItem,
    Table,
    Text,
    Thumb,
    ToolBar,
    Tree,
    TreeItem,
    Window,

    //virtual types
    TableRow,
    TreeCell,
}

Control filter condition `Criteria` class

CukeTest uses the Criteria object as a condition to filter the controls in the current environment to obtain the target control and operate it. For details, see Criteria Class.

Process startup options class `SpawnOptions`

The class used to configure process startup options, the supported properties can refer to child_processs.spawn:

cwd: string or URL, The working directory of the child process, the default value is the running project directory, which is the value of process.cwd().
env: Object, Key-value pairs of environment variables. Default: process.env.
argv0: string, Explicitly set the value of argv[0] sent to the child process. If not specified, this will be set to the command.
detached: boolean, Prepares a child process to run independently of its parent. Behavior is platform dependent, see options.detached).
uid: number, Set the user identity of the process (refer to setuid(2)).
gid: number, Set the group ID of the process (refer to setgid(2)).
serialization: string, Specifies the serialization type used to send messages between processes. Possible values are "json" and "advanced". See Advanced Serialization for details. Default:json.
shell: boolean or string If true, will start a terminal to run the command. Use '/bin/sh' on Unix, process.env.ComSpec on Windows. You can pass string type to specify the path of the terminal. Default value: false (no terminal).
windowsVerbatimArguments: boolean does not quote or escape arguments on Windows. Ignored on Unix. It is automatically set to true when shell is specified and is CMD. Default: false.
windowsHide: boolean Hide the subprocess console window normally created on Windows systems. Default: false.
timeout: number, The maximum amount of time a process is allowed to run, in milliseconds. Default: undefined.
killSignal: string or int, The signal value to use when the spawned process will be killed by a timeout or abort signal. Default: "SIGTERM".

Action method definition

click(x, y, mousekey): Promise<void>

The mouse clicks the control, and the relative position (offset) of the click and the mouse button used can be modified by passing in parameters.

x: number, The click position is relative to the horizontal pixel point of the upper left corner of the control, 0 represents the horizontal midpoint position of the control, and the default value is 0;
y: number, The click position is relative to the vertical pixel point of the upper left corner of the control, 0 represents the vertical midpoint position of the control, and the default value is 0;
mouseKey: MouseKey, The mouse button to operate, the default is 1, which is the left mouse button.

dblClick(x, y, mousekey): Promise<void>

The mouse double-clicks the control, which is consistent with the calling method of the click() method, and the relative position (offset) of the click and the mouse button used are modified by passing in parameters.

x: number, The click position is relative to the horizontal pixel point of the upper left corner of the control, 0 represents the horizontal midpoint position of the control, and the default value is 0;
y: number, The click position is relative to the vertical pixel point of the upper left corner of the control, 0 represents the vertical midpoint position of the control, and the default value is 0;
mouseKey: MouseKey, The mouse button to operate, the default is 1, which is the left mouse button.

moveMouse(x, y): Promise<void>

Move the mouse cursor to the relative position of the control, by default it moves to the center of the control.

x: number, The click position is relative to the horizontal pixel point of the upper left corner of the control, 0 represents the horizontal midpoint position of the control, and the default value is 0;
y: number, The click position is relative to the vertical pixel point of the upper left corner of the control, 0 represents the vertical midpoint position of the control, and the default value is 0;

wheel(value): Promise<void>

Roll the mouse wheel, the unit of the input parameter is the scale of the scroll wheel.

value: number, The unit is one scroll wheel scale.

exists(time): Promise<boolean>

Check whether the control exists, the input parameter is the maximum time to wait for the check (timeout). The default call will only check once without retrying and waiting.

time: number, Timeout, the unit is seconds, when it is 0, it will not retry and wait, the default parameter is 0.
returns: Asynchronously returns whether the control exists, true is present, false is not present.

hScroll(value): Promise<void>

Perform horizontal scrolling on the control, that is, Horizontal Scroll, and the parameter is the percentage of scrolling.

value: number, The scrolling percentage, 0 is scrolling to the top, 100 is scrolling to the bottom.

vScroll(value): Promise<void>

Perform vertical scrolling on the control, that is, Vertical Scroll, and the parameter is the percentage of scrolling.

value: number, The scrolling percentage, 0 is scrolling to the top, 100 is scrolling to the bottom.

property(propertyIds): Promise<string | boolean | number | Rect>

This method is quite special. Although it belongs to the operation method, it is a method to obtain the specified property, pass in the property name, and return the corresponding property value. For example, for a control, calling the method .property("rect") is actually equivalent to Instead of calling the property method .rect() directly, the values returned by both operations are always the same.

If an attribute name that does not exist is passed in, null will be returned without error.

propertyIds: propertyIds, The name of the control property.
returns: Asynchronously returns various types of property values corresponding to the property name, including string, number, boolean, Rect, etc., depending on propertyIds.

waitProperty(propertyIds, value, timeoutSeconds): Promise<boolean>

Similar to the property() method above, but it will wait until the property has a certain value, and returns indicate whether it is successful to wait until the value appears.

propertyIds: propertyIds, The name of the control property.
value: string, expected value.
timeoutSeconds: number, Timeout time, the unit is second. Wait for timeout seconds setting, the default is 10 seconds, -1 will wait infinitely.
returns: boolean, Waiting for the result, true means successfully waiting for the value, otherwise it means exceeding the waiting time. If an error is reported, it means that the control is not matched, it may be that the control does not appear, or there is a problem with the object name.

drag(x, y): Promise<void>

Click the left mouse button on the control position, which is the first step of the drag operation, and wait for the command to release the mouse. The incoming (x, y) is the coordinates of the clicked relative control, when x and yClick the center of the control when it is 0 or default at the same time.

x: (optional) number, The dragging starting point is offset by pixels relative to the horizontal coordinates of the control, negative on the left and positive on the right. The default is 0.
y: (optional) number, The dragging starting point is offset by pixels relative to the horizontal coordinates of the control, negative on the left and positive on the right. The default is 0.
returns: An asynchronous method that returns no value.

drop(x, y): Promise<void>

Release the left mouse button at the position of the control, which is the second step of the drag operation. The incoming (x, y) is the coordinates of the clicked relative control, when x and y are both 0 or the default Click the center of the control. You can execute the drag() method on the A control, and execute the drop() method on the B control to achieve the effect of dragging the A control to the B control , see Drag and Drop Method for specific documents.

x: (optional) number, The dragging starting point is offset by pixels relative to the horizontal coordinates of the control, negative on the left and positive on the right. The default is 0.
y: (optional) number, The dragging starting point is offset by pixels relative to the horizontal coordinates of the control, negative on the left and positive on the right. The default is 0.
returns: An asynchronous method that returns no value.

pressKeys(keys, options?): Promise<void>

Enter a key or string, and focus on the target control before inputting. When passing in a string, some special characters (^+~%{}()) in the string will be used as control keys (Shift key, CTRL key, etc.) to execute without inputting the corresponding symbols, for details, please refer to Appendix: Input Key Correspondence Table. If you want to input plain text and ignore these control key symbols, then You can use the {textOnly: true} option, called like this: pressKeys(str, {textOnly: true}).

keys: string, The key, key combination or string to be input, up to 1024 characters are supported.
options: (optional) Some optional parameters that control the input mode.
- textOnly: Only enter strings, and control characters are also entered as text. The effect is equivalent to calling Keyboard.typeString().
- cpm: The number of input characters per minute is used to control the text input speed. It is recommended to set the cpm value above 200 during automatic operation. Due to the internal implementation of the method, and the processing of text input by each system and application is different, the actual input speed is not necessarily the same Can reach the set cpm. When options is a number, it is equivalent to the cpm parameter.
returns: An asynchronous method that returns no value. For more related instructions or examples, please refer to Method of simulating key input pressKeys.

takeScreenshot(filePath): Promise<string>

Screenshot of the control, you can pass in the path to save the screenshot to the path location.

filePath: (optional) string, The save path and file name of the screenshot, such as"./images/screenshot1.png".
returns: Asynchronously returns the base64 string of the screenshot.

doDefaultAction(): Promise<void>

Execute the default operations defined inside the control, for example, the default operation of checkbox is switching, the default operation of input is editing, etc. The definition of these default operations can be viewed by calling the allProperties() method of the control The LegacyIAccessible.defaultAction property in .

returns: An asynchronous method that returns no value.

property method definition

It should be noted that the property methods of the control are asynchronous. After the call, CukeTest will communicate with the target control to obtain the current property of the control, that is, the real-time property, rather than the identification property in the model object. Listed below The properties of are supported by all controls, but not all have values. You can try similar property methods to get the target property according to the suggestion.

type(): Promise<string>

The type of the control, the possible values are shown in Control Type Enumeration Class.

text(): Promise<string>

The text content of the control, usually editable content, such as text input Edit, multi-line input Document and other controls have this attribute. If the returns do not meet expectations, you can try name() or value() property method.

rawText(): Promise<string>

Return the internal text information of the control. Plug-in support is required to obtain the text information rendered in the target control from the bottom layer of the system. When name(), value() and text() methods all return empty results, you can try Use the rawText() method to get the text.

This method can only get the value on the application under test launched through the "Start Windows application" menu, because the application under test launched in this way will load the corresponding plug-in.

name(): Promise<string>

The name of the control, this name is usually consistent with the identification property Name of the object in the model manager, so almost all controls have the value of this property. If the returns are not as expected, you can try text() Or the value() property method.

hwnd(): Promise<number>

The window handle to which the control belongs (Handle of WiNDow).

x(): Promise<number>

The horizontal desktop coordinate of the control, returns is number, and the unit is px.

y(): Promise<number>

The vertical desktop coordinates of the control, returns is number, and the unit is px.

height(): Promise<number>

The height of the control, returns is number, and the unit is px.

width(): Promise<number>

The width of the control, returns is number, and the unit is px.

enabled(): Promise<boolean>

Whether the control is available.

focused(): Promise<boolean>

Whether the control is focused/selected, true will be returned when the control is selected, usually used to verify whether the control is operated.

helpText(): Promise<string>

The help text of the control, that is, the content of the floating prompt message that appears when the mouse hovers over the control, is usually encountered in applications of WPF and WinForm frameworks.

value(): Promise<string | number>

The value of the control, such as sliding Slider, ScrollBar will return the corresponding value. If you want to get the name or prompt text displayed by the control, you can try the text() or value() attribute method.

processId(): Promise<number>

The process ID (Process Id) of the application to which the control belongs.

rect(): Promise<Rect>

The description rectangle of the control, returns the Rect type, contains the x, y``height and width information of the control, as shown in the definition Recttype.

allProperties(): Promise<object>

Get all runtime properties of the target control and return them in the form of objects. If you want to directly return the value of the specified property, please refer to property(propertyIds)method.

returns: Asynchronously return the object object composed of the attributes of the target model object. E.g:

JavaScript

Python

let properties = await model.getButton("Button").allProperties();
console.log(properties);
console.log('objectName=' + properties['objectName']); //Get only one of the attributes, such as objectName

properties = model.getButton("Button").allProperties()
print(properties)
print('objectName=' + properties['objectName']); #Get only one of the attributes, such as objectName

Will get all the properties of the button and print:

{
  "name": "Normal",
  "className": "CheckBox",
  "automationId": "",
  "accessKey": "",
  "enabled": true,
  "hasKeyboardFocus": false,
  ...
  "Toggle.toggleState": 0,
  "LegacyIAccessible.defaultAction": "selected",
  "LegacyIAccessible.description": "",
  "LegacyIAccessible.help": "",
  "LegacyIAccessible.keyboardShortcut": "",
  "LegacyIAccessible.name": "Normal",
  "LegacyIAccessible.role": 44,
  "LegacyIAccessible.state": 1048576,
  "LegacyIAccessible.value": ""
}

Among them, LegacyIAccessible.defaultAction is the default action of this control, if it is not empty, then the user can execute this default action by calling doDefaultAction.

CukeTest provides a series of control navigation methods, users can use navigation methods to find the parent control, child control and sibling control of the target control. Through the navigation method, a key point control can be used to lead to all the controls in the whole control tree, which is often used in scenarios such as input positioning and control traversal.

The navigation methods are all asynchronous methods, that is, it takes a certain amount of time to retrieve the navigated control in the application and return it, instead of directly navigating in the tree of the model manager, this needs to be distinguished and not confused.

firstChild(controlType?: string): Promise<IWinControl>

To get the first sub-control of the control in the application, you can specify the type of sub-control by passing in the parameter controlType, which can achieve the effect of filtering, that is, to obtain the first sub-control of controlType type in the application , controlType is a string, and the optional value can refer to ControlType. If there is no child control that meets the condition, it will return null. At the same time, child only represents direct child controls, and will not retrieve grandchild controls. If no controls satisfying the conditions are found in child controls, it will directly return null instead of continuing to search in grandchild controls. If you still want To get all the controls under a certain control, you can use the findControls method.

Note that if some controls are not visible or have not been loaded, they will not be counted. To be precise, the firstChild() method just gets the first child object that can be obtained, The lastChild() method below does the same.

controlType: (optional) string type, optional values can refer to ControlType.
returns: Asynchronous arbitrary control type, the specific type depends on the type of control navigated to. Return null when no control is navigated to.

lastChild(controlType?: string): Promise<IWinControl>

Get the last child control of the control in the application, you can specify the type of sub-control by passing in the parameter controlType, and you can achieve the filtering effect, that is, get the last sub-control of controlType type in the application, controlType is a string, optional values can refer to ControlType. If there is no sub-control that meets the condition, return null.

Note that if some controls are not visible or have not been loaded, they will not be counted. To be precise, the firstChild() method just gets the first child object that can be obtained, The lastChild() method below does the same.

controlType: (optional) string type, optional values can refer to ControlType.
returns: Asynchronous arbitrary control type, the specific type depends on the type of control navigated to. Return null when no control is navigated to.

next(controlType?: string): Promise<IWinControl>

Get the next sibling control of the application, usually the adjacent right/bottom node, depending on the layout direction of the container. You can pass in the parameter controlType to specify the type of sibling control, which can be filtered Effect, that is, to get the sibling control of the next controlType in the application, controlType is a string, and the optional value can refer to ControlType. If there is no sibling control that meets the conditions, return null, for example, null will be returned when the control is the lastChild of the parent control.

controlType: (optional) string type, optional values can refer to ControlType.
returns: Asynchronous arbitrary control type, the specific type depends on the type of control navigated to. Return null when no control is navigated to.

previous(controlType?: string): Promise<IWinControl>

Contrary to the next() method, get the previous sibling control of the application, usually the adjacent left/upper node, depending on the layout direction of the container. You can pass in the parameter controlType Specify the sibling control type to achieve the filtering effect, that is, to get the sibling control of the next controlType in the application, controlType is a string, and the optional value can refer to ControlType. If there is no The sibling controls that meet the conditions will return null, for example, when the control is the firstChild of the parent control, it will return null.

controlType: (optional) string type, optional values can refer to ControlType.
returns: Asynchronous arbitrary control type, the specific type depends on the type of control navigated to. Return null when no control is navigated to.

parent(): Promise<IWinControl>

Contrary to firstChild() and lastChild() methods, get the parent control of the control, and return null if the control is the topmost control.

returns: Asynchronous arbitrary control type, the specific type depends on the type of control navigated to. Return null when no control is navigated to.

Examples of application scenarios

Example 1: Input positioning

There are forms that need to be filled in some applications, namely:

form:
    [label 1]: [input 1]
    [label 2]: [input 2]
    [label 3]: [input 3]
    [label 4]: [input 4]

For some applications, the input may not have enough attributes for identification, and all tags and inputs belong to the same container (form) without isolation, then CukeTest may treat input1-4 as the same input, at this time, by adding a label object, and then using the navigation method to obtain the input control on the right side, you can get the target input:

JavaScript

Python

let labels = ["label1", "label2", "label3", "label4"]
for(let label of labels){
    let input = await model.getText(label).next();
    await input.set("value to be entered");
}

labels = ["标签1", "标签2", "标签3", "标签4"]
for label in labels:
    input = model.getText(label).next()
    input.set("需要输入的值")

Example 2: Control traversal

You can traverse all controls in the entire application by calling the navigation method recursively.

Common API Usage

type definition file

possible class definitions

Rect class

MouseKey

PropertyIds

ControlType

Control filter condition Criteria class

Process startup options class SpawnOptions

Action method definition

click(x, y, mousekey): Promise<void>

dblClick(x, y, mousekey): Promise<void>

moveMouse(x, y): Promise<void>

wheel(value): Promise<void>

exists(time): Promise<boolean>

hScroll(value): Promise<void>

vScroll(value): Promise<void>

property(propertyIds): Promise<string | boolean | number | Rect>

waitProperty(propertyIds, value, timeoutSeconds): Promise<boolean>

drag(x, y): Promise<void>

drop(x, y): Promise<void>

pressKeys(keys, options?): Promise<void>

takeScreenshot(filePath): Promise<string>

doDefaultAction(): Promise<void>

property method definition

type(): Promise<string>

text(): Promise<string>

rawText(): Promise<string>

name(): Promise<string>

hwnd(): Promise<number>

x(): Promise<number>

y(): Promise<number>

height(): Promise<number>

width(): Promise<number>

enabled(): Promise<boolean>

focused(): Promise<boolean>

helpText(): Promise<string>

value(): Promise<string | number>

processId(): Promise<number>

rect(): Promise<Rect>

allProperties(): Promise<object>

Advanced method: Navigation

firstChild(controlType?: string): Promise<IWinControl>

lastChild(controlType?: string): Promise<IWinControl>

next(controlType?: string): Promise<IWinControl>

previous(controlType?: string): Promise<IWinControl>

parent(): Promise<IWinControl>

Examples of application scenarios

Example 1: Input positioning

Example 2: Control traversal

results matching ""

No results matching ""

Control filter condition `Criteria` class

Process startup options class `SpawnOptions`