[request] Save original and last known location of image to be used in the next script execution (performance!)

Bug #681781 reported by Bruno Vieira
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
SikuliX
In Progress
Wishlist
RaiMan

Bug Description

1.1.0: partly implemented with new Image class (last match location is stored and checked first at subsequent searches)
1.2: make it persistent including the information where it was captured
-------------------------------------------------------------------
In order to improve image finding speed, Sikuli could save together with an image some meta information about that image.
the meta information could have the following, for a start:
- last seen coordinates
- original coordinates

So whenever a new image is created, Sikuli would populate these fields.
whenever this image is called by a script, Sikuli would update these fields.

The idea is that whenever the script runs, it can start to look the image on the "last seen coordinates", then on the "original coordinates" and if both fail, the entire screen as it normally does.
I believe this specific Find of the image (using these fields) can be done with a small timing (1s or 2s) so that the script can start searching the entire screen with not much delay.

Best case scenario: Most of the times, a specific image appears on the same location as the last time or very close to it (the last know coordinates can be a bit bigger area than the actual image occupies to allow some variation).

Worst case scenario: the image appears always in differernt places of the screen, thus making Sikuli loose those one or two seconds defined above.

Ir order to get rid of the worst case scenario, Sikuli IDE would allow the user to define a "always search the entire screen" or "contextualized search based on last known location" - or whatever better names for this :)

Tags: fkt-region
Bruno Vieira (giuliapo)
description: updated
Revision history for this message
RaiMan (raimund-hocke) wrote :

My first reaction was: mmh, good idea.

But some thoughts later I do not think we really need it as a feature, at least not with the images - may be with the match objects.

In the philosophy of Sikuli, the image as such is the wrong place to store some find-history. It is the Match object. And this in the moment lacks the feature to recall the image that was used. So you should be able to ask a match, wether the reason for its existence (the image) is still were it was found in the past.

I bridge this gap by having a naming convention and nearly never use images directly in the find operations, I store them in variables.

And I never use a second find, when I already have a match.

I have seen many scripts, that just repeat the searches by using the same image again instead of the match that is already there. And even if you do not store it in variables on a regular base, you have getLastMatch() if needed. I think, that in most workflows key visual objects, that may be reused stay in the same place during the lifetime of a script - and then you have the match to reuse.

but if you are not sure and want to have performance, then already now with the available means, it is easily possible to do what you want:

img = "path-to-an-image"
mImg = find(img)
# some actions and some time later
mImg = mImg.getLastMatch() if mImg.exists(img, 0) else find(img)

So if you need it more often or if you have "shivering objects", that may drift a little bit around, you may have a def():
def checkLastLoc(reg, img, border=50):
    r = reg.nearby(border)
    return r.getLastMatch() if r.exists(img, 0) else find(img)

(just quick and dirty: I would optimize it with exists(), to handle FindFailed situations)

so even shorter:
mImg = checkLastLoc(mImg, img) # using the default of 50 pixel around

BTW: having def()'s for everything you need more often together with import support for .sikuli's will be available with upcoming Sikuli X (I have it already now with my 10.2 ;-)

To speed up my scripts I use regions as often as possible, no unnecessary repeated searches by storing my matches, exists(,0) as often as possible (to not have to wait 3 seconds if something is not there that should be there), keyboard shortcuts instead of finding / clicking buttons if possible and last but not least calculated clickpoints based on offsets to known reference matches.

Revision history for this message
Bruno Vieira (giuliapo) wrote :

I think I understood your idea.
But it seems to me it does not solve the question between executions and this is the main point from my point of view.
Normally I click a button once when I execute a script and I go to the next screen (workflow of the application). I normally don't need to click again on a specific item on the same execution.
But between executions (maybe I didn't explain this well) is where we always try the whole screen or a region if we program the script like that.
From my point of view, it would speed a lot finding the image for the first time between executions if we save its last location.
Or, as you say, saving the match.

You were not referring to this point, right?

Revision history for this message
RaiMan (raimund-hocke) wrote :

ups, now I got it. you need the information to be persistent.

So you would need Sikuli to store the match information with the image FILE in some extra fields.

Or Sikuli has to store some logging information in a file located in the .sikuli, that can be recalled with the next run.

Still I do not think, that this is a feature that could be useful very often, since it needs the visual objects to be (nearly) in the same location on the screen, the next time they are looked for, otherwise it would search the whole screen.

So let's wait, what the developers will comment.

Revision history for this message
Tsung-Hsiang Chang (vgod) wrote :

I think saving absolute may not help much, since the window's location may be totally different. But maybe saving the location of widgets relative to their parent window is a good idea, because the layout of a GUI doesn't change often.
Anyways, this is a good idea and inspires me the possibility to save some interesting information in image files, e.g. the underlying GUI component references obtained from accessibility APIs. I will see what I can do on this feature.

RaiMan (raimund-hocke)
summary: - Feature request: Save original and last known location of image
+ [request] Save original and last known location of image
RaiMan (raimund-hocke)
Changed in sikuli:
importance: Undecided → Wishlist
RaiMan (raimund-hocke)
Changed in sikuli:
status: New → In Progress
assignee: nobody → RaiMan (raimund-hocke)
tags: added: fkt-region
removed: find image information meta speed
RaiMan (raimund-hocke)
Changed in sikuli:
importance: Wishlist → High
milestone: none → x1.1
RaiMan (raimund-hocke)
Changed in sikuli:
status: In Progress → Fix Committed
RaiMan (raimund-hocke)
Changed in sikuli:
milestone: 1.1.0 → 1.2.0
description: updated
RaiMan (raimund-hocke)
Changed in sikuli:
status: Fix Committed → In Progress
importance: High → Medium
milestone: 2.0.0 → 1.1.4
RaiMan (raimund-hocke)
summary: - [request] Save original and last known location of image
+ [request] Save original and last known location of image to be used in
+ the next script execution (performance!)
RaiMan (raimund-hocke)
Changed in sikuli:
importance: Medium → Wishlist
RaiMan (raimund-hocke)
Changed in sikuli:
milestone: 1.1.4 → 2.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.