DIY Debugging of Search Strings

[ChrisGreaves]

This recent post and most of my early posts made me think that there could be a way for users to find out why their search failed, rather than a plaintive post (I'm speaking of my posts!) asking "Why didn't my search work as I had expected?". The most common problem might be that many of us arrive at Everything fueled with our use of DOS-like search patterns "My*.DO?" and the like. Then we meet the Filters and leave "Picture" in place while we experiment with "EXT:DOC". A kind soul in the forums straightens us out and we head for the next failed search with increased enthusiasm and stub our toe yet again.

I reason that, regardless of the implementation language, the logic of an application such as Everything must be that of a winnowing process:-

Code: Select all

If the path is good
    If the name is good
        If <the next condition> is good
            If <the next condition> is good

and so on until the search string has been completely processed and the result list can be produced.

Suppose that we had a Do-It-Yourself debugging switch that could dump out (somewhere) the conditions tested and the results at each step of the winnowing process.

Then the New User could read that dump and discover that the number of results plummeted to zero immediately after the filter "Pictures" had been applied (or the extent DO? or the path or name tests etc.) and the optional slapping of the forehead could take place in privacy.

This feature would take time to implement, but would it be cost-effective? To answer that we would need to add up the time it takes to implement (and test) the feature AND add up the time spent by New Users scratching their heads, posting the question, and Keen Members of the forums posting the replies. Add in the delay, in days, while a project hangs in the balance, and you have the cost of the problem.

I will maintain that the cost of the problem can be computed by an individual, but will NOT be computed by the general population in these forums. The actual cost of our new-user false starts can not be calculated.

Which means that we cannot evaluate the benefits of this idea. No-one can come back and argue "It's not worth the cost"

But programmers can work out the cost of implementation, because, in well-written code, those points at which the tests are made (and the results are narrowed down) are well-defined, and we might anticipate an automated procedure that can locate those decision points and insert a snippet of debugging code that reads

Code: Select all

[AirCode]
If the name is good
   If DIYDebug then Output the decision result Endif
Else
   If DIYDebug then Output the decision result Endif
Endif
[/AirCode]

What do you think?

I understand that Everything 1.5 has a search-string pre-processor that takes my characters from the search string box, parses them, and then arranges them in a sensible order (examine Indexed file names BEFORE testing un-indexed content and so on), in which case there may be a focal point in the code where a DIY-debug could be inserted, rather than writing one-time code to automate the process of scattering debugging snippets among 000s of lines of code.

Cheers, Chris

[void]

I recommend breaking your search down to check which part is good.
Use the clipboard to cut the end of the search off.

For example, perform the path search for:

c:\windows

If this shows results, you can assume the path search is good.
Then search for the name:

c:\windows\ notepad

If this shows results, you can assume the path and name search is good.
Then search for some condition:

c:\windows\ notepad size:<1mb

and continue from there..




Everything shows a search tree in verbose debug mode (advanced):

• In Everything 1.5, from the Tools menu, under the Debug submenu, check Verbose.
• From the Tools menu, under the Debug submenu, check Console.
• Perform a search.
• A search tree is shown in the debug console.

For example:

c:\windows notepad size:196kb dm:2024-05-23

Shows the following search tree:

Code: Select all

FILE OPs:
             /- MATCH
         /- 234
        |    \- NOMATCH
     /- 232
    |    \- NOMATCH
 /- 5 notepad
|    \- NOMATCH
163
 \- NOMATCH

While this doesn't show which part of the search is good, it does show how the search is broken in to OPcodes and what Everything is doing internally..
The tree is easier to process if you rotate it clockwise 90 degrees.
After rotating, the top node is the root and this is where the search starts.
The number is the OPcode.
The node on the right is followed when the OPcode matches.
The node on the left is followed when the OPcode does not match.
163 is the path search opcode, 5 is a normal search, 232 is a size search and 234 is a dm search.



You can use custom columns to show good search strings:
Consider the following:

Code: Select all

addcolumn:a sort:a-descending <c:\windows notepad size:196kb dm:2024-05-23 a:=4> | <c:\windows notepad size:196kb a:=3> | <c:\windows notepad a:=2> | <c:\windows a:=1>

It's a little tricky to setup the search, but column A will show you which parts of the search matched (1=path only, 2=path+name, 3=path+name+size, 4=path+name+size+dm).



I will consider a UI feature to make it easier to see what matches and what doesn't..
Maybe a search preview..
A scripting language in Everything is on my TODO list.
For now, the Everything SDK might be the way to go..
Thank you for the suggestion.

[ChrisGreaves]

Everything shows a search tree in verbose debug mode:
You can use custom columns to show good search strings:

Void, thanks for this response.
I did a quick essay with Verbose/Console mode and didn't see the tree.

Verbose.jpg (119.85 KiB) Viewed 1609 times

I did get a burst of data which appeared to repeat ("Wait 1,4").
I did not yet test the columns method.
I shall set aside some time tonight and tomorrow to work on this, using a few simple test-bed examples such as "ext:doc" with filter=Picture and a few other searches which have tripped me up in the past.
I already use the divide-and-conquer method (ever since I learned about binary-chop!) but am very interested in heading off queries of the form "why didn't THIS search work" by a quick glance at the screen, rather than posting another complaint (by me) to the forums. To that end I think a GUI might be quite useful.

Thanks again, Chris

[therube]

Try it again.
First time I did it, I didn't see it, but then checked again, & ensured that both Console & Verbose were both checked, & then it worked.

(That said, it's very noisy & hard to decipher.)

[ChrisGreaves]

Try it again.

Thanks, TheRube.
I have just killed Everything after ten minutes of scrolling on a seemeingly endless loop.
It is probably something to do with my setup (old DELL Win7 system), so later today I will do a reboot and try asgain.
But I can try it out too on the Acer Aspire Win11 machine sitting on the adjacent table.
I'll report back in a few hours.
Cheers, Chris
PS. Here is a PrtScr of the window after 10+ minutes of non-stop scrolling.

Verbose2.jpg (164.21 KiB) Viewed 1593 times

[therube]

ten minutes of scrolling

That's the "noise".

If you (after getting enough so that you find your wanted data in the mix) deselect, Verbose, that should quiet things down.


I had something like:

Code: Select all

FOLDER OPs:
 /- MATCH
41
 \- NOMATCH
FILE OPs:
 /- MATCH
41
 \- NOMATCH

[void]

It might be easier to view the search tree in the log:

• In Everything 1.5, from the Tools menu, under the Debug submenu, check Verbose.
• From the Tools menu, under the Debug submenu, click Start Debug Logging.
• Perform a search.
• From the Tools menu, under the Debug submenu, click Stop Debug Logging.
  ---This will open your %TEMP%\Everything Debug Log.txt in notepad
• Search for:
  
  FILE OPs:

[ChrisGreaves]

It might be easier to view the search tree in the log:

And here I am on the Acer Aspire, Win11 latop.
I think I must be suffering from Refrigerator Blindness
Attached is the log file from a few minutes ago.
I can locate lines that contain "File Ops", but in no way do I see any sort of tree structure.
I understand OpCodes and tokenizing, but " 5 is a normal search, 232 is a size search and 234 is a dm search." leaves me stranded.
How is a novice user supposed to understand this tree structure, once it is located?

When I think of debugging a search string by the end-user (as distinct from The Developer) I think of the original search string provided by the user - or possibly re-arranged in optimal order by a per-processor, looking something like this:-

Code: Select all

ext:doc z: dm:>=2024-06-15 content:datascene
                                 ^

with a caret indicating where the results List dropped to zero items.

Better yet would be a numeric indicator:-

Code: Select all

ext:doc z: dm:>=2024-06-15 content:datascene
 4,000   1,000     200            17            

with the Result List population at each successive step.
In the example above I might ask myself "Why is there such a huge drop when I isolate results to Z:, my mammoth global all-time backup drive?"

Perhaps with the right knowledge of Op Codes and other tags one could write a "debugging file translator" that could construct, from the verbose data, the data that the user needs in order to identify weak points in their constructed search string, but that sounds like a lot of work to document what must, surely, be available as Everything works its way through a search string?
Cheers, Chris

[void]

The tree is listed below FILE OPs:

search:

t: ext:doc dm:>=2024-06-01

Code: Select all

2024-09-03 06:47:18.925: FILE OPs:
2024-09-03 06:47:18.925:          /- MATCH
2024-09-03 06:47:18.925:      /- 234
2024-09-03 06:47:18.925:     |    \- NOMATCH
2024-09-03 06:47:18.925:  /- 205
2024-09-03 06:47:18.925: |    \- NOMATCH
2024-09-03 06:47:18.925: 163
2024-09-03 06:47:18.925:  \- NOMATCH

163 = t: path search
205 = doc extension
234 = date modified search



The search tree is for advanced users.




Wiggly underline could be useful too when a term doesn't produce any results.
For example:

Code: Select all

ext:doc z: dm:>=2024-06-15 content:datascene
                           ~~~~~~~~~~~~~~~~~

I will consider a pointer or wiggly red underline when a term matches nothing.
Thank you for the suggestions.



The counts for each term could be useful, I will consider an advanced information page to show this information.
Everything does very little when a term matches or doesn't match, keeping a match tally is rather expensive.
Match tally would only be calculated on user request.