Validating OSF

[d8a4n6]

I've been doing some testing .... so far, so good.

I'm using Brian Carrier's test images (http://dftt.sourceforge.net/)

Going through the searches for image #3 (NTFS Keyword Search #1) OSF did not locate the "n-frag" string, test #7 (http://dftt.sourceforge.net/test3/index.html)

Any ideas?

Jeff

[David (PassMark)]

It is probably worth making the point that these test images were designed about a decade ago and technology has somewhat moved on. The author of the images assumes you are doing a binary grep across the entire hard drive. No one would do this nowadays. The reason is that that doing a low level grep for strings will miss most important data. e.g. all data in .DOCX will be missed, everything in .ZIP files will be missed. In fact most data will be missed because no attempt is made to understand the higher level file format of the files when a grep is done.

Anyway, lets assume you really did want to do this. I further assume you are indexing these files, then doing a search?

I was able to find the desired text in this file, but there are a couple of tricks.

Trick #1. The file in question is a '.dat' file, file-n-3.dat.
There is no standard as to what a .dat file is. The default in Zoom is to treat the file as a text file. You need to change this to a binary file in the advanced settings. Setting the file type as binary means OSF will grep the file for strings (as the image author was expecting).

Trick #2. The default string extraction process isn't very aggressive for binary files in OSF, as you tend to pick up too much garbage, and the real gems are lost in the garbage. But you can pick 'aggressive' string extraction if you want.

Then the string will be found.


Screen shot below shows the text being found in the file as expected.

[d8a4n6]

It is probably worth making the point that these test images were designed about a decade ago and technology has somewhat moved on. The author of the images assumes you are doing a binary grep across the entire hard drive. No one would do this nowadays.

Is binary grep the same as just a plain text search of the raw data on the HDD? Carrier's instructions are, referring to the searches, "These should all be performed case sensitive and not as regular expressions."

The reason is that that doing a low level grep for strings will miss most important data. e.g. all data in .DOCX will be missed, everything in .ZIP files will be missed. In fact most data will be missed because no attempt is made to understand the higher level file format of the files when a grep is done.

Anyway, lets assume you really did want to do this. I further assume you are indexing these files, then doing a search?

No, I did not index, I used the search function in the Raw Disk Viewer, configured to search for ASCII and Unicode text.

I was able to find the desired text in this file, but there are a couple of tricks.

Trick #1. The file in question is a '.dat' file, file-n-3.dat.
There is no standard as to what a .dat file is. The default in Zoom is to treat the file as a text file. You need to change this to a binary file in the advanced settings. Setting the file type as binary means OSF will grep the file for strings (as the image author was expecting).

Trick #2. The default string extraction process isn't very aggressive for binary files in OSF, as you tend to pick up too much garbage, and the real gems are lost in the garbage. But you can pick 'aggressive' string extraction if you want.

Then the string will be found.


Screen shot below shows the text being found in the file as expected.

I found the file using this method. (Feature request = Ability to edit the extensions in the Indexer Advanced Configuration without having to delete and add (R. click?))

I also found the string using the Raw Disk View search function as "n-f" at the end of Sector 8058 and "rag" at the beginning of sector 8072. Not that it does any good doing it this way if you don't know were the fragmentation occurred.

I don't think using an index was the point of the exercise though. Actually, using the index, even with .dat files set to binary and using the extreme option, several of the other searches do not resolve, such as the $logfile and ADS searches, yet I found those strings using the Raw Disk View search function. Am I missing a way to do this within the indexer?

I see how I can identify which files (but not folders) have ADS usign the File Name Search function. But, I don't see how to actually search for a string in an ADS.?

[David (PassMark)]

I also found the string using the Raw Disk View search function as "n-f"

It is normal that you see disk fragmentation if you are using the raw disk viewer.

I don't think using an index was the point of the exercise though.

It isn't really stated what the goal is. If the goal of the test image is to verify that a forensics package can find certain strings in certain files, then there is an easier way.

You can open the image in the File Browser, then pick the file and open the file in the internal viewer and do you search.

Here is an example searching the hidden operating system file, $logfile for the desired text.



Here is an example searching the hidden ADS file stream, file-r-3.dat:here for the desired text.

[d8a4n6]

Thanks for the above examples. I understand that it is possible to search for a specified string in the ADS and hidden files using the file browser and internal viewer. But, it would require that you know or at least have an idea which file might contain the string you are looking for. If you don't know, you would have to open and search each individual hidden file and/or file with ADS.

Considering OSF can identify files with ADS, is it possible to also index the ADS and report hits as coming from ADS and the file?

[David (PassMark)]

The indexer module makes an effort to understand the high level file format of the files being indexed.

So, for example, it knows that a .DOCX file need to be uncompressed and then parsed to discover the text in the file. It knows that .MBOX E-mail files have attachments that are base64 encoded. It knows which types of image files have EXIF encoded meta data. It know what to do about an Excel file attached to an E-mail in a PST file which is compressed in a .ZIP file, etc, etc..

So in the case where a document format makes use of alternate data streams, the indexer could also open it up and index it. However NTFS ADS is pretty much never used for anything important. At least not by anyone in their right mind. The reason for this is compatibility. ADS data is lost when you move the file to a FAT disk or CD/DVD. This stops software developers using ADS for any important purpose.

So if you can think of a file type where it makes sense to index the ADS then we can add it. We can't think of such as case however. The example image above is a contrived artificial case.

If you really suspect a user or virus might be using ADS for hiding something here is a better detection method,
1) Open the "File name search" module.
2) Click on "Config"
3) Select "Gather alternate stream info"
4) Enter 1 as min number of alternate streams
5) Enter 1KB as min size of alternate streams (this will skip the common 26byte Zone.identifier stream that IE creates)
6) Click OK and Search.

In about 1 min the entire hard drive will be searched for files with ADS. Out of the 100,000 files on the typical Windows machine typically there will be zero or just a couple of files will ADS streams larger than 1KB. Making it easier to pick out which files to manually inspect.

[d8a4n6]

Roger, thanks.