How to create custom search rules for raw recovery
In the majority of cases, UFS Explorer performs a thorough analysis of file system structures and interprets this information to locate and recover the missing data. Yet, under certain circumstances, crucial service records may get severely damaged or even overwritten, like during formatting, initialization or other manipulations with the storage.
Hint: To learn more about file systems and their structure, please, refer to the basics of file systems.
This makes it impossible to achieve a satisfactory recovery result using the mentioned approach. To solve this problem or to expand the obtained result, one can enable IntelliRAW or the method of "raw recovery". This technique encompasses the search for predefined fragments of file content or the so-called file signatures (also "magic numbers"). These binary sequences are encountered at a certain offset at the beginning and sometimes at the end of the file as well and can be used to identify files of a certain type.
UFS Explorer already contains a broad set of IntelliRAW rules with signatures for most common types of files, like documents, images, multimedia, archives, etc. Yet, working with rare or proprietary file formats, one might want to extend the available list with custom rules that can be loaded to UFS Explorer and employed during the process of raw recovery.
A special free utility called IntelliRAW rules editor provides the possibility for easy definition of custom file types for IntelliRAW as well as numerous options for their configuration. Using this software, you can:
- Create a new file with a custom file type, open an existing file and complement it with new file types or even merge rules from several existing files;
- Specify a file extension which will be assigned to the found files of the corresponding custom file type;
- Provide a name for the type of the files which will also be used as the folder name in the results of raw recovery;
- Specify a rule for the start of the file using one or several signatures in the form of a hexadecimal sequence, an ASCII or Unicode string. The software can be set to scan the entire data block in search for the needed pattern, find it in a block at a given offset or divide the block into sub-blocks of a given size and look for the pattern at a specified position.
- Specify a rule for the end of the file:
- By a trailer signature and a specified number of bytes after it;
- By the size of the file defined as a field at a given offset with a given size. The value can be multiplied by the chosen integer and increased by the given number of bytes.
The software also supports the following special commands:
- Define the circumstances under which the match will occur:
- The format of data (binary, ASCII or Unicode)
- The scope of analysis which can coincide with the parent object, be a subset of a particular size defined at a certain offset or by a given hexadecimal signature;
- Hit conditions: the defined rule can be set to apply for all instances, when none of the other rules matches, when all the defined signatures are found or when any of them hits.
To define your own custom file type and use it during storage scan with UFS Explorer follow the given procedure:
Download, install and launch CI Hex Viewer. This utility provides convenient means for analysis of the hexadecimal content of files on the presence of file signatures.
Open several sample files of the needed format in CI Hex Viewer. For this use "A file" subitem of the "Open" menu and choose the "Plain file, as it is" option. As an illustrative example we use five simple JPEG images.
Explore the binary data of each opened file to find identical patterns, especially at the very beginning and at the end. Sometimes it will be easier to navigate using the text representation field. For more accurate results you will need to examine as many files as possible. In our samples we can see that each of the files starts with the signatureFF D8 FF (˙Ř˙) and ends with FF D9 (˙Ů).
Note: In some cases, it might be difficult to tell the difference between the true file signatures and simply similar data patterns that contain service information. If possible, try taking sample files that are not closely related, for instance, videos recorded by different cameras.
Select the found signature(s) and copy it (them) to the clipboard using the "Copy raw data tool". Also, note the position of the first signature in the "Selection start" field. In our case, it is 0x0.
Now that you've obtained the needed file signature, you can create a file with rules for your custom file type that will be used by UFS Explorer. For this you will need to download IntelliRAW rules editor.
Run the software, create a new file by pressing the "New file" button and click "New type". In the opened tab provide the name of the file type and the file extension. In our case, these are JPEG images and jpg.
Define the obtained signature for the file start. For this press the "Start" button and paste the signature in the "Search for" field. If the position of the signature is different from 0x0, provide it in the "Position" field and hit "OK".
If your file type has a signature for the end of the file, provide a rule for it as well. To do that click "Termination" and select "Matching signature" in the opened menu. Paste the ending signature in the "Search for" field and hit "OK".
If available, provide additional parameters for file search or proceed with the default ones.
Change the "Rule hits" status to "active" and save the created rule to a file using the "Save file" button.
In UFS Explorer when defining the parameters for scan, select "I’m interested in the result of recovery by known content" and enable "I want to use my own data search rules". Press "Load rules", navigate to the "*.urrs" file you’ve created and hit "OK".
After you complete the procedure, UFS Explorer will use your rule and provide the files found with its help in the $Custom folder. The files will be automatically assigned new names as this information is not available with raw recovery. Also, you should be aware that this method has its flaws and provides poor results in case of extensive file fragmentation. Moreover, if there is no clear signature for the end of the file, many of them may appear to be damaged.
Last update: October 05, 2020