How to create custom search rules for raw recovery

creation of custom scanning rules for raw data recovery with ufs explorer program

In the majority of cases, UFS Explorer performs thorough analysis of the file system structures and interprets this information to locate and recover the missing data. Yet, under certain circumstances, crucial service records may get severely damaged or even overwritten, like during formatting, initialization or other manipulations with the storage. This obstacle often makes it impossible to bring the files back using the mentioned approach. To solve the problem or to expand the obtained result, one can enable IntelliRAW or the method of "raw recovery". Read on to learn its essence and be able to create your own search rules for the program to rely on..


What is raw data recovery and how does it work?

The terms "raw recovery", "recovery by known content" or "IntelliRAW" refer in fact to the same technique. It encompasses the search for predefined templates or the so-called file signatures (also "magic numbers"). These binary sequences occur mostly at the beginning of the file and uniquely characterize each particular type. When the files are not fragmented, UFS Explorer can use the respective signature to identify them within a given data range and retrieve their content, even in the absence of the file system’s metadata.

The software already contains a basic set of IntelliRAW rules with signatures for common file types, like documents, images, multimedia, archives, etc. Apart from the raw recovery itself, they are used to enhance the quality of a scan for certain file systems, control the state of the restored items and to assign a proper extension to files of non-readable formats, like *.chk.

list of intelliraw rules by default in ufs explorer software interface

Yet, in some situations, one might need to extend the available list with custom rules, for instance, when rare or proprietary files have been lost. Such a possibility is presented in all editions of UFS Explorer. The software allows creating a template for any lacking type and configure it manually. At the same time, the function should be used thoughtfully: in case of an inaccurate template, the scan may slow down considerably and produce false results.

The procedure of setting up a custom file type consists of three separate stages: determining text or binary signatures that are distinctive for the given format, defining a rule based on which they will be detected by the program and applying it upon the specification of parameters for the storage scan. If the files of this type are quite commonplace, it is advised to contact our team so that the rule for it could be included into the embedded set. This can accelerate their processing by the software and potentially allows getting more detailed information about them (descriptive file name, dates of creation and last modification, etc.).

Discovery of a file’s signature

To acquire a signature for the file format of your choice, you will need to prepare some samples, preferably ones that are not closely related – for instance, videos recorded by different cameras. When the files are similar, it might be difficult to tell the difference between the true file signatures and coincidental data patterns that contain service information.

With the files at hand, you may complete the following steps:

  1. Decide on the means you will use for examining the hexadecimal content of files for the presence of file signatures. It is possible to perform hex analysis in UFS Explorer or employ any other alternative tool of your preference.

  2. Open the hexadecimal representations of several sample files of the needed format. As an illustrative example, we have SSF-files – this is a vendor-specific type of CCTV recordings.

    several sample files opened in hexadecimal viewer of ufs explorer software

  3. Explore the content of each opened file and determine whether the format is binary or text-based. Look for identical patterns in them, especially at the very beginning and at the end. For more accurate results, there should be at least 32 unique bits.

    Our samples are binary, and we can see that each of the files starts with common sequences: STL Stream Format v1.0 (53 54 4C 20 53 74 72 65 61 6D 20 46 6F 72 6D 61 74 20 76 31 2E 30) and STL Stream Format v3.0 (53 54 4C 20 53 74 72 65 61 6D 20 46 6F 72 6D 61 74 20 76 33 2E 30). Thus, the rule for our custom type can be defined as 'STL Stream Format v1.0' or 'Stream Format v3.0'.

    identical patterns selected in hexadecimal contents of several sample files in ufs explorer program interface

  4. Select the signature found closer to the start of your sample files and copy it to the clipboard using the tool for copying raw (hexadecimal) data. If your signature is not binary, employ the tool for copying text data instead. Paste and save this sequence to any safe location in order to be able to use it later.

    Note the position of your signature indicated in the "Selection start" field. In our case, it is 0x0, but your signature may not be necessarily located at the very beginning.

    two available options to copy text and hex text in hexadecimal viewer of usf explorer software interface

  5. If you’ve spotted a signature at the end of your sample files, perform the copying operation for this signature as well.

Creating a search rule based on the identified signatures

Now that you've obtained the needed file signatures, you can define a custom search rule that will be used by UFS Explorer for their recognition.

Starting from version 9.4, the software offers a graphical in-built editor that makes it possible to create, view and edit such rules directly in the program. It can also be used to load a custom file type from the existing *.xml file of any supported third-party format.

Those who didn’t update their products may download a special free utility called IntelliRAW rules editor and follow the instruction for creating custom search rules in older versions of UFS Explorer (up to version 9.3). Newer software remains compatible with old rules (*.urrs file format). Yet, they can be managed only in IntelliRAW rules editor and are not supported by the internal editor.

If you use a contemporary version of the program, proceed as follows:

  1. Run UFS Explorer and, if necessary, change its settings in the corresponding pane.

  2. Open the "Tools" item from the main menu and choose the "IntelliRAW rules" option to launch the embedded editor.

    intelliraw rules subelement of tools element on main menu panel of ufs explorer interface

  3. In the opened dialog, you will see the list of file types that are enabled in the software by default. Please note that the pre-defined rules cannot be viewed or modified.

    full list of intelliraw rules enabled by default in ufs explorer

  4. To create your own file type, press the "New type" button from the toolbar at the top.

    new type button in intelliraw rules editor of ufs explorer software

  5. Choose the type of rule you are going to define, depending on the format of your signatures (hexadecimal sequences or text strings).

    option rule for binary data selected in popup window to choose rule type to define in intelliraw rules editor of ufs explorer

    option rule for text files selected in popup window to choose rule type to define in intelliraw rules editor of ufs explorer

  6. Specify a file extension that will be used for the found files of the given format.

    field to enter extension for new file type in add rule window of intelliraw rules editor of ufs explorer

  7. Provide a name for your custom file type in the respective field. This name will also be assigned to the container for these files in the results of raw recovery.

    field to enter name of new file type in add rule window of intelliraw rules editor of ufs explorer

  8. If you use a text-based signature, you may choose the required encoding via the drop-down list next to the "Text format" property.

    drop-down list of text encodings of text format property in add rule window of ufs explorer intelliraw rules editor

  9. Press the "Add rule" button found at the top and insert the first signature obtained at the previous stage into the field next to the "Value" property.

    add rule button in add new rule window of ufs explorer intelliraw rules editor

    These rules also support a special syntax available for the advanced hex search tool in the Hexadecimal Viewer component of UFS Explorer. Based on this syntax, we can combine our two signatures in a single rule defined as 'STL Stream Format v'{31,33}'.0'.

    value field in advanced hex match dialog of ufs explorer intelliraw rules editor

  10. In case of a hexadecimal signature, provide its position in the "Rule offset" field.

    rule offset field in advanced hex match dialog of intelliraw rules editor of ufs explorer

    In case of a text-based one, you may enable case-sensitivity to distinguish between uppercase and lowercase characters.

    option to enable and disable case sensitivity for text based rule in intelliraw rules editor of ufs explorer

  11. Once you press "OK", the specified rule will get listed in the bottom pane. It can be edited anytime using the "View/Edit" button.

    button to view and edit selected user defined rule in ufs explorer intelliraw rules editor

  12. If you have another signature, press the "Add rule" button again and repeat the previous steps.

    button to add more rules in add new rule window of ufs explorer intelliraw rules editor

  13. Define the conditions that must be true for a match to take place: use the "Rules logic" property to establish whether all of the signatures or at least one of them must be present in the content of a file.

    rules logics parameter in add rule window of ufs explorer intelliraw rules editor

  14. Hit "OK", and the created custom file type will be added to the available set of IntelliRAW rules. It will be indicated with a different color and have "User-defined" property.

    custom file type selected in intelliraw rules list of ufs explorer intelliraw rules editor

  15. New rules get enabled by default immediately after their creation. You may deactivate your rule by selecting it in the list and pressing the "Disable" button from the toolbar at the top and use the "Enable" button later for its activation.

    button to deactivate selected custom rule in ufs explorer intelliraw rules editor

After that, the editor can be closed. The defined custom file type will remain in the program after its restart. You may also refer to this component later if you want to edit, disable or delete it.

Implementation of the custom rules during a scan

To look for your own custom file type when scanning a storage with UFS Explorer, follow the given procedure:

  1. At the stage of setting up a scan, enable the "Yes, I’m interested in the result of recovery by known content" option.

    checkbox to enable recovery by known contents on step 3 of storage scan configuration in ufs explorer interface

  2. Right after that, another option that allows using your own data search rules will emerge. Tick it off as well.

    checkbox to enable custom search rules usage on step 3 of storage scan configuration in ufs explorer interface

  3. The software will display the number of custom rules that are currently defined. If you need to make any adjustments to them, use the "Manage rules" option. If not, proceed to the scan right away.

option to manage custom search rules on step 3 of storage scan configuration in ufs explorer interface

UFS Explorer will use your rule and provide the files found with its help in the $Custom folder. The files will be automatically assigned new names, as this information is not available with raw recovery. Also, you should be aware that this method has its flaws and provides poor results in case of extensive file fragmentation.

Export/import of the created rules

Starting from version 9.5, UFS Explorer provides the means for saving selected custom search rules as *.xml files that can be used for backup purposes or for further import into the software launched on another computer.

  1. In order to create such a file, open the embedded editor using the "IntelliRAW rules" item from the program’s main menu.

    intelliraw rules item under tools element of ufs explorer main menu

  2. After that, get the "Export" tool from the toolbar at the top.

    function to export selected user defined file types to a file in ufs explorer intelliraw rules editor

  3. In the opened window, select all the custom file types you want to export.

    dialog to select user defined file types to export in ufs explorer intelliraw rules editor

  4. Finally, press the "Export" button.

    export button in custom file types selecting dialog in ufs explorer intelliraw rules editor

  5. Choose the location and define the name for the *.xml file to be created.

    dialog to select location and enter name for xml file with exported custom file types

  6. Click the "Save" button, and the file will be saved to the specified folder. All the rule values in it will be presented as standard base64 characters that do not conflict with the XML parsing. The file itself is encoded using the UTF-8 encoding and can be edited in a text processor. If the value is XML-safe and cannot be wrongfully interpreted as markup, it can be defined as plain text or an advanced hex string without the base64 prefix.

Analogously, custom search rules can be imported from the existing compatible *.xml file using the embedded editor.

  1. For this, pick out the "Import" tool from its toolbar.

    function to import xml file with user defined file types in ufs explorer intelliraw rules editor

  2. In the opened dialog, navigate to the folder where the file is stored, select it and hit the "Open" button.

    dialog to select previously exported xml file with user defined file types to be imported to ufs explorer intelliraw rules editor

  3. If it contains definitions for multiple custom file types, tick off the ones to be imported.

    dialog to select custom file types to be imported to ufs explorer intelliraw rules editor from exported xml file with user defined file types

  4. After you hit the "Import" button, the selected rules will be added to the current set as if they were just specified manually.

    import button in user defined file types selecting dialog in ufs explorer intelliraw rules editor

Import of legacy rules

Though older search rules created for UFS Explorer up to version 9.3 cannot be managed in the contemporary software, it is still possible to employ them in the process of data recovery. For this, you need to perform slightly different steps:

  1. When configuring a scan, put a checkmark next to "Yes, I’m interested in the result of recovery by known content" and then tick off "I want to use my own search rules".

    checkboxes to enable raw reovery on step 3 of storage scan configuration in ufs explorer interface

  2. After that, you will see a number of options. Choose "Import legacy rules" to load the *.urrs file.

    function to import legacy rules on step 3 of storage scan configuration in ufs explorer interface

  3. In the opened window, find the file in its folder and select it.

    dialog to choose urrs file with legacy rules to be imported in ufs explorer

  4. After you click "Open", the rules defined in the file will be added to the current IntelliRAW rules set. The software will use them during the procedure along with the rest of search rules.

    open button to add legacy rules from urrs file to ufs explorer intelliraw search

Setting up custom search rules in a video-guide

The following video-guide will give you a more complete picture of this process and its nuances:

Watch on YouTube

Last update: February 16, 2023

If you liked this article, you can share it on social media: