Mime-Detective 24.12.2

Mime-Detective

Mime-Detective is a blazing-fast, low-memory file type detector for .NET. It uses Magic-Number and Magic-Word signatures to accurately identify over 14,000 different file variants by analyzing a raw stream or array of bytes. It also allows you to easily convert between file extensions and mime types.

How Does it Work?

Mime-Detective is a signature-based detection library that looks for patterns within the raw content of a file. Based on the presence (or absence) of certain bytes, most files can be accurately predicted. For example, every JPG file starts with 0xFFD8 FFE0 0010 4A46 4946 0001 0101 0047.

Limitations

Binary file work best because they often have an identifiable signature. Some file types, such as .txt files, have no identifying mark and may not be well predicted. Because of this, you may need to choose a fallback type if Mime-Detective is not able to predict the file type.

Getting Started

There are three main ways you can use Mime-Detective.

• The Default definition pack which includes a very small set of detection rules.
• The Condensed definition pack which includes an expanded set of detection rules.
• The Exhaustive definition pack which includes over 14,000 detection rules.

More information on these definitions is included toward the end of this file.

Installing from Nuget

Installing the Default (Small) Definition Pack

install-package Mime-Detective

Installing the Condensed (Medium) Definition Pack

install-package Mime-Detective
install-package Mime-Detective.Definitions.Condensed

Installing the Exhaustive (Large) Definition Pack

install-package Mime-Detective
install-package Mime-Detective.Definitions.Exhaustive

Create the ContentInspector

Create the Default ContentInspector

using MimeDetective;
var Inspector = new ContentInspectorBuilder() {
    Definitions = MimeDetective.Definitions.Default.All()
}.Build();

Create the Condensed ContentInspector

using MimeDetective;
var Inspector = new ContentInspectorBuilder() {
    Definitions = new Definitions.CondensedBuilder() {
        UsageType = Definitions.Licensing.UsageType.PersonalNonCommercial
    }.Build()
}.Build();

Create the Exhaustive ContentInspector

using MimeDetective;
var Inspector = new ContentInspectorBuilder() {
    Definitions = new Definitions.ExhaustiveBuilder() {
        UsageType = Definitions.Licensing.UsageType.PersonalNonCommercial
    }.Build()
}.Build();

Inspect Content

Once you have a ContentInspector you can use it to inspect a stream, file, or array of bytes:

var Results = Inspector.Inspect(ContentByteArray);
var Results = Inspector.Inspect(ContentStream);
var Results = Inspector.Inspect(ContentFileName);

Group Results by File Extension or Mime Type

var ResultsByFileExtension = Results.ByFileExtension();
var ResultsByMimeType = Results.ByMimeType();

Definition Packs

Definition packs make it easy to expand or limit the number of definitions that the Inspector will use. You can use one of the provided definition packs, create a limited subset of a definition pack, or create entirely new definition packs from scratch.

Default Definitions

The default definitions are included with the Mime-Detective nuget package and are located in the MimeDetective.Definitions.Default static class. You can create a copy of all definitions by calling MimeDetective.Definitions.Default.All() or just a limited subset by calling something like MimeDetective.Definitions.Default.FileTypes.Documents.All().

It can be used by anyone for any purpose and requires no additional licensing.

Mime-Detective.Definitions.Condensed

install-package Mime-Detective.Definitions.Condensed

This is a condensed library containing the most common file signatures.

It is derived from the publicly available TrID file signatures which may be used for personal/non-commercial use (free) or with a paid commercial license (usually around 300€).

Create a copy of these definitions by using the following code:

var AllDefintions = new Definitions.CondensedBuilder() { 
    UsageType = Definitions.Licensing.UsageType.PersonalNonCommercial //Change this to be your usage type
}.Build();

Mime-Detective.Definitions.Exhaustive

install-package Mime-Detective.Definitions.Exhaustive

This library contains the exhaustive set of 14,000+ file signatures.

It is derived from the publicly available TrID file signatures which may be used for personal/non-commercial use (free) or with a paid commercial license (usually around 300€).

Create a copy of these definitions by using the following code:

var AllDefintions = new Definitions.ExhaustiveBuilder() { 
    UsageType = Definitions.Licensing.UsageType.PersonalNonCommercial //Change this to be your usage type
}.Build();

Custom Definitions

Here is an example showing how to create a custom definition pack. This example uses all of the predefined "MP3" formats as well as a custom ".magic" file type:

internal static class CustomContentInspector {

    public static ContentInspector Instance { get; }

    static CustomContentInspector() {

        var MyDefinitions = new List<Definition>();
                
        //Add a predefined definition
        MyDefinitions.AddRange(MimeDetective.Definitions.Default.FileTypes.Audio.MP3());

        //Add a custom definition
        MyDefinitions.Add(new() {
            File = new() {
                Categories = new[] { Category.Other }.ToImmutableHashSet(),
                Description = "Magic File Type",
                Extensions = new[] { "magic" }.ToImmutableArray(),
                MimeType = "application/octet-stream",
            },
            //All of these rules must match
            Signature = new Segment[] {
                StringSegment.Create("MAGIC"), //anywhere in the file, expect "MAGIC" (exact case)
                PrefixSegment.Create(100, "4d 41 47 49 43") //At offset 100 in the file, expect the bytes "MAGIC".
            }.ToSignature(),
        });

        Instance = new ContentInspectorBuilder() {
            Definitions = MyDefinitions,
            StringSegmentOptions = new() {
                OptimizeFor = Engine.StringSegmentResourceOptimization.HighSpeed,
            },
        }.Build();
    }

}

Optimizing/Balancing Performance and Memory

The ContentInspector is designed to be a fast, high-speed utility. In order to achieve maximum performance and lowest memory usage, there are a few things you want to do.

1. Trim the Data You Don't Need

If you are positive that a file is going to be one of a few different types, create a definition set that only contains those definitions and trim out unnecessary fields.

var AllDefintions = new Definitions.ExhaustiveBuilder() { 
    UsageType = Definitions.LicensingLicensing.UsageType.PersonalNonCommercial
}.Build();

var Extensions = new[]{
    "aif", "cda","mid", "midi","mp3", "mpa", "ogg","wav","wma", "wpl",
}.ToImmutableHashSet(StringComparer.InvariantCultureIgnoreCase);

var ScopedDefinitions = AllDefinitions
    .ScopeExtensions(Extensions) //Limit results to only the extensions provided
    .TrimMeta() //If you don't care about the meta information (definition author, creation date, etc)
    .TrimDescription() //If you don't care about the description
    .TrimMimeType() //If you don't care about the mime type
    .ToImmutableArray()
    ;

var Inspector = new ContentInspectorBuilder() {
    Definitions = ScopedDefinitions,
}.Build();

2. Slow Initialization = Fast Execution

When the ContentInspector is first built, it will perform optimizations to ensure fastest execution. This is a tax best paid only once. If you have a list of files to analyze, build the Inspector once and reuse it.
Do not create a new Inspector every time you need to detect a single file.

3. Parallel = True/False

The ContentInspectorBuilder.Parallel option controlls whether multiple threads will be used to perform detections. If you have lots of definitions or want to make optimal usage of your CPU, this should be set to true. If you have a low number of definitions or you want more balanced CPU usage, set this to false.

4. Read Definitions Once

Materializing definitions causes a new instance of each definition to be created. If you are going to use the same definitions for multiple purposes, load them once and reuse them.

var AllDefintions = new Definitions.ExhaustiveBuilder() { 
    UsageType = Definitions.Licensing.UsageType.PersonalNonCommercial
}.Build();

var Inspector = new ContentInspectorBuilder() {
    Definitions = AllDefintions,
}.Build();

var MimeTypeToFileExtensions = new MimeTypeToFileExtensionLookupBuilder() {
    Definitions = AllDefintions,
}.Build();

var FileExtensionToMimeTypes = new FileExtensionToMimeTypeLookupBuilder() {
    Definitions = AllDefintions,
}.Build();

Benchmark

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000
AMD Ryzen 7 2700, 1 CPU, 16 logical and 8 physical cores
.NET SDK=6.0.202
  [Host]     : .NET 6.0.4 (6.0.422.16404), X64 RyuJIT
  Job-QGXQKV : .NET 6.0.4 (6.0.422.16404), X64 RyuJIT

Platform=X64  Runtime=.NET 6.0