Cynicszm.FlatFiles 9.2.2

FlatFiles

Reads and writes CSV, fixed-length and other flat file formats with a focus on schema definition, configuration and speed. Supports mapping directly between files and classes.

Download using NuGet: Cynicszm.FlatFiles

This is a fork of FlatFiles by Travis Parks, continued for .NET 10 after the original was archived. The assembly is still FlatFiles.dll and the namespaces are unchanged. Through 7.6.0 every type and member name was unchanged too, so only the package reference differed; 8.0.0 renames a few of them, and the CHANGELOG names each one.

You can check out all of the awesome enhancements and new features in the CHANGELOG.

Overview

Plain-text formats primarily come in two variations: delimited (CSV, TSV, etc.) and fixed-width. FlatFiles comes with support for working with both formats. Unlike most other libraries, FlatFiles puts a focus on schema definition. You build and pass a schema to a reader or writer and it will use the schema to extract or write out your values.

A schema is defined by specifying what data columns are in your file. A column has a name, a type and an ordinal position in the file. The order matches whatever order you add the columns to the schema, so you're left just specifying the name and the type. Beyond that, you have a lot of control over the parsing/formatting behaviour when reading and writing, respectively. Most of the time, the out-of-the-box options will just work, too. But when you need that level of extra control, you don't have to bend over backward to work around the API, like with many other libraries. FlatFiles was designed to make handling oddball edge cases easier.

If you are working with data classes, defining schemas is even easier. You can use the type mappers to map your properties directly. This saves you from having to specify column names or types, since both can be derived from the property. For those working with ADO.NET, there's even support for DataTables and IDataReader. If you really want to, you can read and write values using raw object[].

Table of Contents

• Overview
• Type Mappers
  • Auto-mapping
  • Constructor mapping
  • Attribute mapping
• Schemas
  • Column types
  • Creating your own columns
• Delimited Files
  • Splitting one record's text
• Fixed Length Files
• Matching columns to the header
• Records that do not fit the schema
• Character Encoding
• Handling Nulls
  • Default Values
  • Null Formatters
• Ignored Fields
• Metadata
  • Writing metadata
  • Creating your own metadata columns
• Skipping Records
  • Comments and blank lines
  • One name, two different events
• Error Handling
• Files Containing Multiple Schemas
  • Choosing by the record's text
  • Passing records over
• Custom Mapping
• Runtime Mapping
• Disabling Optimisation
• Mapping Under Native AOT
  • Turning generation off
• Non-Public Classes and Members
• ADO.NET DataTables
• FlatFileDataReader
• How This Library Is Verified
• License

Type Mappers

Using the type mappers, you can directly read file contents into your classes:

customer_id,name,created,avg_sales
1,bob,20120321,12.34
2,Susan,20130108,13.88
3,Tom,20180519,88.23

var mapper = DelimitedTypeMapper.Define<Customer>();
mapper.Property(c => c.CustomerId).ColumnName("customer_id");
mapper.Property(c => c.Name).ColumnName("name");
mapper.Property(c => c.Created).ColumnName("created").InputFormat("yyyyMMdd");
mapper.Property(c => c.AverageSales).ColumnName("avg_sales");
using (var reader = new StreamReader(File.OpenRead(@"C:\path\to\file.csv")))
{
    var options = new DelimitedOptions() { IsFirstRecordSchema = true };
    var customers = mapper.Read(reader, options).ToList();
}

To define the schema when working with type mappers, call Property in the order that the fields appear in the file. The type of the column is determined by the type of the mapped property. Each property configuration provides options for controlling the way FlatFiles handles strings, numbers, date/times, GUIDs, enums and more. Once the properties are configured, you can call Read or Write on the type mapper.

Note The Read method only retrieves records from the underlying file on-demand. To bring the entire file into memory at once, just call ToList or ToArray, or loop over the records inside of a foreach. This is good news for people working with enormous files!

Writing to a file is just as easily:

mapper.Property(c => c.Created).OutputFormat("yyyyMMdd");
mapper.Property(c => c.AverageSales).OutputFormat("N2");
using (var writer = new StreamWriter(File.OpenCreate(@"C:\path\to\file2.csv")))
{
    var options = new DelimitedOptions() { IsFirstRecordSchema = true };
    mapper.Write(writer, customers, options);
}

Note I was able to customise the OutputFormat of properties that were previously configured. The first time Property is called on a property, FlatFiles assumes it's the next column to appear in the flat file. However, subsequent configuration on the property doesn't change the order of the columns or reset any other settings.

Auto-mapping

If your delimited file (CSV, TSV, etc.) has a schema with column names that match your class's property names, you can use the GetAutoMappedReader method as a shortcut. This method will read the schema from your file and map the columns to the properties automatically, returning a reader for retrieving the data. It's important to note that you cannot customise the parsing behaviour of any of the columns, at which point you are better off explicitly defining the schema. Fortunately, FlatFiles uses pretty liberal parsing out-of-the-box, so most common formats will work.

By default, columns and properties are matched by name (case-insensitive). If you need more control over how columns and properties are matched, you can pass in your own IAutoMapMatcher. Given an IColumnDefinition and a MemberInfo, a matcher must determine whether the two map to one another. For convenience, you can also use the AutoMapMatcher.For method to pass a Func<IColumnDefinition, MemberInfo, bool> delegate rather than implement the interface.

Similarly, use the GetAutoMappedWriter method to automatically write out a delimited file. Note that there's no way to control the column formatting. However, you can control the name and position of the columns by passing an IAutoMapResolver. The IAutoMapResolver interface provides the GetPosition and a GetColumnName methods, both accepting a MemberInfo. For convenience, you can also use the AutoMapResolver.For method to pass delegates for determining the names/positions, rather than implement the interface.

Constructor mapping

Define<T>() does not need your class to have a parameterless constructor. If it hasn't got one, FlatFiles builds it by handing its constructor the values it parsed, matching each parameter to a mapped member by name, ignoring case.

That means a positional record maps with nothing extra:

public record Customer(int CustomerId, string Name, DateTime Created, decimal AverageSales);

var mapper = DelimitedTypeMapper.Define<Customer>();
mapper.Property(c => c.CustomerId).ColumnName("customer_id");
mapper.Property(c => c.Name).ColumnName("name");
mapper.Property(c => c.Created).ColumnName("created").InputFormat("yyyyMMdd");
mapper.Property(c => c.AverageSales).ColumnName("avg_sales");

The same applies to a class whose properties are get-only, which nothing could fill before, and to one that checks what it is given rather than being built empty and assigned afterwards:

public class Customer
{
    public Customer(int customerId, string name)
    {
        ArgumentOutOfRangeException.ThrowIfNegative(customerId);
        CustomerId = customerId;
        Name = name;
    }

    public int CustomerId { get; }
    public string Name { get; }
}

The rules are worth knowing before you rely on them:

• If your class has a parameterless constructor, FlatFiles uses it and assigns the properties afterwards, exactly as it always has. Nothing about an existing mapping changes.
• If you passed a factory to Define or UseFactory, you have said how the class is built and FlatFiles does not look for a constructor.
• Otherwise the greediest constructor whose parameters all match mapped members wins, so a class offering both a full constructor and a partial one is built as completely as the record allows.
• A member the constructor takes is not assigned again afterwards. One it does not take is assigned as usual, so you can mix a constructor with settable properties.
• If nothing matches — a parameter named something no mapped member is called, or one whose type will not take what the member parses to — you get a FlatFileException naming the class, the constructors it tried and the members that were mapped.

If your entity is a struct or a record struct, it can be written whichever way it is built, but it can only be read where its constructor takes every mapped member — which is what a readonly record struct does, and what the rules above already arrange. A value type is handed to the deserialiser by value, so anything assigned to it after it is built is assigned to a copy and lost. FlatFiles raises a FlatFileException naming the type rather than handing you empty entities.

One thing to be aware of if you validate in a constructor: when it rejects a record, its own exception reaches you. It is not wrapped in a RecordProcessingException, so a RecordError handler will not skip the record for you. That matches how FlatFiles treats any failure while building an entity, including a CustomMapping reader that throws.

Attribute mapping

Where a class exists to mirror a file, saying so beside the members is often plainer than writing the mapping out somewhere else. DefineFromAttributes builds the mapping from the attributes on the type:

public class Customer
{
    [Column( "customer_id", Order = 0 )]
    public int CustomerId { get; set; }

    [Column( "name", Order = 1 )]
    public string Name { get; set; }

    [Column( "created", Order = 2, Format = "yyyyMMdd" )]
    public DateTime Created { get; set; }
}

var mapper = DelimitedTypeMapper.DefineFromAttributes<Customer>();

A member without [Column] is not mapped. The column's type comes from the member's type, exactly as it does for an auto-mapped file, so the attribute carries only what the member cannot say for itself: a name, an order and a format.

Order is explicit on purpose. Reflection does not promise the order it reports members in, so a member that gives an Order is placed by it and one that does not keeps the order reflection gave it, after all of them. The source generator could read declaration order from your source, and deliberately does not: generation can be turned off, and a mapping that changed shape depending on whether the generator ran would make that switch unsafe.

Fixed-length files are where this earns its keep, because a width belongs beside the member it describes:

[IgnoredWindow( 2, 4 )]                        // four characters of filler no member maps to
public class Transaction
{
    [Column( Order = 0 ), Window( 12 )]
    public string Reference { get; set; }

    [Column( Order = 1, Format = "0.00" ), Window( 10, Alignment = FixedAlignment.RightAligned, FillCharacter = '0' )]
    public decimal Amount { get; set; }

    [Column( Order = 3, Format = "yyyyMMdd" ), Window( 8 )]
    public DateTime Booked { get; set; }
}

var mapper = FixedLengthTypeMapper.DefineFromAttributes<Transaction>();

A fixed-length mapping requires both Order and [Window] on every member, because a fixed-length record is its order and a column that will not say how wide it is cannot be placed. A stretch of the record no member maps to is declared on the class with [IgnoredWindow], which takes its own place in the order.

What comes back is an ordinary mapper. Anything the attributes do not cover stays fluent, and a later call wins:

var mapper = DelimitedTypeMapper.DefineFromAttributes<Customer>();
mapper.Property( c => c.Name ).NullFormatter( NullFormatter.ForValue( "NULL" ) );

Three things worth knowing:

• An attribute-mapped entity is found by the source generator, so it keeps the accessors that make a mapped read and write cost the same under Native AOT as anywhere else.
• An enum column writes its numeric value unless you give it a formatter, which the attributes do not. Set one fluently if you want the name.
• Anything the attributes cannot express refuses rather than guesses: a type with no [Column] members at all, two columns claiming the same order, a fixed-length member missing its order or its window, or a member of a type no column reads. Each throws a FlatFileException naming the type and the member.

Schemas

Under the hood, type mapping internally defines a schema, giving each column a name, order and type in the flat file. You can get access to the schema by calling GetSchema on the mapper.

You can work directly with schemas if you don't plan on using the type mappers. For instance, this is how we would define a CSV file schema:

var schema = new DelimitedSchema();
schema.AddColumn(new Int64Column("customer_id"))
      .AddColumn(new StringColumn("name"))
      .AddColumn(new DateTimeColumn("created") { InputFormat = "yyyyMMdd", OutputFormat = "yyyyMMdd" })
      .AddColumn(new DoubleColumn("avg_sales") { OutputFormat = "N2" });

Or, if the schema is for a fixed-length file:

var schema = new FixedLengthSchema();
schema.AddColumn(new Int64Column("customer_id"), 10)
  .AddColumn(new StringColumn("name"), 255)
  .AddColumn(new DateTimeColumn("created") { InputFormat = "yyyyMMdd", OutputFormat = "yyyyMMdd" }, 8)
  .AddColumn(new DoubleColumn("avg_sales") { OutputFormat = "N2" }, 10);

The FixedLengthSchema class is the same as the DelimitedSchema class, except it associates a Window to each column. A Window records the Width of the column in the file. It also allows you to specify the Alignment (left or right) in cases where the value doesn't fill the entire width of the column (the default is left aligned). The FillCharacter property can be used to say what character is used as padding. You can also set the TruncationPolicy to say whether to chop off the front or the back of values that exceed their width.

Note Some fixed-length files may have columns that are not used. The fixed-length schema doesn't provide a way to specify a starting index for a column. Look at the Ignored Fields section below to learn about ways to to handle this.

Column types

Every column below reads and writes one .NET type. The type mappers' Property method has an overload for each of these types, and a second for the nullable form of each value type, and the dynamic mappers have a matching ...Property method, so you rarely name a column class yourself unless you are building a schema by hand.

Column	.NET type	Settings of its own
BooleanColumn	bool	TrueString, FalseString
ByteColumn	byte	the number settings below
SByteColumn	sbyte	the number settings below
Int16Column	short	the number settings below
UInt16Column	ushort	the number settings below
Int32Column	int	the number settings below
UInt32Column	uint	the number settings below
Int64Column	long	the number settings below
UInt64Column	ulong	the number settings below
SingleColumn	float	the number settings below
DoubleColumn	double	the number settings below
DecimalColumn	decimal	the number settings below
StringColumn	string	Trim
CharColumn	char	AllowTrailing
CharArrayColumn	char[]	Trim
ByteArrayColumn	byte[]	Encoding
DateTimeColumn	DateTime	InputFormat, OutputFormat, FormatProvider
DateTimeOffsetColumn	DateTimeOffset	InputFormat, OutputFormat, FormatProvider
DateOnlyColumn	DateOnly	InputFormat, OutputFormat, FormatProvider
TimeOnlyColumn	TimeOnly	InputFormat, OutputFormat, FormatProvider
TimeSpanColumn	TimeSpan	InputFormat, OutputFormat, FormatProvider
GuidColumn	Guid	InputFormat, OutputFormat
EnumColumn<TEnum>	TEnum	Parser, Formatter

The numeric columns share FormatProvider, NumberStyles and OutputFormat. A date or time column with an InputFormat parses exactly, using ParseExact; without one it parses generically, which accepts more but is slower and culture-dependent. EnumColumn<TEnum> reads a member's name or its number without regard to case unless you give it a Parser, and writes its number unless you give it a Formatter.

Every column also carries the settings that belong to all of them: ColumnName, IsNullable, DefaultValue and NullFormatter, covered under Handling Nulls, and the OnParsing, OnParsed, OnFormatting and OnFormatted hooks for stepping in either side of a value being converted.

If you reach for OnParsing, reach for OnParsingSpan instead. It answers the same question -- change this value before the column parses it -- but it is handed the value where it sits in the record along with a buffer to write its answer into, rather than being handed a string and asked to return another:

mapper.Property(x => x.Amount, new Window(9))
    .OnParsingSpan((context, value, destination) => Unpack(value, destination));

Return how many characters you wrote, or SpanParsingHooks.Unchanged to have the value parsed exactly as it lies, which copies nothing at all. If the buffer is too small, return SpanParsingHooks.NeedsLength(n) and you will be called again with one at least that long; the buffer comes from the array pool either way, so neither attempt allocates. Neither span outlives the call, so do not keep hold of them.

The reason to prefer it is not only that it allocates nothing. OnParsing deals in strings, so a column carrying one cannot be read straight onto an entity -- the mapping falls back to parsing every record into an array of objects, and one hook on one column does that to the whole mapping. OnParsingSpan does not. On a fixed-length read of twenty thousand records with ten packed decimal columns that was 860 bytes a record against 3,891, and roughly twice as quick.

Where both are set, OnParsing wins and the span hook is not called: a column that has to build a string for one of them may as well hand it to both.

TimeSpanColumn also has static methods for files that store a duration as a plain number: TimeSpanColumn.FromSeconds(new DoubleColumn("elapsed")) and its FromDays, FromHours, FromMinutes, FromMilliseconds and FromTicks counterparts wrap a numeric column and convert what it reads.

Four columns do not map to a value of their own:

Column	What it does
IgnoredColumn	Occupies a position in the file whose value is thrown away. See Ignored Fields.
RecordNumberColumn	Supplies the record's number rather than reading anything. See Metadata.
DelimitedComplexColumn	Reads a whole delimited record out of a single field, using its own DelimitedOptions.
FixedLengthComplexColumn	Reads a whole fixed-length record out of a single field, using its own FixedLengthOptions.

Creating your own columns

The column types that ship with FlatFiles cover the usual types, but you can add your own by deriving from ColumnDefinition<T> and implementing two methods:

protected override T OnParse(IColumnContext? context, string value);

protected override string OnFormat(IColumnContext? context, T value);

That is all a column needs, and it is all any column needed before FlatFiles 7.1.0. Everything around them -- trimming, null handling, default values and the OnParsing/OnParsed hooks -- is applied for you.

Two further overloads are optional, and let a column avoid creating a string for its value:

protected override T OnParse(IColumnContext? context, ReadOnlySpan<char> value);

protected override void OnFormat(IColumnContext? context, T value, IBufferWriter<char> destination);

The readers hand a column the characters of its value where they can, rather than a string cut out of the record, and the writers let a column format straight into the record buffer rather than return a string to be copied. Implement these and no string is created for the value in either direction. Leave either one out and FlatFiles falls back to the string overload, so a column written against an earlier version keeps working unchanged.

Delimited Files

If you are working with delimited files, such as comma-separated (CSV) or tab-separated (TSV) files, you want to use the DelimitedTypeMapper. Internally, the mapper uses the DelimitedReader and DelimitedWriter classes, both of which work in terms of raw object arrays. In effect, all the mapper does is map the values in the array to the properties in your data objects. These classes read data from a TextReader, such as a StreamReader or a StringReader, and write data to a TextWriter, such as a StreamWriter or a StringWriter. Internally, the mapper will build a DelimitedSchema based on the property/column configuration; this is where you customise the schema to match your file format. For more global settings, there is also a DelimitedOptions object that allows you to customise the read/write behaviour to suit your needs.

Within delimited files, fields can be surrounded with double quotes ("). This way they can include the separator within the field. You can override the "quote" character in the DelimitedOptions class, if needed. The DelimitedOptions class supports a Separator property for specifying the string/character that separates your fields. A comma (,) is the default separator; you'd obviously want to change this to tab (\t) for a TSV file.

The RecordSeparator property specifies what string/character is used to separate records. By default, FlatFiles will look for \r, \n or \r\n, which are the default line separators for Mac, Linux and Windows, respectively. When writing files, Environment.NewLine is used by default; this means by default you'll get different output if you run the same code on different platforms. If you need to target a specific platform, be sure to set the RecordSeparator property explicitly.

When working directly with the DelimitedReader class, the IsFirstRecordSchema option tells the reader to treat the first record in the file as the schema. This is useful when you don't know the schema ahead of time or you are just dumping files into staging tables. If you provide a schema, this setting tells the reader to simply skip the first record. If you are using it to determine the schema from the file, the reader treats every column as a string - it doesn't try to interpret the type from the data. While occasionally useful, you will normally want to provide a schema to give you the most control over the parsing process - that's what FlatFiles is good at!

When working directly with the DelimitedWriter class, setting IsFirstRecordSchema to true option causes a header to be written to the file upon writing the first record.

A record with more or fewer values than the schema declares is governed by ShortRecordHandling and LongRecordHandling, described under Records that do not fit the schema below. The second of those is worth reading before you rely on the default. If you provide a schema and the file has a header, see also Matching columns to the header, which decides whether that header is checked against your schema or thrown away.

Splitting one record's text

If you already hold one record's text - a line taken out of a file whose records are not all the same shape, say - DelimitedReader.ParseRecord gives you its values without standing a reader up around it:

string[] values = DelimitedReader.ParseRecord("a,\"b,c\",d");   // a | b,c | d

It takes a DelimitedOptions too, so a tab-separated line or a different quote character works the same way.

It reads a record, not a line, and the difference is the part to be careful with. A record separator inside a quoted value belongs to the value, so text that looks like two lines may be one record:

var text = "a,\"b" + Environment.NewLine + "c\",d";
string[] values = DelimitedReader.ParseRecord(text);   // three values; the second holds a newline

Text holding more than one record is refused with an ArgumentException rather than answered with the first, because a caller who thought they had one record and did not should hear about it. A single trailing record separator is not a second record. Text holding no record at all comes back empty, which is not the same as a record of empty values: "" gives no values, and "," gives two.

Fixed Length Files

If you have a file with fixed length columns, you will want to use the FixedLengthTypeMapper class. Internally, the mapper uses the FixedLengthReader and FixedLengthWriter classes, both of which work in terms of raw object arrays. In effect, all the mapper does is map the values in the array to the properties in your data objects. These classes read data from a TextReader, such as StreamReader or StringReader, and write data to a TextWriter, such as StreamWriter or StringWriter. Internally, the mapper will build a FixedLengthSchema based on the property/column configuration; this is where you customise the schema to match your file format. For more global settings, there is also a FixedLengthOptions object that allows you to customise the read/write behaviour to suit your needs.

Since each column has a fixed length, FlatFiles provides configuration options to specify how to handle values that are too short or too long: FillCharacter, Alignment and TruncationPolicy. FillCharacter specifies what character is used to pad values on the left or the right, using space ( ) by default. You can configure whether the padding should go to the left or the right using the Alignment property, putting padding to the right by default (LeftAligned). TruncationPolicy tells FlatFiles how to crop values that exceed the width of their column when writing out to a file, removing leading characters by default. These options can be specified globally in the FixedLengthOptions object or overridden at the column level using the Window object.

The RecordSeparator property specifies what string/character is used to separate records. By default, FlatFiles will look for \r, \n or \r\n, which are the default line separators for Mac, Linux and Windows, respectively. When writing files, Environment.NewLine is used by default; this means by default you'll get different output if you run the same code on different platforms. If you need to target a specific platform, be sure to set the RecordSeparator property explicitly.

By default, FlatFiles assumes there is a separator string/character between each record. If you set the HasRecordSeparator to false, FlatFiles will read the next record immediately following the last character of the previous record. When writing, it will not insert a separator, writing immediately after the last character of the previous record.

If the FixedLengthOptions's IsFirstRecordHeader property is set to true, the first record in the file will be skipped when reading. Unlike the DelimitedReader, you must always provide a schema for fixed-length files, since the width of the columns cannot be determined from the file format. When writing, a header will be written to the file upon writing the first record.

IsRaggedRight reads a ragged-right file, where every column but the last has a fixed width and the last runs from its offset to the end of the line, however long or short: the last column takes whatever follows the other windows, a record that ends before the last column is read as far as it goes, with a window it ends inside taking the characters that are there and a window it never reaches parsing as null. When writing with IsRaggedRight, the last column is written as formatted, neither padded nor truncated to its window, so a file read ragged and written ragged keeps its shape.

var options = new FixedLengthOptions { IsRaggedRight = true };
var reader = new FixedLengthReader(new StreamReader(path), schema, options);

Anything else about a record that does not fit its schema is said with ShortRecordHandling and LongRecordHandling, described under Records that do not fit the schema below, which the delimited reader has too.

Matching columns to the header

When you give a reader a schema and the file has a header, the header is read and discarded: each value belongs to the column in the same position. That is still the default, because a file with no header has nothing else to go on - but it means a file whose columns were reordered since the schema was written is read straight into the wrong columns, and nothing says so. HeaderMatching takes each value from under the heading of the same name instead.

var options = new DelimitedOptions
{
    IsFirstRecordSchema = true,
    HeaderMatching = HeaderMatching.ByName
};

ByName requires every column the schema declares to appear in the header, and does not read a heading the schema does not declare. ByNameWhereFound is the same except that a column the header does not carry is read as empty rather than treated as an error. Both need IsFirstRecordSchema set, since there is otherwise no header to match.

What you get back does not move. The values are taken from different places in the record, but they are handed to you in the order the schema declares them, so GetValues(), GetOrdinal and a type mapper's properties all mean what they meant before. Only where a value is read from changes.

A few things worth knowing:

• Names are compared with HeaderComparer, which is StringComparer.OrdinalIgnoreCase by default - the same comparison the schema itself uses to look a column up by name. Set it to StringComparer.Ordinal if the file's capitalisation is meaningful.
• The header says how wide a record should be, not the schema, so a file carrying columns your schema does not declare is not a file of long records. ShortRecordHandling and LongRecordHandling still apply, judged against the header.
• A metadata column such as RecordNumberColumn takes nothing from the record, so it needs no heading.
• An IgnoredColumn with a name is matched like any other column and still yields nothing. One without a name is refused, because it exists only to hold a position and positions stop mattering here.
• A name the header carries twice is refused only if a column actually asks for it.
• It needs a schema rather than a DelimitedSchemaSelector: a header describes the file, while a selector chooses a schema for each record. The two together are refused.

This is a delimited feature. A fixed-length header cannot be partitioned without knowing the widths in the order the file has them, which is the thing the header would be telling you.

Records that do not fit the schema

Both readers can be told what a record carrying more or less than the schema asks for means, in the same two words. A delimited record is short when it has fewer values than the schema has columns to fill and long when it has more; a fixed-length one is short when it ends before the last window does and long when characters follow that window.

var options = new DelimitedOptions
{
    ShortRecordHandling = ShortRecordHandling.Pad,
    LongRecordHandling = LongRecordHandling.Refuse
};

ShortRecordHandling is Refuse by default: the record is reported as a RecordProcessingException and not returned, so a RecordError handler can skip it and carry on. Pad reads the columns the record does not reach as empty, which their null handling turns into null - which suits a file whose trailing columns are simply left off when they have no value.

LongRecordHandling is Discard by default, which is what both readers have always done: what the record carries beyond the schema is not read. This is the setting worth knowing about, because discarding is silent. A delimited record carrying a separator inside an unquoted value has one value too many, so every value after that point is read one column to the left, and whether anything notices depends on whether a shifted value happens to reach a column that will not parse it. Refuse reports the record instead, with the same record context a short record gets, so both can be handled the same way. It is not the default because a file whose records carry trailing content that was always ignored would otherwise stop reading.

On FixedLengthOptions, LongRecordHandling is what says this; it replaced an IsLongRecordRejected property that said the same thing, which was obsolete through 8.7.0 and removed in 9.0.0. IsRaggedRight answers both questions by itself and overrides both: no ragged-right record is too long, and one that ends early is read as far as it goes. Where the records are short but the last column still has the width the schema gives it, use ShortRecordHandling.Pad rather than IsRaggedRight.

Character Encoding

FlatFiles works in characters rather than bytes. You can hand a reader a TextReader you opened, in which case the character encoding is decided by the StreamReader you open, before any FlatFiles class is involved, and the same schema and options work with any encoding.

using (var reader = new StreamReader(File.OpenRead(@"C:\path\to\file.csv"), Encoding.UTF8))
{
    var csvReader = new DelimitedReader(reader, schema);
    // ...
}

Reading can take the stream instead, in which case FlatFiles opens the reader for you:

using (var stream = File.OpenRead(@"C:\path\to\file.csv"))
{
    var csvReader = new DelimitedReader(stream, schema);
    // ...
}

The same overload is on FixedLengthReader, on Read and GetReader for both type mappers, and on GetAutoMappedReader. Each takes an optional Encoding, and UTF-8 is used when none is given.

Three things it decides for you, which are the reasons it exists:

• A byte order mark is always honoured and always taken off, whatever encoding you asked for. A file that says what it is is read as what it says, and a mark left in the text becomes part of the first value - turning Bob into \uFEFFBob, or a header column into one that nothing matches. That is a quiet failure and it is the main thing this saves you from.
• The stream is left open. A reader does not own what it was handed, and none of these readers is disposable, so closing the stream stays the job of whoever opened it.
• It is a convenience, not a saving. Decoding a megabyte and a third of UTF-8 costs about a fifth of a millisecond and a byte a record, because a StreamReader reuses its buffers; the same file read through one rather than through a StringReader allocates the same bytes a record either way.

Writing still takes a TextWriter, so the encoding there is yours to choose as below.

UTF-8 is the default for both StreamReader and StreamWriter. A StreamReader also detects a byte order mark by default and consumes it, so a file that begins with EF BB BF reads the same as one that does not. Keep that detection on: if you construct a reader with detectEncodingFromByteOrderMarks: false, the mark is read as text and becomes part of the first column's name ("\uFEFFId" rather than "Id"), and that column is never matched. To write a file with a mark, pass new UTF8Encoding(true) to the writer; Encoding.UTF8 writes one too, and new UTF8Encoding(false) writes none.

Windows-1252 is what most files from Windows tools are in when they are not UTF-8. It is not available in .NET by name until its provider is registered, once, at start-up:

Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
var windows1252 = Encoding.GetEncoding(1252);

ISO-8859-1 (Latin-1) is available directly as Encoding.Latin1. It agrees with Windows-1252 everywhere except the bytes 0x80 to 0x9F, which Windows-1252 uses for the euro sign, curly quotes, the trademark sign and similar, and which ISO-8859-1 leaves as control characters. A Windows-1252 file read as Latin-1 therefore looks right until the first euro sign, which comes back as U+0080. When a file is described as "Latin-1" and contains a €, it is Windows-1252.

Two consequences of the encoding sitting outside FlatFiles:

• A character the encoding cannot represent is replaced when writing, not rejected. A € written through Encoding.Latin1 arrives as ?. Choose the encoding to fit the data, or use UTF-8.
• Fixed length windows are measured in characters, not bytes. A name such as Müller fills a six-character window in every encoding, even though it is seven bytes in UTF-8, so widths taken from a byte-oriented specification are right only for single-byte encodings.

The tests in CharacterEncodingTester pin each of these behaviours, including what the misreads look like.

Handling Nulls

Each column can be marked as "nullable", using the IsNullable property. By default, all columns are nullable, meaning null is considered a valid value. Setting IsNullable to false will cause FlatFiles to throw an exception whenever a null is encountered.

Type mappers will automatically configure columns to be nullable based on the type of the property. If you need to support nulls, make sure you use nullable types for your properties, for example int? instead of int.

Default Values

When working with non-nullable columns, you can specify a default value to use whenever a null is encountered, rather than throw an exception. You simply set a custom IDefaultValue on the column. The DefaultValue class provides helper methods for generating default values. For example:

column.DefaultValue = DefaultValue.Use(0);

Or, if you are using type mappings, you can simply use the DefaultValue method:

mapper.Property(c => c.Amount)
    .ColumnName("amount")
    .DefaultValue(DefaultValue.Use(0));

Null Formatters

By default, FlatFiles will treat blank or empty strings as null. If nulls are represented differently in your file, you can set a custom INullFormatter on the column. If it is a fixed value, you can use the NullFormatter.ForValue method.

var dateColumn = new DateTimeColumn("created") 
{
    NullFormatter = NullFormatter.ForValue("NULL") 
};

Or, if you are using type mappers, you can simply use the NullFormatter method:

mapper.Property(c => c.Created)
    .ColumnName("created")
    .NullFormatter(NullFormatter.ForValue("NULL"));

You can implement the INullFormatter interface if you need to support something more complex. It has an optional overload taking a ReadOnlySpan<char>, which a reader uses to decide whether a value is null without creating a string for it; the default copies the characters and calls the string overload, so implementing only the string overload is enough.

Ignored Fields

Most of the time, we don't have control over the flat file we're working with. This usually leads to columns that our code just doesn't care about. Both type mappers and schemas expect columns to be listed in the order they appear in the document. For type mappers, this would mean defining properties in your classes that you'd never use (e.g., Ignored1, Ignored2, etc.). Another common problem with fixed-length files is that they will separate fields with pipes (|), or other characters, even though they are fixed-length.

Types mappers support the Ignored method that will tell FlatFiles to simply ignore the next column in the file. Unlike other mapper methods, it does not take a property, since the value is simply thrown away. You can optionally call ColumnName if you want to provide a column name when writing out schemas.

When working with the IFixedLengthTypeMapper, Ignored takes a Window. This is a great way to skip unused sections within the document. You can even set the FillCharacter property to insert pipes (|), or another character, between fields.

Under the hood, type mappers is adding an IgnoredColumn to the underlying schema. IgnoredColumn has a constructor to optionally specify a column name. IgnoredColumn affects the way you work with readers and writers. The readers will initially retrieve all columns from the document and then throw away any ignored values. You'll only see values for the columns that aren't ignored. Also, you do not need to provide values to the writers for ignored columns; the schema will automatically take care of writing out blanks for them. From a development perspective, it's as if those columns didn't exist in the underlying document.

Metadata

It is often useful to incorporate metadata with the records you are reading from a file. The most common example is tracking a record's line number, so users can be informed where to look in their files when something goes wrong.

Currently, the only out-of-the-box metadata column is RecordNumberColumn; however, it's really easy to create your own custom metadata columns (more on that below).

The RecordNumberColumn class provides options for controlling how the record number is generated. The IncludeSchema property indicates whether the schema or header row should be included in the count. The IncludeSkippedRecords property specifies whether to count records that are skipped.

By default, RecordNumberColumn will only count records that are actually returned, starting from 1, then 2, 3, 4, 5 and so on. If you count the schema, records will always start at 2. Including the schema and skipped records is what you probably want if you're trying to simulate line number. The only time the record # wouldn't be the same as the line # is if a record spanned multiple lines. Here's an example showing how to capture this pseudo line number:

var mapper = DelimitedTypeMapper.Define(() => new Person());
mapper.Property(x => x.Name);
mapper.CustomMapping(new RecordNumberColumn("RecordNumber")
{
    IncludeSchema = true,
    IncludeSkippedRecords = true
}).WithReader(p => p.RecordNumber);

var options = new DelimitedOptions() { IsFirstRecordSchema = true };
var results = mapper.Read(reader, options).ToArray();

Writing metadata

I've not yet come up with a reason why you'd want to write out metadata, but I provide support for it anyway (feel free to provide me an example!).

var mapper = FixedLengthTypeMapper.Define(() => new Person());
mapper.Property(x => x.Name, 10);
mapper.Ignored(1);
// No need to define a reader or a writer - the underlying schema handles writing the metadata 
mapper.CustomMapping(new RecordNumberColumn("RecordNumber") { IncludeSchema = true }, 10);
mapper.Ignored(1);
mapper.Property(x => x.CreatedOn, 10).OutputFormat("MM/dd/yyyy");

Creating your own metadata columns

FlatFiles provides the MetadataColumn<T> abstract base class to allow you to create your own metadata columns. To implement this interface, you must implement the methods:

T OnParse(IColumnContext context);

string OnFormat(IColumnContext context);

Within IColumnContext, the following information is currently provided:

• PhysicalIndex - The index of the column in the file.
• LogicalIndex - The index of the column, excluding ignored columns.
• RecordContext - Details about the record this column pertains to.
  • PhysicalRecordNumber - The actual number of records read from the file.
  • LogicalRecordNumber - The number of records that have not been skipped. This count does not yet include the current record.
  • Record - The text of the record. Always available for a fixed-length file; for a delimited file, only when PreserveRecordText is set.
  • Values - The raw, unparsed values of the record. This is a copy, built the first time you ask for it, so writing to it does not change what the remaining columns parse. To substitute a value before the record is parsed, use the column's OnParsing hook, or the RecordRead and RecordPartitioned events described under Skipping Records.
  • ExecutionContext - Details about the current read/write operation.
    • Schema - The schema being used to parse the file.
    • Options - The options passed to the reader/writer.

Skipping Records

If you work directly with DelimitedReader or FixedLengthReader, you can call Skip to arbitrarily skip records in the input file. However, you often need the ability to inspect the record to determine whether it needs skipped. But what if you are trying to skip records because they can't be parsed? If you need more control over what records to skip, FlatFiles provides events for inspecting records during the parsing process. These events can be wired up whether you use type mappers or are working directly with readers.

Parsing a record goes through the following life-cycle:

1. Read text until a record terminator (usually a newline) is found.
2. For fixed-length records, partition the text into string columns based on the configured windows.
3. Convert the string columns to the designated column types, as defined in the schema.

For CSV files, breaking a record into columns is automatically performed while searching for the record terminator. Prior to trying to convert the text to ints, date/times, etc., FlatFiles provides you the opportunity to inspect the raw string values and skip records.

The DelimitedReader class provides a RecordRead event, which allows you to skip unwanted records. For example, you could use the code below to find and skip CSV records missing the necessary number of columns:

reader.RecordRead += (sender, e) =>
{
    e.IsSkipped = e.Values.Length < 10;
};

Fixed-length files come in two flavors: those with and without record terminators. If there is no record terminator, the assumption is all records are the same length. Otherwise, each record can be a different length. For that reason, FlatFiles provides an extra opportunity to filter out records prior to splitting the text into columns. This is useful for filtering out records not meeting a minimum length requirement or those using a character to indicate something like comments.

The FixedLengthReader class provides the RecordRead event, which allows you to skip unwanted records. For example, you could use the code below to find and skip records starting with a # symbol:

reader.RecordRead += (sender, e) => 
{
    e.IsSkipped = e.Record.StartsWith("#");
};

e.RecordContext tells the handler which record it is being offered, so a handler that skips a line can report which line it skipped. This event is raised before a schema has been chosen, so that context carries no schema - what it carries is the record number.

Similar to CSV files, you can also filter out fixed-length records after they are broken into columns. However, it is important to note that the record is expected to fit the configured windows.

Again, the FixedLengthReader class provides the RecordPartitioned event, which allows you to skip unwanted records. For example, you could use the code below to find and skip records whose third column has a flag:

reader.RecordPartitioned += (sender, e) => 
{
    e.IsSkipped = e.Values[2] == "ERROR";
};

Comments and blank lines

Two of the things those handlers most often get written for are on the options instead, where they cost nothing:

var options = new DelimitedOptions
{
    CommentPrefix = "#",
    IsBlankRecordSkipped = true
};

Both are on FixedLengthOptions too, and both are answered from the record before anything is parsed. A record passed over this way is not a record: it never reaches a schema selector, a RecordRead handler, or the header, so a file whose header sits under a banner of comments is read as though the banner were not there.

That is the reason to prefer them over a handler rather than a matter of taste. Attaching a RecordRead handler to a DelimitedReader makes every record in the file copy its values out of the parser's buffer, whether the handler looks at them or not, because the handler is given an array it is allowed to write into. The options look at the record where it lies.

Three details worth knowing:

• CommentPrefix is matched ordinally against the start of the record, before any trimming a column would do, so # indented is not a comment.
• IsBlankRecordSkipped is off by default, because a blank line in a delimited file is a record of one empty value and some files mean it.
• A record passed over still counts towards the physical record number, which is where a record sits in the file. An error that reports one is only useful if it agrees with what a text editor shows.

One name, two different events

RecordRead means different things on the two readers, which is worth getting straight before you go looking for a hook that is not there. Lined up by what they actually do:

event	raised	given
FixedLengthReader.RecordRead	before a schema is chosen	the record's text
FixedLengthReader.RecordPartitioned	after a schema is chosen	the record's values
DelimitedReader.RecordRead	after a schema is chosen	the record's values

So DelimitedReader.RecordRead is the counterpart of FixedLengthReader.RecordPartitioned, not of the fixed-length event sharing its name. They take the same arguments and are raised at the same point in the life-cycle. What the delimited reader has no counterpart for is the first row: it offers you no hook before a schema has been chosen.

That matters as soon as a file holds more than one kind of record, because by the time the delimited reader raises RecordRead, a record no schema matched has already been reported as an error. No arrangement of handlers makes both readers pass over the same lines: the delimited one would have to be told what every other section looks like purely to keep it quiet.

Where the records you want to ignore can be recognised at all, say so on the selector instead - see Passing records over. That decides it the same way in both readers, and before either raises an error, which is what the missing row would otherwise have been for.

Error Handling

The reader and writer classes support two events for handling errors: RecordError and ColumnError. The ColumnError event is raised whenever an error occurs while reading/writing a column; for example, when a value can't be parsed. In that case, an instance of ColumnErrorEventArgs will be sent to the listener(s), which provides access to the context (ColumnContext), the value that caused the error (ColumnValue) and the exception that was thrown (Exception).

Furthermore, the IsHandled property can be set to true to inform FlatFiles that the exception should not be propagated. In that case, a value should be provided for the Substitution property. This property is an object, so it's important to make sure the substituted value is the same type as the column or a runtime error may occur.

The following code could be used to detect issues with a file so that errors can be reported back to a user:

public class ErrorDetail
{
    public int RecordNumber { get; set; }
    public string ColumnName { get; set; }
    public string ErrorValue { get; set; }
    public string ErrorMessage { get; set; }
} 
//...
var details = new List<ErrorDetail>();
var csvReader = new DelimitedReader(textReader, schema);
csvReader.ColumnError += (sender, e) =>
{
    var columnContext = e.ColumnContext;
    var detail = new ErrorDetail()
    {
        RecordNumber = columnContext.RecordContext.PhysicalRecordNumber,
        ColumnName = columnContext.ColumnDefinition.ColumnName,
        ErrorValue = e.ColumnValue?.ToString(),
        ErrorMessage = e.Exception.InnerException?.Message ?? e.Exception.Message
    };
    details.Add(detail);
    e.IsHandled = true;
    e.Substitution = null;  // May not work for non-nullable value types
};

If a column-level exception is not handled, the exception propagates. This, along with other record-level errors, will cause the RecordError event to be raised. Listeners will be sent an instance of RecordErrorEventArgs, which provides access to the context (RecordContext) and the exception that was thrown (Exception). Similarly, it provides an IsHandled property, that when set to true, will prevent the exception from propagating. In the case of record-level errors, this causes the record to be skipped while reading or writing.

Files Containing Multiple Schemas

Some flat file formats will contain multiple schemas. Often, data appears within "blocks" with a header and perhaps a footer. It can be extremely useful to parse each type of record using a different schema and return them in the same order they appear in the file.

FlatFiles provides support for these file formats using the SchemaSelector and TypeMapperSelector classes. Once you define the schemas or type mappers, you can register them with the selector.

var selector = new DelimitedTypeMapperSelector();
var dataMapper = getDataTypeMapper();
selector.When(x => x.Length == 10).Use(dataMapper);
selector.When(x => x.Length == 2).Use(getHeaderTypeMapper());
selector.When(x => x.Length == 3).Use(getFooterTypeMapper());
selector.WithDefault(dataMapper);

Each type mapper is associated with a predicate, which accepts the preprocessed record. If the predicate succeeds, that type mapper will be used to parse the record. Each predicate is tested in the order it is registered, so be sure to make your predicates smart enough to correctly identify the record type.

Note: You'll might have noticed dataMapper is registered twice, once up-front with a specific condition and then as the default. This isn't necessary; however, making sure the most common schema is tried first may help performance in some cases.

Once your selector is configured, you can call GetReader, passing in the TextReader and the options object. From there you can register events and read the objects from the file. Since different types can be returned, the reader will return instances of object.

If you want to work directly with schemas and readers, you can build a SchemaSelector by registering schemas with predicates in a similar fashion. The DelimitedReader and FixedLengthReader classes provide a constructor accepting a SchemaSelector. Note that whenever you work with selectors, calls to GetSchema will return null.

var selector = new DelimitedSchemaSelector();
var recordSchema = getDataSchema();
selector.When(values => values.Length == 10).Use(recordSchema);
selector.When(values => values.Length == 2).Use(getHeaderSchema());
selector.When(values => values.Length == 3).Use(getFooterSchema());
selector.WithDefault(recordSchema);

var reader = new DelimitedReader(fileStream, selector);
while (reader.Read())
{
    object[] values = reader.GetValues();
    processRecord(values);
}

Choosing by the record's text

A predicate registered with When is given the record's values, which means the delimited reader has to split every record and copy each value out of its buffer before it can decide what the record is - whether the predicate looks at more than the first value or not. WhenText is given the record's characters instead, and costs neither:

var selector = new DelimitedTypeMapperSelector();
selector.WhenText(record => record.StartsWith("PER")).Use(getPersonMapper());
selector.WhenText(record => record.StartsWith("ORD")).Use(getOrderMapper());

The predicate is a RecordTextPredicate, which takes a ReadOnlySpan<char> rather than a string, so nothing is allocated to answer it. The span is only valid for the length of the call - do not keep hold of it.

Where every predicate on a selector reads the text, and every mapping behind it can be read without boxing, the reader takes each record straight onto its entity, which is the path a single-mapping delimited read and a fixed-length selector already use. Across the three delimited samples the integration harness measures, that is 8,911 bytes a record against 193, 2,107 against 192, and 5,305 against 174 - between eleven and forty-six times, widening with the number of columns, because what the values path costs is a string per column per record and an array to hold them. A text predicate reads exactly level with a plain single mapping. One predicate that wants the values puts every record back on the values path, so this is all or nothing per selector - which is the choice it is there to give you: match on text and keep the faster path, or match on values and pay for them.

Both DelimitedSchemaSelector and DelimitedTypeMapperSelector have it, and the two kinds of predicate can be mixed on one selector where some layouts really can only be told apart by their values.

The fixed-length selectors already take the record's text, because a fixed-length reader has to have it in hand to partition the record at all.

Passing records over

A record nothing matches is an error. Sometimes that is right, and sometimes the file simply carries sections meant for somebody else and you want them gone. Skip says so:

var selector = new DelimitedSchemaSelector();
selector.WhenText(record => record.StartsWith("#")).Skip();
selector.WhenText(record => record.StartsWith("PER")).Use(getPersonSchema());

A record a Skip matched is passed over without being read and without being reported. A record nothing matched at all is still an error, as before - the two are different answers and stay different.

It reads as When(...).Skip() or WhenText(...).Skip(), beside Use, and FixedLengthSchemaSelector has it too:

var selector = new FixedLengthSchemaSelector();
selector.When(record => record.StartsWith("#")).Skip();
selector.When(record => record.StartsWith("PER")).Use(getPersonSchema());

Predicates are asked in the order they were registered, skips included. A skip registered before a When takes records from it; one registered after it does not. Register the narrow conditions first, as you would with Use.

OnMatch works on a skip as it does on a Use, which is how you count what went past:

var ignored = 0;
selector.WhenText(record => record.StartsWith("#")).Skip().OnMatch(() => ++ignored);

This is the answer to the delimited reader having no hook before a schema is chosen, described under One name, two different events: a selector decides the same way in both, and decides before either reader reports anything.

If you want to create multi-schema files, there are "injector" equivalents for each "selector" class. For example:

var selector = new FixedLengthTypeMapperInjector();
selector.WithDefault(getRecordTypeMapper());
selector.When<HeaderRecord>().Use(getHeaderTypeMapper());
selector.When<FooterRecord>().Use(getFooterTypeMapper());

var stringWriter = new StringWriter();
var writer = injector.GetWriter(stringWriter);
writer.Write(new HeaderRecord() { BatchName = "First Batch", RecordCount = 2 });
writer.Write(new DataRecord() { Id = 1, Name = "Bob Smith", CreatedOn = new DateTime(2018, 06, 04), TotalAmount = 12.34m });
writer.Write(new DataRecord() { Id = 2, Name = "Jane Doe", CreatedOn = new DateTime(2018, 06, 05), TotalAmount = 34.56m });
writer.Write(new FooterRecord() { TotalAmount = 46.9m, AverageAmount = 23.45m, IsCriteriaMet = true });

The When method accepts a predicate if you need more than just the record type to decide what type mapper/schema to use.

Custom Mapping

The Property methods are best when your file's schema pretty much matches one-to-one with your classes. One column goes to one property. Frequently, though, your classes are more structured than your flat files. Starting with FlatFiles 3.0, you can provide your own mapping logic to control serialising and deserialising your objects.

First, let's see how to use the CustomMapping method to simulate the Property methods:

var mapper = DelimitedTypeMapper.Define(() => new Person());
mapper.CustomMapping(new StringColumn("FirstName"))
    .WithReader(p => p.FirstName)
    .WithWriter(p => p.FirstName);

This mapping tells FlatFiles to use a StringColumm to read/write the values from the file. It then says to read the values into and write out of the FirstName property of your class.

It's important to note that the type returned by the IColumnDefinition must exactly match the type of the property. You must deal with nulls and conversions yourself. There are several other overloads that provide more control -- this particular example would be better handled sticking to the Property methods.

Here is a more complicated example where two columns are used to build a Geolocation property.

public class Geolocation { public decimal Longitude; public decimal Latitude; }
public class Person { public Geolocation Location { get; set; } }
//...
var mapper = DelimitedTypeMapper.Define(() => new Person() 
{ 
    Location = new Geolocation() 
});
mapper.CustomMapping(new DecimalColumn("Longitude"))
    .WithReader((p, v) => p.Location.Longitude = (decimal)v)  // long way
    .WithWriter(p => p.Location.Longitude);
mapper.CustomMapping(new DecimalColumn("Latitude"))
    .WithReader(p => p.Location.Latitude)  // short way
    .WithWriter(p => p.Location.Latitude);

Finally, here is an example configuration, where multiple columns are stored in a collection.

public class Contact
{
    public int Id { get; set; }
    public string Name { get; set; }
    public List<string> PhoneNumbers { get; set; } = new List<string>();
}
//...
var mapper = FixedLengthTypeMapper.Define(() => new Contact());
mapper.CustomMapping(new Int32Column("Id"), 10)
    .WithReader(c => c.Id)
    .WithWriter(c => c.Id);
mapper.CustomMapping(new StringColumn("Name"), 10)
    .WithReader(c => c.Name)
    .WithWriter(c => c.Name);
mapper.CustomMapping(new StringColumn("Phone1"), 12)
    .WithReader(PhoneReader)
    .WithWriter(c => c.PhoneNumbers.Count > 0 ? c.PhoneNumbers[0] : null);
mapper.CustomMapping(new StringColumn("Phone2"), 12)
    .WithReader(PhoneReader)
    .WithWriter(c => c.PhoneNumbers.Count > 1 ? c.PhoneNumbers[1] : null);
//...
public void PhoneReader(Contact contact, string phoneNumber)
{
    if (phoneNumber != null)
    {
        contact.PhoneNumbers.Add(phoneNumber);
    }
}

In benchmarks, using CustomMapping is only slightly slower than using Property, making it a great option when you need a little extra control.

There are versions of WithReader and WithWriter that provide contextual information (IColumnContext), so you can access metadata while reading and writing. The WithWriter method also provides an overload passing along the underlying array being written to, so you can write to multiple columns simultaneously or inspect previously serialised values.

Runtime Mapping

Even if you don't know the type of a class at compile time, it can still be beneficial to use the type mappers to populate these objects from a file. Or, if you are working in a language without support for expression trees, you'll be glad to know FlatFiles provides an alternative way to configure type mappers.

The code below illustrates how you could define a mapper for a type that is only known at runtime:

// Assume there is a runtime-generated type called "entityType" and a "TextReader".
var mapper = DelimitedMapper.DefineDynamic(entityType);
mapper.Int32Property("Id");
mapper.StringProperty("Name");
var entities = mapper.Read(reader).ToArray();
// Do something with the entities.

Disabling Optimisation

FlatFile's type mappers can serialise and deserialise extremely quickly by generating code at runtime, using classes in the System.Reflection.Emit namespace. For most of us, that's awesome news because it means mapping values to and from your entities is almost as fast as if you had done the mapping by hand. However, there are some environments, like Mono running on iOS, that do not support runtime JIT'ing, so FlatFiles would not work.

Where the runtime cannot generate code at all, as under Native AOT or with the System.Runtime.IsDynamicCodeSupported feature switch off, the mappers detect it through RuntimeFeature.IsDynamicCodeSupported and use reflection without being asked, so a published AOT application maps entities without throwing and without a code change.

Elsewhere, mappers support a method OptimiseMapping that can be used to switch to (A.K.A., slow) reflection. For example:

var mapper = DelimitedTypeMapper.Define<Person>(() => new Person());
mapper.Property(x => x.Id);
mapper.Property(x => x.Name);
mapper.OptimiseMapping(false);  // Use normal reflection to get and set properties

Mapping Under Native AOT

Where the runtime cannot generate code, a type mapper still works - it falls back to reflection, as described above - but it gives up the path that reads a record onto an entity without boxing any value. That path closes ColumnSetter<TEntity, T> over each column's type, which needs MakeGenericType and a delegate built from a MethodInfo, and neither is available for a value type the runtime was not built with.

This is handled for you. The package carries a source generator that looks for the calls naming an entity - Define<T>() and DefineDynamic( typeof( T ) ) - and writes those accessors for each type it finds, registering them from a module initialiser. You do nothing, and a mapped read costs what it costs where the runtime can generate code.

It tells you, at information level, about any member it could not write for and why: no setter, a setter that is not public, an init-only setter, or a type no column reads. Those members are read the slower way, which is what would have happened anyway.

The same applies to writing: a record is formatted straight from the entity, without its values being boxed into an array first, and the generator writes the accessors for that direction too.

A property need not allow both. One that is get-only, or whose setter is private or init, is written from but not read onto, and the generator emits what it can for each direction separately.

Where the generator cannot see a mapping - a type named only at run time, or one from an assembly that does not reference this package - you can register the accessors yourself:

using FlatFiles.CodeGeneration;

MappingAccessors.AddFactory( () => new Customer() );
MappingAccessors.AddSetter<Customer, int>( "CustomerId", ( e, v ) => e.CustomerId = v );
MappingAccessors.AddSetter<Customer, string>( "Name", ( e, v ) => e.Name = v );
MappingAccessors.AddNullableSetter<Customer, DateTime>( "Closed", ( e, v ) => e.Closed = v );

// And for writing:
MappingAccessors.AddGetter<Customer, int>( "CustomerId", e => e.CustomerId );
MappingAccessors.AddGetter<Customer, string>( "Name", e => e.Name );
MappingAccessors.AddNullableGetter<Customer, DateTime>( "Closed", e => e.Closed );

Each call closes its own generic types where you write it, which is why this works at all: nothing has to be closed later. Register once, before you read - a [ModuleInitializer] is the natural home, and is where the generator puts its own - and nothing else about your mapping changes.

Measured on 10,000 records of 13 columns with the dynamic-code switch off, a delimited read through a type mapper allocates 622 bytes a record unregistered and 359 registered. With the switch on the same read costs 362, so registering reaches what the runtime would have generated rather than approaching it. Where the runtime can generate code, registering makes no difference either way - the mapper was already on that path.

Three things worth knowing:

• Registering nothing changes nothing, so you can adopt this one type at a time, and a member you do not register is read exactly as it was.
• A registration written for one column type is ignored where the mapping uses another, rather than failing part way through a read.
• A member is registered against the type that declares it. A property hiding one of the same name on a base class is a registration of its own, named by the derived type.

Turning generation off

The generator writes registrations for every entity it finds, which is what makes it worth having and is also not everybody's preference. Set one property to have it write nothing:

<PropertyGroup>
  <FlatFilesGenerateAccessors>false</FlatFilesGenerateAccessors>
</PropertyGroup>

Nothing breaks when it is off. A mapping builds its own accessors at run time, exactly as it did before the generator existed - which costs a boxed value per member on a runtime that cannot generate code, and nothing at all on one that can. You can also pass it on the command line, -p:FlatFilesGenerateAccessors=false, and you can still register accessors by hand as above.

ExcludeAssets="analyzers" on the package reference does not turn it off, which is why this property exists.

Non-Public Classes and Members

Generated code lives in an assembly of its own, so it cannot see a class the rest of the world cannot see. As of FlatFiles 8.2.0 that is handled for you: mapping onto a type out of its reach uses reflection instead, which is slower per value and works, so nothing is needed to map onto an internal class. Before 8.2.0 it emitted a factory that could not reach the type, and the first record threw MethodAccessException.

Non-public members are a separate matter and still need one of the steps below, as does an internal type you would rather keep on the faster path.

The simplest solution is to make your classes and members public. Alternatively, you can disable optimisations which will cause FlatFiles to use normal reflection, which should be able to access anything, at the cost of some runtime overhead.

Another option is to grant FlatFiles access to your internal classes and members by adding the following line to you Assembly.cs file:

[assembly: InternalsVisibleTo("FlatFiles.DynamicAssembly,PublicKey=00240000048000009400000006020000002400005253413100040000010001009b9e44f637b293021ec4d8625071e5fe1682eeb167c233b46314cca79bf2769606285d5d1225cba8ce1e75be9e8ab7251d17eaf2c3b00fde5eac50a0f7dc7fec2f70279ff71c72341ad2738661babfdc6792479f14fd64d841285644d5c09c2902e9467f574e0d369161caee632087c5d819c3c36f76622306b09a4f868230c1")]

Notice that this only grants access to internal types/members. You will still not be able to access private members. An assembly that grants it keeps internal types on the generated path rather than the reflection fallback.

As a final option, you can use the CustomMapping method, passing delegates to read/write your members. Since the delegates are part of your project, they can access non-public members without trouble. There is almost no runtime overhead using CustomMapping instead of Property.

ADO.NET DataTables

If you are using DataTables, you can read and write to a DataTable using the ReadFlatFile and WriteFlatFile extension methods. Just pass the corresponding reader or writer object.

var customerTable = new DataTable("Customer");
using (var streamReader = new StreamReader(File.OpenRead(@"C:\path\to\file.csv")))
{
    var reader = new DelimitedReader(streamReader, schema);
    customerTable.ReadFlatFile(reader);
}

FlatFileDataReader

For low-level ADO.NET file reading, you can use the FlatFileDataReader class. It provides an IDataReader interface to the records in the file, making it compatible with other ADO.NET interfaces.

// The DataReader Approach
using (var fileReader = new StreamReader(File.OpenRead(@"C:\path\to\file.csv"))
{
    var csvReader = new DelimitedReader(fileReader, schema);
    var dataReader = new FlatFileDataReader(csvReader);
    var customers = new List<Customer>();
    while (dataReader.Read())
    {
        var customer = new Customer();
        customer.CustomerId = dataReader.GetInt32(0);
        customer.Name = dataReader.GetString(1);
        customer.Created = dataReader.GetDateTime(2);
        customer.AverageSales = dataReader.GetDouble(3);
        customers.Add(customer);
    }
    return customers;
}

Usually in cases like this, it is just easier to use the type mappers. However, this could be useful if you are swapping out an actual database call with CSV data inside of a unit test.

FlatFiles also provides helpful extension methods on the IDataReader interface to make it easier to extract data. It provides GetNullable* variants of the IDataReader methods, so you don't need to constantly call IsDBNull. There are also variants of each method accepting the column name rather than the ordinal position.

There are also generic GetValue<T> methods that can deal with type conversions automatically for you. For example, anytime you read in a CSV file without providing the schema, FlatFiles assumes each column is a string. When calling GetValue<int>("Id") or GetValue<DateTime?>("CreatedOn"), FlatFiles will try to convert the values for you. This is extremely helpful when you don't want to provide (or can't provide) a schema but can still determine the column types. For example, when you know the column names and their types but their order could be different between runs.

How This Library Is Verified

Besides the unit tests, this repository reads six files end to end before every release and compares what that cost against figures committed alongside them. The files are generated from profiles describing the shape of real ones — 379 columns; 344,352 records; every field quoted; fourteen record layouts chosen by a single character; a 3,500-character record — and they are committed rather than built, so the bytes being read are fixed and a change in the numbers means a change in the library.

Each file is read four ways: every column as text, every single-typed column given its type, that again with the values asked for, and the file read onto entities through a type mapper. The last is there because the first three never build an entity, so nothing taken from them can move when the mapping path does.

Three figures are compared. How many records each file yields is exact, and so is the requirement that none be refused. What a read allocates per record is compared to within 2%, being deterministic for given bytes on a given runtime. How long a read takes is reported and never compared, because it is not stable enough to decide anything.

The check runs in the publish workflow only, so a release fails if a figure moved without somebody meaning it to. FlatFiles.IntegrationTest/README.md has the detail: the six files and what each stresses, the tables of measurements, what every column means, and how to take a new baseline when a change moves one on purpose.

License

This is free and unencumbered software released into the public domain.

Anyone is free to copy, modify, publish, use, compile, sell, or distribute this software, either in source code form or as a compiled binary, for any purpose, commercial or non-commercial, and by any means.

In jurisdictions that recognize copyright laws, the author or authors of this software dedicate any and all copyright interest in the software to the public domain. We make this dedication for the benefit of the public at large and to the detriment of our heirs and successors. We intend this dedication to be an overt act of relinquishment in perpetuity of all present and future rights to this software under copyright law.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

For more information, please refer to http://unlicense.org