ChoETL is an open source ETL (extract, transform and load) framework for .NET. It is a code based library for extracting data from multiple sources, transforming, and loading into your very own data warehouse in .NET environment. You can have data in your data warehouse in no time.
Contents
- Introduction
- Requirement
- "Hello World!" Sample
- Quick load - Data First Approach
- Code First Approach
- Configuration First Approach
- Code First with declarative configuration
- Reading All Records
- Read Records Manually
- Customize XML Record
- Customize XML Fields
- DefaultValue
- ChoFallbackValue
- Type Converters
- Declarative Approach
- Configuration Approach
- Validations
- Callback Mechanism
- Using ChoXmlReader events
- Implementing IChoNotifyRecordRead interface
- BeginLoad
- EndLoad
- BeforeRecordLoad
- AfterRecordLoad
- RecordLoadError
- BeforeRecordFieldLoad
- AfterRecordFieldLoad
- RecordLoadFieldError
- Customization
- AsDataReader Helper Method
- AsDataTable Helper Method
- Using Dynamic Object
- DefaultValue
- ChoFallbackValue
- FieldType
- Type Converters
- Validations
- Working with sealed POCO object
- Exceptions
- Using MetadataType Annotation
- Configuration Choices
- Manual Configuration
- Auto Map Configuration
- Attaching MetadataType class
- LoadText Helper Method
- Advanced Topics
- Override Converters Format Specs
- Currency Support
- Enum Support
- Boolean Support
- DateTime Support
- CDATA Support
- Fluent API
- WithXPath
- WithXmlNamespaceManager
- WithXmlNamespace
- WithFields
- WithField
- ColumnCountStrict
- History
ChoETL is an open source ETL (extract, transform and load) framework for .NET. It is a code based library for extracting data from multiple sources, transforming, and loading into your very own data warehouse in .NET environment. You can have data in your data warehouse in no time.
This article talks about using ChoXmlReader
component offered by ChoETL framework. It is a simple utility class to extract XML data from file / source to objects.
UPDATE: Corresponding XmlWriter
article can be found here.
Features
- Uses XML Reader, parses XML file in fastest manner
- Stream based parsers allow for ultimate performance, low resource usage, and nearly unlimited versatility scalable to any size data file, even tens or hundreds of gigabytes
- Event based data manipulation and validation allows total control over the flow of data during the bulk insert process
- Exposes
IEnumerable
list of objects - which is often used with LINQ query for projection, aggregation and filtration, etc. - Supports deferred reading
- Supports processing files with culture specific date, currency and number formats
- Supports different character encoding
- Recognizes a wide variety of date, currency, enum, boolean and number formats when reading files
- Provides fine control of date, currency, enum, boolean, number formats when writing files
- Detailed and robust error handling, allowing you to quickly find and fix problems
This framework library is written in C# using .NET 4.x Framework / .NET Core 2.x.
- Open VS.NET 2017 or higher
- Create a sample VS.NET (.NET Framework 4.x / .NET Core 2.x) Console Application project
- Install ChoETL via Package Manager Console using Nuget Command:
Install-Package ChoETL
Install-Package ChoETL.NETStandard
- Use the
ChoETL
namespace
Let's begin by looking into a simple example of reading XML file having two columns.
Listing 3.1 Sample XML Data File
<Employees>
<Employee Id='1'>
<Name>Tom</Name>
</Employee>
<Employee Id='2'>
<Name>Mark</Name>
</Employee>
</Employees>
There are a number of ways you can get the XML file parsing started with minimal setup.
It is the zero config, quick way to load a XML file in no time. No POCO object is required. Sample code below shows how to load the file.
Listing 3.1.1 Load XML File using Iterator
foreach (dynamic rec in new ChoXmlReader("emp.xml"))
{
Console.WriteLine(rec.Id);
Console.WriteLine(rec.Name);
}
Sample fiddle: https://dotnetfiddle.net/X4nJO9
Listing 3.1.2 Load XML File using Loop
var reader = new ChoXmlReader("emp.xml");
dynamic rec;
while ((rec = reader.Read()) != null)
{
Console.WriteLine(rec.Id);
Console.WriteLine(rec.Name);
}
Sample fiddle: https://dotnetfiddle.net/bm47gw
This is another zero config way to parse and load XML file using POCO class. First, define a simple data class to match the underlying XML file layout.
Listing 3.2.1 Simple POCO Entity Class
public partial class EmployeeRec
{
public int Id { get; set; }
public string Name { get; set; }
}
In the above, the class defines two properties matching the sample XML file template.
Listing 3.2.2 Load XML File
foreach (var e in new ChoXmlReader<EmployeeRec>("emp.xml"))
{
Console.WriteLine(e.Id);
Console.WriteLine(e.Name);
}
Sample fiddle: https://dotnetfiddle.net/C2B7KO
In this model, we define the XML configuration with all the necessary parsing parameters along with XML columns matching with the underlying XML file.
Listing 3.3.1 Define XML Configuration
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
config.XmlRecordFieldConfigurations.Add(new ChoXmlRecordFieldConfiguration("Id"));
config.XmlRecordFieldConfigurations.Add(new ChoXmlRecordFieldConfiguration("Name"));
In the above, the class defines two properties matching the sample XML file template.
Listing 3.3.2 Load XML File without POCO Object
foreach (dynamic e in new ChoXmlReader("emp.xml", config))
{
Console.WriteLine(e.Id);
Console.WriteLine(e.Name);
}
Sample fiddle: https://dotnetfiddle.net/hHbVLP
Listing 3.3.3 Load XML file with POCO Object
foreach (var e in new ChoXmlReader<EmployeeRec>("emp.xml", config))
{
Console.WriteLine(e.Id);
Console.WriteLine(e.Name);
}
Sample fiddle: https://dotnetfiddle.net/U3QO5e
This is the combined approach to define POCO entity class along with XML configuration parameters decorated declaratively. Id
is required column and Name
is optional value column with default value "XXXX
". If Name
is not present, it will take the default value. This is OptIn
method, telling only decorated members to participate in the serialization operation. Any members not decorated using ChoXmlNodeRecordFieldAttribute
will be ignored.
Listing 3.4.1 Define POCO Object
public class EmployeeRec
{
[ChoXmlNodeRecordField(XPath = "//@Id")]
[Required]
public int Id
{
get;
set;
}
[ChoXmlNodeRecordField(XPath = "//Name")]
[DefaultValue("XXXX")]
public string Name
{
get;
set;
}
public override string ToString()
{
return "{0}. {1}".FormatString(Id, Name);
}
}
The code above illustrates about defining POCO object to carry the values of each record line in the input file. First thing defines property for each record field with ChoXmlNodeRecordFieldAttribute
to qualify for XML record mapping. XPath
is a optional property. If not specified, framework automatically discovers and loads the values either from xmlattribute
or xmlelement
. Id
is decorated with RequiredAttribute
, if the value is missing, it will throw an exception. Name
is given default value using DefaultValueAttribute
. It means that if the Name
XML column contains empty value in the file, it will be defaulted to 'XXXX
' value.
It is very simple and ready to extract XML data in no time.
Listing 3.4.2 Main Method
foreach (var e in new ChoXmlReader<EmployeeRec>("emp.xml"))
{
Console.WriteLine(e.Id);
Console.WriteLine(e.Name);
}
We start by creating a new instance of ChoXmlReader
object. That's all! All the heavy lifting of parsing and loading XML data stream into the objects is done by the parser under the hood.
By default, ChoXmlReader
discovers and uses default configuration parameters while loading XML file. These can be overridable according to your needs. The following sections will give details about each configuration attributes.
Sample fiddle: https://dotnetfiddle.net/lnO6ft
3.4.1. Using BCL Serialization Attributes
This is one other approach to decorate members of the POCO object using BCL serialization attributes, aka. XmlElementAttribute, XmlAttributeAttribute
, etc.
Listing 3.4.1.1 Define POCO Object
public class EmployeeRec
{
[XmlAttribute]
[Required]
[ChoXPath("@Id")]
public int Id
{
get;
set;
}
[XmlElement]
[DefaultValue("XXXX")]
public string Name
{
get;
set;
}
public override string ToString()
{
return "{0}. {1}".FormatString(Id, Name);
}
}
The code above illustrates about defining POCO object to carry the values of each record line in the input file. First thing defines property for each record field either XAttributeAttribute / XElementAttribute
to specify the type of node to map to. It is optional to use these attribute. If not specified, the framework automatically discovers and load the members appropriately. ChoXPath
is an optional property to specify xpath of the node to be loaded. Id
is decorated with RequiredAttribute
, if the value is missing, it will throw an exception. Name
is given default value using DefaultValueAttribute
.
It means that if the Name
XML column contains empty value in the file, it will be defaulted to 'XXXX
' value.
It is very simple and ready to extract XML data in no time.
Listing 3.4.1.2 Main Method
foreach (var e in new ChoXmlReader<EmployeeRec>("emp.xml"))
{
Console.WriteLine(e.Id);
Console.WriteLine(e.Name);
}
We start by creating a new instance of ChoXmlReader
object. That's all. All the heavy lifting of parsing and loading XML data stream into the objects is done by the parser under the hood.
By default, ChoXmlReader
discovers and uses default configuration parameters while loading XML file. These can be overridable according to your needs. The following sections will give details about each configuration attributes.
It is as easy as setting up POCO object match up with XML file structure, you can read the whole file as enumerable pattern. It is a deferred execution mode, but take care while making any aggregate operation on them. This will load the entire file records into memory.
Listing 4.1 Read XML File
foreach (var e in new ChoXmlReader<EmployeeRec>("emp.xml"))
{
Console.WriteLine(e.Id);
Console.WriteLine(e.Name);
}
or:
Listing 4.2 Read XML File Stream
foreach (var e in new ChoXmlReader<EmployeeRec>(textReader))
{
Console.WriteLine(e.Id);
Console.WriteLine(e.Name);
}
This model keeps your code elegant, clean, easy to read and maintain. Also leverages LINQ extension methods to perform grouping, joining, projection, aggregation, etc.
Listing 4.3 Using LINQ
var list = (from o in new ChoXmlReader<EmployeeRec>("emp.xml")
where o.Name != null && o.Name.StartsWith("R")
select o).ToArray();
foreach (var e in list)
{
Console.WriteLine(e.Id);
Console.WriteLine(e.Name);
}
It is as easy as setting up POCO object match up with XML file structure, you can read the whole file as enumerable pattern.
Listing 5.1 Read XML File
var reader = new ChoXmlReader<EmployeeRec>("emp.xml");
EmployeeRec rec = null;
while ((rec = reader.Read()) != null)
{
Console.WriteLine(rec.Id);
Console.WriteLine(rec.Name);
}
Sample fiddle: https://dotnetfiddle.net/BYK1n4
Using ChoXmlRecordObjectAttribute
, you can customize the POCO entity object declaratively.
Listing 6.1 Customizing POCO Object for Each Record
[ChoXmlRecordObject]
public class EmployeeRec
{
[ChoXmlNodeRecordField(XPath = "/@Id")]
public int Id { get; set; }
[ChoXmlNodeRecordField(XPath = "/Name")]
[Required]
[DefaultValue("ZZZ")]
public string Name { get; set; }
}
Here are the available attributes to carry out customization of XML load operation on a file:
XPath
- Optional. XPath
expression used to pick the elements to load. If not specified, the member value will be discovered and loaded automatically. ColumnCountStrict
- This flag indicates if an exception should be thrown if reading an expected field is missing. ErrorMode
- This flag indicates if an exception should be thrown if reading and an expected field is failed to load. This can be overridden per property. Possible values are:
IgnoreAndContinue
- Ignore the error, record will be skipped and continue with next ReportAndContinue
- Report the error to POCO entity if it is of IChoNotifyRecordRead
type ThrowAndStop
- Throw the error and stop the execution
IgnoreFieldValueMode
- A flag to let the reader know if a record should be skipped when reading if it's empty / null
. This can be overridden per property. Possible values are:
Null
- N/A DBNull
- N/A Empty
- skipped if the record value is empty WhiteSpace
- skipped if the record value contains only whitespaces
ObjectValidationMode
- A flag to let the reader know about the type of validation to be performed with record object. Possible values are:
Off
- No object validation performed. (Default) MemberLevel
- Validation performed at the time of each XML property gets loaded with value. ObjectLevel
- Validation performed after all the properties are loaded to the POCO object.
For each XML column, you can specify the mapping in POCO entity property using ChoXmlNodeRecordFieldAttribute
.
Only use this attribute if you want to use custom xpath to map to this column. Otherwise use the simple variation attributes aka. ChoXmlElementRecordFieldAttribute/ChoXmlAttributeRecordFieldAttribute.
Listing 7.1 Customizing POCO object for XML columns
public class EmployeeRec
{
[ChoXmlNodeRecordField(XPath="/@Id")]
public int Id { get; set; }
[ChoXmlNodeRecordField(XPath="/Name")]
[Required]
[DefaultValue("ZZZ")]
public string Name { get; set; }
}
Here are the available members to add some customization to it for each property:
XPath
- Optional. XPath expression uses a path notation, like those used in URLs, for addressing parts of an XML document. If not specified, ChoXmlReader
automatically discover and load the value either from XmlElement
or XmlAttribute
. If not specified, ChoXmlReader
discover and load the value from XElement
or XAttribute
matching the field name. ErrorMode
- This flag indicates if an exception should be thrown if reading and an expected field failed to load. Possible values are:
IgnoreAndContinue
- Ignore the error and continue to load other properties of the record. ReportAndContinue
- Report the error to POCO entity if it is of IChoRecord
type. ThrowAndStop
- Throw the error and stop the execution.
IgnoreFieldValueMode
- A flag to let the reader know if a record should be skipped when reading if it's empty / null
. Possible values are:
Null
- N/A DBNull
- N/A Empty
- skipped if the record value is empty. WhiteSpace
- skipped if the record value contains only whitespaces.
It is the value used and set to the property when the XML value is empty or whitespace (controlled via IgnoreFieldValueMode
).
Any POCO entity property can be specified with default value using System.ComponentModel.DefaultValueAttribute
.
It is the value used and set to the property when the XML value failed to set. Fallback
value only set when ErrorMode
is either IgnoreAndContinue
or ReportAndContinue
.
Any POCO entity property can be specified with fallback value using ChoETL.ChoFallbackValueAttribute
.
Most of the primitive types are automatically converted and set them to the properties. If the value of the XML field can't automatically be converted into the type of the property, you can specify a custom / built-in .NET converters to convert the value. These can be either IValueConverter
or TypeConverter
converters.
There are couple of ways you can specify the converters for each field:
- Declarative Approach
- Configuration Approach
This model is applicable to POCO entity object only. If you have POCO class, you can specify the converters to each property to carry out necessary conversion on them. Samples below shows the way to do it.
public class EmployeeRec
{
[ChoXmlNodeRecordField(XPath: "/@Id")]
[ChoTypeConverter(typeof(IntConverter))]
public int Id { get; set; }
[ChoXmlNodeRecordField(XPath: "/Name")]
[Required]
[DefaultValue("ZZZ")]
public string Name { get; set; }
}
Listing 7.3.1.2 IntConverter Implementation
public class IntConverter : IValueConverter
{
public object Convert(object value, Type targetType,
object parameter, CultureInfo culture)
{
return value;
}
public object ConvertBack(object value, Type targetType,
object parameter, CultureInfo culture)
{
return value;
}
}
In the example above, we defined custom IntConverter
class. And showed how to use it with 'Id
' Xml property.
This model is applicable to both dynamic and POCO entity object. This gives freedom to attach the converters to each property at runtime. This takes precedence over the declarative converters on POCO classes.
Listing 7.3.2.2 Specifying TypeConverters
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
ChoXmlNodeRecordFieldConfiguration idConfig =
new ChoXmlNodeRecordFieldConfiguration("Id", XPath: "/@Id");
idConfig.AddConverter(new IntConverter());
config.XmlRecordFieldConfigurations.Add(idConfig);
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name", XPath: "/Name"));
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name1", XPath: "/Name1"));
In the above, we construct and attach the IntConverter
to 'Id
' field using AddConverter
helper method in ChoXmlNodeRecordFieldConfiguration
object.
Likewise, if you want to remove any converter from it, you can use RemoveConverter
on ChoXmlNodeRecordFieldConfiguration
object.
ChoXmlReader
leverages both System.ComponentModel.DataAnnotations and Validation Block
validation attributes to specify validation rules for individual fields of POCO entity. Refer to the MSDN site for a list of available DataAnnotations
validation attributes.
Listing 7.4.1 Using Validation Attributes in POCO Entity
[ChoXmlRecordObject]
public partial class EmployeeRec
{
[ChoXmlNodeRecordField(FieldName = "id", XPath: "/@Id")]
[ChoTypeConverter(typeof(IntConverter))]
[Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
[ChoFallbackValue(1)]
public int Id { get; set; }
[ChoXmlNodeRecordField(FieldName = "Name", XPath: "/Name")]
[Required]
[DefaultValue("ZZZ")]
[ChoFallbackValue("XXX")]
public string Name { get; set; }
}
In the example above, used Range
validation attribute for Id
property. Required
validation attribute to Name
property. ChoXmlReader
performs validation on them during load based on Configuration.ObjectValidationMode
is set to ChoObjectValidationMode.MemberLevel or ChoObjectValidationMode.ObjectLevel
.
Sometimes, you may want override the defined declarative validation behaviors that come with POCO class, you can do with Cinchoo ETL via configuration approach. The sample below shows the way to override them.
static void ValidationOverridePOCOTest()
{
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
var idConfig = new ChoXmlNodeRecordFieldConfiguration("Id", XPath: "/@Id");
idConfig.Validators = new ValidationAttribute[] { new RequiredAttribute() };
config.XmlRecordFieldConfigurations.Add(idConfig);
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name", XPath: "/Name));
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Salary", XPath: "/Salary"));
using (var parser = new ChoXmlReader<EmployeeRecWithCurrency>("emp.xml", config))
{
object rec;
while ((rec = parser.Read()) != null)
{
Console.WriteLine(rec.ToStringEx());
}
}
}
public class EmployeeRecWithCurrency
{
public int? Id { get; set; }
public string Name { get; set; }
public ChoCurrency Salary { get; set; }
}
In some cases, you may want to take control and perform manual self validation within the POCO entity class. This can be achieved by inheriting POCO object from IChoValidatable
interface.
Listing 7.4.2 Manual Validation on POCO Entity
[ChoXmlRecordObject]
public partial class EmployeeRec : IChoValidatable
{
[ChoXmlNodeRecordField(FieldName = "id", XPath: "/@Id")]
[ChoTypeConverter(typeof(IntConverter))]
[Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
[ChoFallbackValue(1)]
public int Id { get; set; }
[ChoXmlNodeRecordField(FieldName = "Name", XPath: "/Name")]
[Required]
[DefaultValue("ZZZ")]
[ChoFallbackValue("XXX")]
public string Name { get; set; }
public bool TryValidate
(object target, ICollection<ValidationResult> validationResults)
{
return true;
}
public bool TryValidateFor
(object target, string memberName, ICollection<ValidationResult> validationResults)
{
return true;
}
public void Validate(object target)
{
}
public void ValidateFor(object target, string memberName)
{
}
}
The sample above shows how to implement custom self-validation in POCO object.
IChoValidatable
interface exposes the below methods:
TryValidate
- Validate entire object, return true
if all validation passed. Otherwise return false
. Validate
- Validate entire object, throw exception if validation is not passed. TryValidateFor
- Validate specific property of the object, return true
if all validation passed. Otherwise return false
. ValidateFor
- Validate specific property of the object, throw exception if validation is not passed.
ChoXmlReader
offers industry standard XML parsing out of the box to handle most of the parsing needs. If the parsing is not handling any of the needs, you can use the callback mechanism offered by ChoXmlReader
to handle such situations. In order to participate in the callback mechanism, you can use either of the following models:
- Using event handlers exposed by
ChoXmlReader
via IChoReader
interface. - Inheriting POCO entity object from
IChoNotifyRecordRead / IChoNotifyFileRead / IChoNotifyRecordFieldRead
interfaces - Inheriting
DataAnnotation
's MetadataType
type object by IChoNotifyRecordRead
/ IChoNotifyFileRead / IChoNotifyRecordFieldRead
interfaces. - Inheriting
IChoNotifyRecordFieldConfigurable
/ IChoNotifyRecordFieldConfigurable
configuration interfaces
Note: Any exceptions raised out of these interface methods will be ignored.
IChoReader
exposes the below events:
BeginLoad
- Invoked at the begin of the XML file load EndLoad
- Invoked at the end of the XML file load BeforeRecordLoad
- Raised before the XML record load AfterRecordLoad
- Raised after XML record load RecordLoadError
- Raised when XML record load errors out BeforeRecordFieldLoad
- Raised before XML field value load AfterRecordFieldLoad
- Raised after XML field value load RecordFieldLoadError
- Raised when XML field value errors out SkipUntil
- Raised before the XML parsing kicks off to add custom logic to skip record lines. DoWhile
- Raised during XML parsing where you can add custom logic to stop the parsing.
IChoNotifyRecordRead
exposes the below methods:
BeforeRecordLoad
- Raised before the XML record load AfterRecordLoad
- Raised after XML record load RecordLoadError
- Raised when XML record load errors out
IChoNotifyFileRead
exposes the below methods:
BeginLoad
- Invoked at the begin of the XML file load EndLoad
- Invoked at the end of the XML file load SkipUntil
- Raised before the XML parsing kicks off to add custom logic to skip record lines DoWhile
- Raised during XML parsing where you can add custom logic to stop the parsing
IChoNotifyRecordFieldRead
exposes the below methods:
BeforeRecordFieldLoad
- Raised before XML field value load AfterRecordFieldLoad
- Raised after XML field value load RecordFieldLoadError
- Raised when XML field value errors out
IChoNotifyRecordConfigurable
exposes the below method:
RecordConfigure
- Raised for XML record configuration
IChoNotifyRecordFieldConfigurable
exposes the below method:
RecondFieldConfigure
- Raised for each XML record field configuration
This is more direct and simplest way to subscribe to the callback events and handle your odd situations in parsing XML files. The downside is that code can't be reusable as you do by implementing IChoNotifyRecordRead
with POCO record object.
The sample below shows how to use the BeforeRecordLoad
callback method to skip nodes having Id < 100
.
static void IgnoreNodeTest()
{
using (var parser = new ChoXmlReader("emp.xml"))
{
parser.BeforeRecordLoad += (o, e) =>
{
if (e.Source != null)
{
e.Skip = ((XElement)e.Source).Attribute["Id"].Value < 100;
}
};
foreach (var e in parser)
Console.WriteLine(e.Dump());
}
}
Likewise, you can use other callback methods as well with ChoXmlReader
.
The sample below shows how to implement IChoNotifyRecordRead
interface to direct POCO class.
Listing 8.2.1 Direct POCO callback mechanism implementation
[ChoXmlRecordObject]
public partial class EmployeeRec : IChoNotifyRecordRead
{
[ChoXmlNodeRecordField(FieldName = "Id", XPath: "/@Id")]
[ChoTypeConverter(typeof(IntConverter))]
[Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
[ChoFallbackValue(1)]
public int Id { get; set; }
[ChoXmlNodeRecordField(FieldName = "Name", XPath: "/Name")]
[Required]
[DefaultValue("ZZZ")]
[ChoFallbackValue("XXX")]
public string Name { get; set; }
public bool AfterRecordLoad(object target, int index, object source)
{
throw new NotImplementedException();
}
public bool BeforeRecordLoad(object target, int index, ref object source)
{
throw new NotImplementedException();
}
public bool RecordFieldLoadError
(object target, int index, string propName, object value, Exception ex)
{
throw new NotImplementedException();
}
}
The sample below shows how to attach Metadata
class to POCO class by using MetadataTypeAttribute
on it.
Listing 8.2.2 MetaDataType based callback mechanism implementation
[ChoXmlRecordObject]
public class EmployeeRecMeta : IChoNotifyRecordRead
{
[ChoXmlNodeRecordField(FieldName = "Id", XPath: "/@Id")]
[ChoTypeConverter(typeof(IntConverter))]
[Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
[ChoFallbackValue(1)]
public int Id { get; set; }
[ChoXmlNodeRecordField(FieldName = "Name", XPath: "/Name")]
[Required]
[DefaultValue("ZZZ")]
[ChoFallbackValue("XXX")]
public string Name { get; set; }
public bool AfterRecordLoad(object target, int index, object source)
{
throw new NotImplementedException();
}
public bool BeforeRecordLoad(object target, int index, ref object source)
{
throw new NotImplementedException();
}
public bool RecordFieldLoadError
(object target, int index, string propName, object value, Exception ex)
{
throw new NotImplementedException();
}
}
[MetadataType(typeof(EmployeeRecMeta))]
public partial class EmployeeRec
{
public int Id { get; set; }
public string Name { get; set; }
}
The sample below shows how to attach Metadata
class for sealed or third party POCO class by using ChoMetadataRefTypeAttribute
on it.
Listing 8.2.3 ChoMetaDataRefType based callback mechanism implementation
[ChoMetadataRefType(typeof(EmployeeRec))]
[ChoXmlRecordObject]
public class EmployeeRecMeta : IChoNotifyRecordRead
{
[ChoXmlNodeRecordField(FieldName = "Id", XPath: "/@Id")]
[ChoTypeConverter(typeof(IntConverter))]
[Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
[ChoFallbackValue(1)]
public int Id { get; set; }
[ChoXmlNodeRecordField(FieldName = "Name", XPath: "/Name")]
[Required]
[DefaultValue("ZZZ")]
[ChoFallbackValue("XXX")]
public string Name { get; set; }
public bool AfterRecordLoad(object target, int index, object source)
{
throw new NotImplementedException();
}
public bool BeforeRecordLoad(object target, int index, ref object source)
{
throw new NotImplementedException();
}
public bool RecordLoadError(object target, int index, object source, Exception ex)
{
throw new NotImplementedException();
}
}
public partial class EmployeeRec
{
public int Id { get; set; }
public string Name { get; set; }
}
This callback invoked once at the beginning of the XML file load. source
is the XML file stream object. In here, you have a chance to inspect the stream, return true
to continue the XML load. Return false
to stop the parsing.
Listing 10.3.1 BeginLoad Callback Sample
public bool BeginLoad(object source)
{
StreamReader sr = source as StreamReader;
return true;
}
This callback invoked once at the end of the XML file load. source
is the XML file stream object. In here, you have a chance to inspect the stream, do any post steps to be performed on the stream.
Listing 8.4.1 EndLoad Callback Sample
public void EndLoad(object source)
{
StreamReader sr = source as StreamReader;
}
This callback invoked before each record line in the XML file is loaded. target
is the instance of the POCO record object. index
is the line index in the file. source
is the XML record line. In here, you have a chance to inspect the line, and override it with new line if you want to.
TIP: If you want to skip the XNode
from loading, set the source to null
.
TIP: If you want to take control of parsing and loading the record properties by yourself, set the source to String.Empty
.
Return true
to continue the load process, otherwise return false
to stop the process.
Listing 8.5.1 BeforeRecordLoad Callback Sample
public bool BeforeRecordLoad(object target, int index, ref object source)
{
XNode node = source as XNode;
return true;
}
This callback invoked after each record line in the XML file is loaded. target
is the instance of the POCO record object. index
is the line index in the file. source
is the XML record line. In here, you have a chance to do any post step operation with the record line.
Return true
to continue the load process, otherwise return false
to stop the process.
Listing 8.6.1 AfterRecordLoad Callback Sample
public bool AfterRecordLoad(object target, int index, object source)
{
XNode node = source as XNode;
return true;
}
This callback is invoked if error is encountered while loading record line. target
is the instance of the POCO record object. index
is the line index in the file. source
is the XML record line. ex
is the exception object. In here, you have a chance to handle the exception. This method is invoked only when Configuration.ErrorMode
is ReportAndContinue
.
Return true
to continue the load process, otherwise return false
to stop the process.
Listing 8.7.1 RecordLoadError Callback Sample
public bool RecordLoadError(object target, int index, object source, Exception ex)
{
XNode node = source as XNode;
return true;
}
This callback invoked before each XML record column is loaded. target
is the instance of the POCO record object. index
is the line index in the file. propName
is the XML record property name. value
is the XML column value. In here, you have a chance to inspect the XML record property value and perform any custom validations, etc.
Return true
to continue the load process, otherwise return false
to stop the process.
Listing 8.8.1 BeforeRecordFieldLoad Callback Sample
public bool BeforeRecordFieldLoad
(object target, int index, string propName, ref object value)
{
return true;
}
This callback invoked after each XML record column is loaded. target
is the instance of the POCO record object. index
is the line index in the file. propName
is the XML record property name. value
is the XML column value. Any post field operation can be performed here, like computing other properties, validations, etc.
Return true
to continue the load process, otherwise return false
to stop the process.
Listing 8.9.1 AfterRecordFieldLoad Callback Sample
public bool AfterRecordFieldLoad
(object target, int index, string propName, object value)
{
return true;
}
This callback invoked when error encountered while loading XML record column value. target
is the instance of the POCO record object. index
is the line index in the file. propName
is the XML record property name. value
is the XML column value. ex
is the exception object. In here, you have a chance to handle the exception. This method is invoked only after the below two sequences of steps have been performed by the ChoXmlReader
.
ChoXmlReader
looks for FallbackValue
value of each XML property. If present, it tries to assign its value to it. - If the
FallbackValue
value is not present and the Configuration.ErrorMode
is specified as ReportAndContinue
., this callback will be executed.
Return true
to continue the load process, otherwise return false
to stop the process.
Listing 8.10.1 RecordFieldLoadError Callback Sample
public bool RecordFieldLoadError
(object target, int index, string propName, object value, Exception ex)
{
return true;
}
ChoXmlReader
automatically detects and loads the configured settings from POCO entity. At runtime, you can customize and tweak these parameters before XML parsing. ChoXmlReader
exposes Configuration
property, it is of ChoXmlRecordConfiguration
object. Using this property, you can customize them.
Listing 9.1 Customizing ChoXmlReader at run-time
class Program
{
static void Main(string[] args)
{
using (var parser = new ChoXmlReader<EmployeeRec>("emp.xml"))
{
object row = null;
parser.Configuration.ColumnCountStrict = true;
while ((row = parser.Read()) != null)
Console.WriteLine(row.ToString());
}
}
ChoXmlReader
exposes AsDataReader
helper method to retrieve the XML records in .NET datareader
object. DataReader
are fast-forward streams of data. This datareader
can be used in few places like bulk copying data to database using SqlBulkCopy
, loading disconnected DataTable
, etc.
Listing 10.1 Reading as DataReader Sample
static void AsDataReaderTest()
{
using (var parser = new ChoXmlReader<EmployeeRec>("emp.xml"))
{
IDataReader dr = parser.AsDataReader();
while (dr.Read())
{
Console.WriteLine("Id: {0}, Name: {1}", dr[0], dr[1]);
}
}
}
ChoXmlReader
exposes AsDataTable
helper method to retrieve the XML records in .NET DataTable
object. It then can be persisted to disk, displayed in grid/controls or stored in memory like any other object.
Listing 11.1 Reading as DataTable sample
static void AsDataTableTest()
{
using (var parser = new ChoXmlReader<EmployeeRec>("emp.xml"))
{
DataTable dt = parser.AsDataTable();
foreach (DataRow dr in dt.Rows)
{
Console.WriteLine("Id: {0}, Name: {1}", dr[0], dr[1]);
}
}
}
So far, the article explained about using ChoXmlReader
with POCO object. ChoXmlReader
also supports loading XML file without POCO object. It leverages .NET dynamic feature. The sample below shows how to read XML stream without POCO object.
If you have XML file, you can parse and load the file with minimal/zero configuration.
The sample below shows it:
Listing 12.1 Loading XML File
class Program
{
static void Main(string[] args)
{
dynamic row;
using (var parser = new ChoXmlReader("emp.xml"))
{
while ((row = parser.Read()) != null)
{
Console.WriteLine(row.Id);
}
}
}
}
The above example automatically discovers the XML elements/attributes and parses the file.
You can override the default behavior of discovering columns automatically by adding field configurations manually and pass it to ChoXmlReader
for parsing file.
The sample shows how to do it.
Listing 12.3 Loading XML file with configuration
class Program
{
static void Main(string[] args)
{
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Id", XPath: "/@Id"));
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name", XPath: "/Name"));
dynamic row;
using (var parser = new ChoXmlReader("Emp.xml", config))
{
while ((row = parser.Read()) != null)
{
Console.WriteLine(row.Name);
}
}
}
}
To completely turn off the auto column discovery, you will have to set ChoXmlRecordConfiguration.AutoDiscoverColumns
to false
.
It is the value used and set to the property when the XML value is empty or whitespace (controlled via IgnoreFieldValueMode
).
Any POCO entity property can be specified with default value using System.ComponentModel.DefaultValueAttribute
.
For dynamic object members or to override the declarative POCO object member's default value specification, you can do so through configuration as shown below:
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
config.XmlRecordFieldConfigurations.Add(new ChoXmlNodeRecordFieldConfiguration("Id"));
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name") { DefaultValue = "NoName" })
It is the value used and set to the property when the XML value failed to set. Fallback
value is only set when ErrorMode
is either IgnoreAndContinue
or ReportAndContinue
.
Any POCO entity property can be specified with fallback value using ChoETL.ChoFallbackValueAttribute
.
For dynamic object members or to override the declarative POCO object member's fallback values, you can do through configuration as shown below:
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
config.XmlRecordFieldConfigurations.Add(new ChoXmlNodeRecordFieldConfiguration("Id"));
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name") { FallbackValue = "Tom" });
In the type less dynamic object model, the reader reads individual field value and populates them to dynamic object members in 'string
' value. If you want to enforce the type and do extra type checking during load, you can do so by declaring the field type at the field configuration.
Listing 12.3.1 Defining FieldType
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Id") { FieldType = typeof(int) });
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name"));
The above sample shows to define field type as 'int
' to 'Id
' field. This instructs the ChoXmlReader
to parse and convert the value to integer before assigning to it. This extra type safety alleviates the incorrect values being loaded to object while parsing.
Most of the primitive types are automatically converted and set to the properties by ChoXmlReader
. If the value of the XML field can't automatically be converted into the type of the property, you can specify a custom / built-in .NET converters to convert the value. These can be either IValueConverter
or TypeConverter
converters.
In the dynamic object model, you can specify these converters via configuration. See the below example on the approach taken to specify type converters for XML columns.
Listing 12.4.1 Specifying TypeConverters
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
ChoXmlNodeRecordFieldConfiguration idConfig =
new ChoXmlNodeRecordFieldConfiguration("Id");
idConfig.AddConverter(new IntConverter());
config.XmlRecordFieldConfigurations.Add(idConfig);
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name"));
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name1"));
In the above, we construct and attach the IntConverter
to 'Id
' field using AddConverter
helper method in ChoXmlNodeRecordFieldConfiguration
object.
Likewise, if you want to remove any converter from it, you can use RemoveConverter
on ChoXmlNodeRecordFieldConfiguration
object.
ChoXmlReader
leverages both System.ComponentModel.DataAnnotations and Validation Block
validation attributes to specify validation rules for individual XML fields. Refer to the MSDN site for a list of available DataAnnotations
validation attributes.
Listing 12.5.1 Specifying Validations
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
config.ThrowAndStopOnMissingField = false;
ChoXmlNodeRecordFieldConfiguration idConfig =
new ChoXmlNodeRecordFieldConfiguration("Id");
idConfig.Validators = new ValidationAttribute[] { new RangeAttribute(0, 100) };
config.XmlRecordFieldConfigurations.Add(idConfig);
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name"));
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name1"));
In the example above, we used Range
validation attribute for Id
property. XmlReader
performs validation on them during load based on Configuration.ObjectValidationMode
is set to ChoObjectValidationMode.MemberLevel or ChoObjectValidationMode.ObjectLevel
.
P.S.: Self validation NOT supported in Dynamic object model.
If you already have existing sealed POCO object or the object is in 3rd party library, we can use them with XmlReader
.
Listing 13.1 Existing Sealed POCO Object
public sealed class ThirdPartyRec
{
public int Id
{
get;
set;
}
public string Name
{
get;
set;
}
}
Listing 13.2 Consuming XML file
class Program
{
static void Main(string[] args)
{
using (var parser = new ChoXmlReader<ThirdPartyRec>("Emp.xml"))
{
object row = null;
while ((row = parser.Read()) != null)
Console.WriteLine(row.ToString());
}
}
}
In this case, XmlReader
reverse discovers the XML columns from the XML file and loads the data into POCO object. If the XML file structure and POCO object matches, the load will success with populating all corresponding data to its properties. In case the property is missing for any XML column, XmlReader
silently ignores them and continues on with rest.
You can override this behavior by setting ChoXmlRecordConfiguration.ThrowAndStopOnMissingField
property to false
. In this case, the XmlReader
will throw ChoMissingRecordFieldException
exception if a property is missing for an XML column.
XmlReader
throws different types of exceptions in different situations.
ChoParserException
- XML file is bad and parser not able to recover. ChoRecordConfigurationException
- Any invalid configuration settings are specified, this exception will be raised. ChoMissingRecordFieldException
- A property is missing for an XML column, this exception will be raised.
Cinchoo ETL works better with data annotation's MetadataType
model. It is a way to attach MetaData
class to data model class. In this associated class, you provide additional metadata
information that is not in the data model. Its role is to add attribute to a class without having to modify this one. You can add this attribute that takes a single parameter to a class that will have all the attributes. This is useful when the POCO classes are auto generated (by Entity Framework, MVC, etc.) by an automatic tool. This is why the second class comes into play. You can add new stuff without touching the generated file. Also this promotes modularization by separating the concerns into multiple classes.
For more information about it, please search in MSDN.
Listing 15.1 MetadataType annotation usage sample
[MetadataType(typeof(EmployeeRecMeta))]
public class EmployeeRec
{
public int Id { get; set; }
public string Name { get; set; }
}
[ChoXmlRecordObject]
public class EmployeeRecMeta : IChoNotifyRecordRead, IChoValidatable
{
[ChoXmlNodeRecordField(FieldName = "id",
ErrorMode = ChoErrorMode.ReportAndContinue )]
[ChoTypeConverter(typeof(IntConverter))]
[Range(1, 1, ErrorMessage = "Id must be > 0.")]
[ChoFallbackValue(1)]
public int Id { get; set; }
[ChoXmlNodeRecordField(FieldName = "Name")]
[StringLength(1)]
[DefaultValue("ZZZ")]
[ChoFallbackValue("XXX")]
public string Name { get; set; }
public bool AfterRecordFieldLoad
(object target, int index, string propName, object value)
{
throw new NotImplementedException();
}
public bool AfterRecordLoad(object target, int index, object source)
{
throw new NotImplementedException();
}
public bool BeforeRecordFieldLoad
(object target, int index, string propName, ref object value)
{
throw new NotImplementedException();
}
public bool BeforeRecordLoad(object target, int index, ref object source)
{
throw new NotImplementedException();
}
public bool BeginLoad(object source)
{
throw new NotImplementedException();
}
public void EndLoad(object source)
{
throw new NotImplementedException();
}
public bool RecordFieldLoadError
(object target, int index, string propName, object value, Exception ex)
{
throw new NotImplementedException();
}
public bool RecordLoadError
(object target, int index, object source, Exception ex)
{
throw new NotImplementedException();
}
public bool TryValidate
(object target, ICollection<ValidationResult> validationResults)
{
return true;
}
public bool TryValidateFor
(object target, string memberName, ICollection<ValidationResult> validationResults)
{
return true;
}
public void Validate(object target)
{
}
public void ValidateFor(object target, string memberName)
{
}
}
In the above, EmployeeRec
is the data class. It contains only domain specific properties and operations. Mark it as a very simple class to look at it.
We separate the validation, callback mechanism, configuration, etc. into metadata type class, EmployeeRecMeta
.
If the POCO entity class is an auto-generated class or exposed via library or it is a sealed class, it limits you to attach XML schema definition to it declaratively. In such a case, you can choose one of the options below to specify XML layout configuration.
- Manual Configuration
- Auto Map Configuration
- Attaching MetadataType class
I'm going to show you how to configure the below POCO entity class on each approach.
Listing 16.1 Sealed POCO entity class
public sealed class EmployeeRec
{
public int Id { get; set; }
public string Name { get; set; }
}
Define a brand new configuration object from scratch and add all the necessary XML fields to the ChoXmlConfiguration.XmlRecordFieldConfigurations
collection property. This option gives you greater flexibility to control the configuration of XML parsing. But the downside is the possibility of making mistakes and it is hard to manage them if the XML file layout is large.
Listing 16.1.1 Manual Configuration
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
config.ThrowAndStopOnMissingField = true;
config.XmlRecordFieldConfigurations.Add(new ChoXmlNodeRecordFieldConfiguration("Id"));
config.XmlRecordFieldConfigurations.Add(new ChoXmlNodeRecordFieldConfiguration("Name"));
This is an alternative approach and very less error-prone method to auto map the XML columns for the POCO entity class.
First, define a schema class for EmployeeRec
POCO entity class as below.
Listing 16.2.1 Auto Map class
public class EmployeeRecMap
{
[ChoXmlNodeRecordField(FieldName = "Id")]
public int Id { get; set; }
[ChoXmlNodeRecordField(FieldName = "Name")]
public string Name { get; set; }
}
Then you can use it to auto map XML columns by using ChoXmlRecordConfiguration.MapRecordFields
method.
Listing 16.2.2 Using Auto Map configuration
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
config.MapRecordFields<EmployeeRecMap>();
foreach (var e in new ChoXmlReader<EmployeeRec>("Emp.xml", config))
Console.WriteLine(e.ToString());
This is one another approach to attach MetadataType
class for POCO entity object. The previous approach simply cares for auto mapping of XML columns only. Other configuration properties like property converters, parser parameters, default/fallback values, etc. are not considered.
This model accounts for everything by defining MetadataType
class and specifying the XML configuration parameters declaratively. This is useful when your POCO entity is sealed
and not partial
class. Also, it is one of the favorable and less error-prone approaches to configure XML parsing of POCO entity.
Listing 16.3.1 Define MetadataType class
[ChoXmlRecordObject]
public class EmployeeRecMeta : IChoNotifyRecordRead, IChoValidatable
{
[ChoXmlNodeRecordField
(FieldName = "Id", ErrorMode = ChoErrorMode.ReportAndContinue )]
[ChoTypeConverter(typeof(IntConverter))]
[Range(1, 1, ErrorMessage = "Id must be > 0.")]
public int Id { get; set; }
[ChoXmlNodeRecordField(FieldName = "Name")]
[StringLength(1)]
[DefaultValue("ZZZ")]
[ChoFallbackValue("XXX")]
public string Name { get; set; }
public bool AfterRecordLoad(object target, int index, object source)
{
throw new NotImplementedException();
}
public bool BeforeRecordLoad(object target, int index, ref object source)
{
throw new NotImplementedException();
}
public bool RecordLoadError(object target, int index, object source, Exception ex)
{
throw new NotImplementedException();
}
public bool TryValidate
(object target, ICollection<ValidationResult> validationResults)
{
return true;
}
public bool TryValidateFor
(object target, string memberName, ICollection<ValidationResult> validationResults)
{
return true;
}
public void Validate(object target)
{
}
public void ValidateFor(object target, string memberName)
{
}
}
Listing 16.3.2 Attaching MetadataType class
ChoMetadataObjectCache.Default.Attach<EmployeeRec>(new EmployeeRecMeta());
foreach (var e in new ChoXmlReader<EmployeeRec>("Emp.xml"))
Console.WriteLine(e.ToString()
This is a little nifty helper method to parse and load XML text string into objects.
Listing 17.1 Using LoadText method
string txt = @"
<Employees>
<Employee Id='1'>
<Name>Tom</Name>
</Employee>
<Employee Id='2'>
<Name>Mark</Name>
</Employee>
</Employees>
";
foreach (var e in ChoXmlReader.LoadText(txt))
Console.WriteLine(e.ToStringEx());
Cinchoo ETL automatically parses and converts each XML column values to the corresponding XML column's underlying data type seamlessly. Most of the basic .NET types are handled automatically without any setup needed.
This is achieved through two key settings in the ETL system.
ChoXmlRecordConfiguration.CultureInfo
- Represents information about a specific culture including the names of the culture, the writing system, and the calendar used, as well as access to culture-specific objects that provide information for common operations, such as formatting dates and sorting strings. Default is 'en-US
'. ChoTypeConverterFormatSpec
- It is a global format specifier class that holds all the intrinsic .NET types formatting specs.
In this section, I'm going to talk about changing the default format specs for each .NET intrinsic data types according to parsing needs.
ChoTypeConverterFormatSpec
is a singleton class, the instance is exposed via 'Instance
' static member. It is thread local, means that there will be a separate instance copy kept on each thread.
There are two sets of format specs members given to each intrinsic type, one for loading and another one for writing the value, except for Boolean
, Enum
, DataTime
types. These types have only one member for both loading and writing operations.
Specifying each intrinsic data type format specs through ChoTypeConverterFormatSpec
will impact system wide, i.e., by setting ChoTypeConverterFormatSpec.IntNumberStyle = NumberStyles.AllowParentheses
, will impact all integer members of XML objects to allow parentheses. If you want to override this behavior and take control of specific XML data member to handle its own unique parsing of XML value from global system wide setting, it can be done by specifying TypeConverter
at the XML field member level. Refer to section 13.4 for more information.
NumberStyles
(optional) used for loading values from XML stream
and Format string
are used for writing values to XML stream
.
In this article, I'll brief about using NumberStyles
for loading XML data from stream
. These values are optional. It determines the styles permitted for each type during parsing of XML file. System automatically figures out the way to parse and load the values from underlying Culture
. In odd situation, you may want to override and set the styles the way you want in order to successfully load the file. Refer to MSDN for more about NumberStyles and its values.
Listing 18.1.1 ChoTypeConverterFormatSpec Members
public class ChoTypeConverterFormatSpec
{
public static readonly ThreadLocal<ChoTypeConverterFormatSpec>
Instance = new ThreadLocal<ChoTypeConverterFormatSpec>(() =>
new ChoTypeConverterFormatSpec());
public string DateTimeFormat { get; set; }
public ChoBooleanFormatSpec BooleanFormat { get; set; }
public ChoEnumFormatSpec EnumFormat { get; set; }
public NumberStyles? CurrencyNumberStyle { get; set; }
public string CurrencyFormat { get; set; }
public NumberStyles? BigIntegerNumberStyle { get; set; }
public string BigIntegerFormat { get; set; }
public NumberStyles? ByteNumberStyle { get; set; }
public string ByteFormat { get; set; }
public NumberStyles? SByteNumberStyle { get; set; }
public string SByteFormat { get; set; }
public NumberStyles? DecimalNumberStyle { get; set; }
public string DecimalFormat { get; set; }
public NumberStyles? DoubleNumberStyle { get; set; }
public string DoubleFormat { get; set; }
public NumberStyles? FloatNumberStyle { get; set; }
public string FloatFormat { get; set; }
public string IntFormat { get; set; }
public NumberStyles? IntNumberStyle { get; set; }
public string UIntFormat { get; set; }
public NumberStyles? UIntNumberStyle { get; set; }
public NumberStyles? LongNumberStyle { get; set; }
public string LongFormat { get; set; }
public NumberStyles? ULongNumberStyle { get; set; }
public string ULongFormat { get; set; }
public NumberStyles? ShortNumberStyle { get; set; }
public string ShortFormat { get; set; }
public NumberStyles? UShortNumberStyle { get; set; }
public string UShortFormat { get; set; }
}
The sample below shows how to load XML data stream having 'se-SE
' (Swedish) culture specific data using XmlReader
. Also the input feed comes with 'EmployeeNo
' values containing parentheses. In order to make the load successful, we have to set the ChoTypeConverterFormatSpec.IntNumberStyle
to NumberStyles.AllowParenthesis
.
Listing 18.1.2 Using ChoTypeConverterFormatSpec in code
static void UsingFormatSpecs()
{
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
config.Culture = new System.Globalization.CultureInfo("se-SE");
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Id") { FieldType = typeof(int) });
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name"));
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Salary")
{ FieldType = typeof(ChoCurrency) });
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("JoinedDate")
{ FieldType = typeof(DateTime) });
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("EmployeeNo")
{ FieldType = typeof(int) });
ChoTypeConverterFormatSpec.Instance.IntNumberStyle = NumberStyles.AllowParentheses;
using (var parser = new ChoXmlReader("Emp.xml", config))
{
object row = null;
while ((row = parser.Read()) != null)
Console.WriteLine(row.ToStringEx());
}
}
Cinchoo ETL provides ChoCurrency
object to read and write currency values in XML files. ChoCurrency
is a wrapper class to hold the currency value in decimal type along with support of serializing them in text format during XML load.
Listing 18.2.1 Using Currency members in dynamic model
static void CurrencyDynamicTest()
{
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Id"));
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name"));
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Salary")
{ FieldType = typeof(ChoCurrency) });
using (var parser = new ChoXmlReader("Emp.xml", config))
{
object rec;
while ((rec = parser.Read()) != null)
{
Console.WriteLine(rec.ToStringEx());
}
}
}
The sample above shows how to load currency values using dynamic object model. By default, all the members of dynamic object are treated as string
type, unless specified explicitly via ChoXmlFieldConfiguration.FieldType
. By specifying the field type as ChoCurrency
to the 'Salary
' XML field, XmlReader
loads them as currency object.
P.S.: The format of the currency value is figured by XmlReader
through ChoRecordConfiguration.Culture
and ChoTypeConverterFormatSpec.CurrencyNumberStyle
.
The sample below shows how to use ChoCurrency XML field in POCO entity class.
Listing 18.2.2 Using Currency members in POCO model
public class EmployeeRecWithCurrency
{
public int Id { get; set; }
public string Name { get; set; }
public ChoCurrency Salary { get; set; }
}
static void CurrencyTest()
{
using (var parser = new ChoXmlReader<EmployeeRecWithCurrency>("Emp.xml"))
{
object rec;
while ((rec = parser.Read()) != null)
{
Console.WriteLine(rec.ToStringEx());
}
}
}
Cinchoo ETL implicitly handles parsing of enum
column values from XML files. If you want to fine control the parsing of these values, you can specify them globally via ChoTypeConverterFormatSpec.EnumFormat
. Default is ChoEnumFormatSpec.Value
.
FYI, changing this value will impact system wide.
There are three possible values that can be used:
ChoEnumFormatSpec.Value
- Enum
value is used for parsing. ChoEnumFormatSpec.Name
- Enum
key name is used for parsing. ChoEnumFormatSpec.Description
- If each enum
key is decorated with DescriptionAttribute
, its value will be used for parsing.
Listing 18.3.1 Specifying Enum format specs during parsing
public enum EmployeeType
{
[Description("Full Time Employee")]
Permanent = 0,
[Description("Temporary Employee")]
Temporary = 1,
[Description("Contract Employee")]
Contract = 2
}
static void EnumTest()
{
ChoTypeConverterFormatSpec.Instance.EnumFormat = ChoEnumFormatSpec.Description;
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Id") { FieldType = typeof(int) });
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name"));
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Salary")
{ FieldType = typeof(ChoCurrency) });
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("JoinedDate")
{ FieldType = typeof(DateTime) });
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("EmployeeType")
{ FieldType = typeof(EmployeeType) });
ChoTypeConverterFormatSpec.Instance.IntNumberStyle = NumberStyles.AllowParentheses;
using (var parser = new ChoXmlReader("Emp.xml", config))
{
object row = null;
while ((row = parser.Read()) != null)
Console.WriteLine(row.ToStringEx());
}
}
Cinchoo ETL implicitly handles parsing of boolean XML column values from XML files. If you want to fine control the parsing of these values, you can specify them globally via ChoTypeConverterFormatSpec.BooleanFormat
. Default value is ChoBooleanFormatSpec.ZeroOrOne
.
FYI, changing this value will impact system wide.
There are four possible values that can be used:
ChoBooleanFormatSpec.ZeroOrOne
- '0
' for false
. '1
' for true
. ChoBooleanFormatSpec.YOrN
- 'Y
' for true
, 'N
' for false
. ChoBooleanFormatSpec.TrueOrFalse
- 'True
' for true
, 'False
' for false
. ChoBooleanFormatSpec.YesOrNo
- 'Yes
' for true
, 'No
' for false
.
Listing 18.4.1 Specifying boolean format specs during parsing
static void BoolTest()
{
ChoTypeConverterFormatSpec.Instance.BooleanFormat = ChoBooleanFormatSpec.ZeroOrOne;
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Id") { FieldType = typeof(int) });
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name"));
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Salary")
{ FieldType = typeof(ChoCurrency) });
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("JoinedDate")
{ FieldType = typeof(DateTime) });
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Active") { FieldType = typeof(bool) });
ChoTypeConverterFormatSpec.Instance.IntNumberStyle = NumberStyles.AllowParentheses;
using (var parser = new ChoXmlReader("Emp.xml", config))
{
object row = null;
while ((row = parser.Read()) != null)
Console.WriteLine(row.ToStringEx());
}
}
Cinchoo ETL implicitly handles parsing of datetime
XML column values from XML files using system Culture
or custom set culture. If you want to fine control the parsing of these values, you can specify them globally via ChoTypeConverterFormatSpec.DateTimeFormat
. Default value is 'd
'.
FYI, changing this value will impact system wide.
You can use any valid standard or custom datetime .NET format specification to parse the datetime
XML values from the file.
Listing 18.5.1 Specifying datetime format specs during parsing
static void DateTimeTest()
{
ChoTypeConverterFormatSpec.Instance.DateTimeFormat = "MMM dd, yyyy";
ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Id") { FieldType = typeof(int) });
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Name"));
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Salary")
{ FieldType = typeof(ChoCurrency) });
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("JoinedDate")
{ FieldType = typeof(DateTime) });
config.XmlRecordFieldConfigurations.Add
(new ChoXmlNodeRecordFieldConfiguration("Active") { FieldType = typeof(bool) });
ChoTypeConverterFormatSpec.Instance.IntNumberStyle = NumberStyles.AllowParentheses;
using (var parser = new ChoXmlReader("Emp.xml", config))
{
object row = null;
while ((row = parser.Read()) != null)
Console.WriteLine(row.ToStringEx());
}
}
The sample above shows how to parse custom datetime
XML values from XML file.
Note: As the datetime
values contain XML separator, it is given with double quotes to pass the parsing.
Cinchoo ETL implicitly handles parsing of CDATA
XML values from XML files. The CDATA
values can be loaded as string
object of ChoCDATA
object as well.
Listing 18.6.1 Loading CDATA as string object
public class EmployeeRec
{
[ChoXmlNodeRecordField()]
[Required]
public int Id
{
get;
set;
}
[ChoXmlNodeRecordField()]
[DefaultValue("XXXX")]
public string Name
{
get;
set;
}
[ChoXmlNodeRecordField()]
[DefaultValue("XXXX")]
public string Message
{
get;
set;
}
public override string ToString()
{
return "{0}. {1}.".FormatString(Id, Name);
}
}
Listing 18.6.2 Loading CDATA as native CDATA object itself
ChoETL
offers ChoCDATA
class to hold the CDATA
XML value in native format. ChoXmlReader
automatically handles the parsing of this value and loads them accordingly.
public class EmployeeRec
{
[ChoXmlNodeRecordField()]
[Required]
public int Id
{
get;
set;
}
[ChoXmlNodeRecordField()]
[DefaultValue("XXXX")]
public string Name
{
get;
set;
}
[ChoXmlNodeRecordField()]
[DefaultValue("XXXX")]
public ChoCDATA Message
{
get;
set;
}
public override string ToString()
{
return "{0}. {1}.".FormatString(Id, Name);
}
}
XmlReader
exposes few frequent to use configuration parameters via Fluent API methods. This will make the programming of parsing of XML files quicker.
This API method sets the XPath
expression to select the nodes to load using XmlReader
.
foreach (var e in new ChoXmlReader<EmployeeRec>
("Emp.xml").WithXPath("Employees/Employee"))
Console.WriteLine(e.ToString());
This API method sets XML namespace manager to provide the scope management for these namespaces. It stores prefixes and namespaces as string
s. This is used with XML nodes with xpath
that reference namespace-qualified element and attribute names.
foreach (var e in new ChoXmlReader<EmployeeRec>("Emp.xml").WithXNamespaceManager(ns))
Console.WriteLine(e.ToString());
This API method sets add XML namespace to XML name table. This is used with XML nodes with xpath
that reference namespace-qualified element and attribute names.
foreach (var e in new ChoXmlReader<EmployeeRec>
("Emp.xml").WithXNamespace("dd", "http://www.cinchoo.com/dd"))
Console.WriteLine(e.ToString());
This API method specifies the list of XML nodes (either attributes or elements) to be considered for parsing and loading. Other fields in the XML nodes will be discarded.
foreach (var e in new ChoXmlReader<EmployeeRec>
("Emp.xml").WithFields("Id", "Name"))
Console.WriteLine(e.ToString());
This API method used to add XML node with xpath
, data type and other parameters. This method is helpful in dynamic object model, by specifying each and individual XML node with appropriate datatype
.
foreach (var e in new ChoXmlReader<EmployeeRec>
("Emp.xml").WithField("Id", "/Id", typeof(int)))
Console.WriteLine(e.ToString());
This API method is used to set the XmlWriter
to perform check on column countness before reading XML file.
foreach (var e in new ChoXmlReader<EmployeeRec>("Emp.xml").ColumnCountStrict())
Console.WriteLine(e.ToString());
- 19th February, 2017: Initial version
- 9th February, 2022: Article updated