Introduction
This article briefly describes the most important basics of the library Manufaktura.Controls which I recently released as Open Source project. The project is a continuation of two other projects which I created eight years ago and which are described in the following articles:
I have made significant improvement in my programming skills in these years but Manufaktura.Controls still uses some code from these two old projects. I completely changed the architecture to allow cross-platform implementations but you can still find some old spaghetti code, especially in the body of rendering strategies.
I’m releasing the code as Open Source project because it did not bring me much profit as a purely commercial project. I hope that it will be more useful as open source and a large community will soon emerge around it.
The source code attached to this article only contains most important libraries due to attachment size restrictions. You can find the whole code in GIT repository: https://bitbucket.org/Ajcek/manufakturalibraries
What is it For (For Those Who are Not Familiar with my Libraries)
Manufaktura.Controls is a set of .NET libraries, written in C#, for rendering music notation in desktop, web and mobile apps. The core libraries are cross platform so they can be used in almost any .NET environments, such as .NET Framework, .NET Core, UWP, Mono, Silverlight, etc. There are implementations for WPF, WinForms, UWP, ASP.NET MVC, ASP.NET Core, etc. There are also legacy libraries for Silverlight and Windows 8.
The main purpose of the libraries is rendering music scores but they also offer other features such as MusicXML parsing, MIDI playback and helper methods for mathematical operations that are useful in music.
Solution Overview
There are two main libraries in the solution: Manufaktura.Controls and Manufaktura.Music. These are cross-platform libraries so they can be used in all .NET environments: web, desktop and mobile. Previously, they functioned as portable class libraries, currently they target .NET Standard 1.1.
Manufaktura.Music defines low-level concepts of music such as intervals, rhythm, proportions, etc. It’s independent from music notation so models defined in Manufaktura.Music are not aware of such concepts as notes, rests, clefs, etc. The library also defines arithmetic operations between models such as transposition of intervals, comparing pitches, etc.
Manufaktura.Controls is the largest library in the solution. It defines western music notation model, renderers, parsers, etc.
Architecture of Manufaktura.Music
The main concepts of Manufaktura.Music are: intervals, pitches, steps, proportions, rhythmic durations and scales.
Sound pitch concept is represented as three structures:
Step – defines a scale step (A, B, C, D, E, F, or G) which can be altered (augmented or diminished). Steps are not aware of exact pitch.Pitch – inherits from Step. This is a Step in exact octave with exact midi pitch.TunedPitch – inherits from Pitch. This is a Pitch with exact frequency. For example: A4 at 440Hz or A4 at 415Hz.
Intervals are also divided into three stuctures:
DiatonicInterval – defined by number of steps. Example: second, third, octave, etc.Interval – inherits from DiatonicInterval. Defines number of steps and number of halftones. Example: minor second, major second, diminished third, etc.BoundInterval – inherits from interval. Represents interval that starts from a certain pitch – concept similar to hooked vector. Example: Major third from C to E.
Proportions are just fractions that can be converted to double or cents. Example: Proportion.Sesquialtera returns a fraction of 3/2.
There are also other classes as RhythmicDurations and Scales. All these classes are more or less used in low-abstraction models in Manufaktura.Controls.
Architecture of Manufaktura.Controls
Manufaktura.Controls is the largest library in the solution. It defines model, parsers (for parsing MusicXml), renderers and render strategies.
Model
Overview
Model contains classes that represent different western music notation concepts such as notes, rests, clefs, barlines, etc. In the large extent, the model is based on MusicXml specification but it also utilizes the concepts from Manufaktura.Music library, for example, Note consists of Pitch and RhythmicDuration. Some classes from Manufaktura.Controls.Model can be treated as a lower level of abstraction from Manufaktura.Music classes, for example Pitch can be promoted to Note, Note can be reduced to Pitch, etc.
Creating Score Model
The Score contains all musical symbols that are to be drawn. There are two ways of creating the score:
- Manually with API
- Automatically by parser
You can learn the basics of manually creating score model from this article:
Staff Rules
Creating model manually is different than creating score with parser. When you parse MusicXml, the parser automatically applies some data contained in MusicXml file such as horizontal note positions in measure, stem directions, etc.
If you add notes and other symbols manually by API, some properties like stem direction are determined by staff rules. Staff rules are classes inheriting from StaffRule. For example, NoteStemRule automatically determines the stem direction upon inserting notes to the Staff. StaffRules are automatically applied when inserted into collections that inherit from ItemManagingCollection<TItem>. Most frequently used implementation of this class is MusicalSymbolCollection that manages items on the Staff.
Renderers and Render Strategies
Manufaktura.Controls uses a single codebase for every platform. This is achieved by abstract classes called Renderers and RenderStrategies:
Renderers – They define how to draw primitive shapes such as lines, texts and bezier curves but they are completely agnostic of musical notation.RenderStrategies – Translate music into primitive shapes such as lines, text and bezier curves but don’t know how to draw them.
ScoreRendererBase class has five main abstract methods:
DrawLineDrawArcDrawTextDrawBezierDrawCharacterInBound
These five methods are implemented in derived classes. For example, WPFCanvasScoreRenderer draws lines by creating a Line shape and placing it on the canvas:
public override void DrawLine(Primitives.Point startPoint,
Primitives.Point endPoint, Primitives.Pen pen, MusicalSymbol owner)
{
if (!EnsureProperPage(owner)) return;
if (Settings.RenderingMode != ScoreRenderingModes.Panorama)
{
startPoint = startPoint.Translate(CurrentScore.DefaultPageSettings);
endPoint = endPoint.Translate(CurrentScore.DefaultPageSettings);
}
var line = new Line();
line.Stroke = new SolidColorBrush(ConvertColor(pen.Color));
line.X1 = startPoint.X;
line.X2 = endPoint.X;
line.Y1 = startPoint.Y;
line.Y2 = endPoint.Y;
line.StrokeThickness = pen.Thickness;
line.Visibility = BoolToVisibility(owner.IsVisible);
Canvas.Children.Add(line);
OwnershipDictionary.Add(line, owner);
}
HtmlSvgScoreRenderer creates SVG tags and adds them to XML document that represents SVG canvas:
public override void DrawLine(Point startPoint, Point endPoint, Pen pen, Model.MusicalSymbol owner)
{
if (!EnsureProperPage(owner)) return;
if (Settings.RenderingMode != ScoreRenderingModes.Panorama &&
!TypedSettings.IgnorePageMargins)
{
startPoint = startPoint.Translate(CurrentScore.DefaultPageSettings);
endPoint = endPoint.Translate(CurrentScore.DefaultPageSettings);
}
var element = new XElement("line",
new XAttribute("x1", startPoint.X.ToStringInvariant()),
new XAttribute("y1", startPoint.Y.ToStringInvariant()),
new XAttribute("x2", endPoint.X.ToStringInvariant()),
new XAttribute("y2", endPoint.Y.ToStringInvariant()),
new XAttribute("style", pen.ToCss()),
new XAttribute("id", BuildElementId(owner)));
var playbackAttributes = BuildPlaybackAttributes(owner);
foreach (var playbackAttr in playbackAttributes)
{
element.Add(new XAttribute(playbackAttr.Key, playbackAttr.Value));
}
if (startPoint.Y < ClippedAreaY) ClippedAreaY = startPoint.Y;
if (endPoint.Y < ClippedAreaY) ClippedAreaY = endPoint.Y;
if (startPoint.X > ActualWidth) ActualWidth = startPoint.X;
if (endPoint.X > ActualWidth) ActualWidth = endPoint.X;
if (startPoint.Y > ActualHeight) ActualHeight = startPoint.Y;
if (endPoint.Y > ActualHeight) ActualHeight = endPoint.Y;
Canvas.Add(element);
}
Notice that Canvas property can be any object (for example, a control, XML document, etc.). Canvas type is provided as type parameter for ScoreRenderer.
RenderStrategies are derived from MusicalSymbolRenderStrategy. This is a generic type – proper render strategy is matched by type parameter. For example, NoteRenderStrategy derives from MusicalSymbolRenderStrategy<Note>. The main method of each renderer is:
public override void Render(Barline element, ScoreRendererBase renderer)
The first parameter is an element that is to be drawn. The second parameter is the score renderer instance. Each platform uses a different score renderer implementation but the code of Render method is independent from any implementation – it just tells the Renderer to draw primitive shapes such as lines, texts, etc. and the Renderer does the rest.
When the score is rendered, different RenderStrategies are injected for each element of the score so each element is rendered using a proper RenderStrategy. Notice that constructors of different renderer strategies take different parameters, for example, KeyRenderStrategy only uses IScoreService but NoteRenderStrategy uses also IBeamingService, IMeasurementService, etc. These are services that are automatically injected to each rendering strategy by simple IoC mechanism. The role of these services is to store shared data between different render strategy instances and provide help for some more complex calculations that can be reused between different renderers. The most commonly used service is IScoreService that stores, among other things, current X position on the staff.
This is an example of render strategy for drawing time signatures:
public class TimeSignatureRenderStrategy : MusicalSymbolRenderStrategy<TimeSignature>
{
public TimeSignatureRenderStrategy(IScoreService scoreService) : base(scoreService)
{
}
public override void Render(TimeSignature element, ScoreRendererBase renderer)
{
var topLinePosition = scoreService.CurrentLinePositions[0];
if (element.Measure.Elements.FirstOrDefault() == element)
scoreService.CursorPositionX += renderer.LinespacesToPixels(1);
if (element.SignatureType != TimeSignatureType.Numbers)
{
renderer.DrawCharacter(element.GetCharacter
(renderer.Settings.CurrentFont), MusicFontStyles.MusicFont,
scoreService.CursorPositionX, topLinePosition +
renderer.LinespacesToPixels(2), element);
element.TextBlockLocation = new Primitives.Point
(scoreService.CursorPositionX, topLinePosition + renderer.LinespacesToPixels(2));
}
else
{
if (renderer.IsSMuFLFont)
{
renderer.DrawString(SMuFLGlyphs.Instance.BuildTimeSignatureNumberFromGlyphs
(element.NumberOfBeats),
MusicFontStyles.MusicFont, scoreService.CursorPositionX,
topLinePosition + renderer.LinespacesToPixels(1), element);
renderer.DrawString(SMuFLGlyphs.Instance.BuildTimeSignatureNumberFromGlyphs
(element.TypeOfBeats),
MusicFontStyles.MusicFont, scoreService.CursorPositionX,
topLinePosition + renderer.LinespacesToPixels(3), element);
element.TextBlockLocation = new Primitives.Point
(scoreService.CursorPositionX, topLinePosition + renderer.LinespacesToPixels(3));
}
else
{
renderer.DrawString(Convert.ToString(element.NumberOfBeats),
MusicFontStyles.TimeSignatureFont, scoreService.CursorPositionX,
topLinePosition + renderer.LinespacesToPixels(2), element);
renderer.DrawString(Convert.ToString(element.TypeOfBeats),
MusicFontStyles.TimeSignatureFont, scoreService.CursorPositionX,
topLinePosition + renderer.LinespacesToPixels(4), element);
element.TextBlockLocation = new Primitives.Point(scoreService.CursorPositionX,
topLinePosition + renderer.LinespacesToPixels(4));
}
}
scoreService.CursorPositionX += 20;
}
}
Platform-Specific Implementations
Overview
The most numerous libraries in the solution are platform-specific implementations of Manufaktura.Controls. These are also the smallest libraries in terms of the number of classes. Typically, they contain the following items:
ScoreRenderer implementations for specific platform- Controls (desktop and mobile) or Razor extensions (web) that utilize a specific
ScoreRenderer to draw scores and sometimes provide some UX logic as note dragging, etc. - Media players – for score playback. WPF and WinForms version use a shared implementation of MIDI playbach contained in
Manufaktura.Controls.Desktop.
Fonts and Font Metrics
Manufaktura.Controls uses two type of fonts:
If you plan to implement your own ScoreRenderer, there are some things you should know about font metrics. First of all, Polihymnia and SMuFL fonts are designed in such way that the position of the baseline coincides with the position of the line on the staff (or center of field). For example, if you place a notehead at Y coordinate of line 3, the center of notehead will exactly appear on line 3. Unfortunately, specific frameworks offer two completely different ways of text positioning:
- Text block coordinates are the coordinates of font baseline. HTML SVG works that way.
- Text block coordinates are the coordinates of lower left corner of text block bounding box (example: TextBlock element in WPF).
The second behavior is undesired so we should correct the text position by translating the text block position by baseline position. Baseline coordinate can be read from font metrics that have to be retrieved in a different way for every platform. For example, WPF does it in the following way:
var baseline = typeface.FontFamily.Baseline * textBlock.FontSize;
Canvas.SetLeft(textBlock, location.X);
Canvas.SetTop(textBlock, location.Y - baseline);
This is done completely differently in WinForms (System.Graphics):
var baselineDesignUnits = font.FontFamily.GetCellAscent(font.Style);
var baselinePixels = (baselineDesignUnits * font.Size) / font.FontFamily.GetEmHeight(font.Style);
Canvas.DrawString(text, font, new SolidBrush(ConvertColor(color)),
new PointF((float)location.X - 4, (float)location.Y - baselinePixels));
There is no easy way to do it in UWP apps so I decided to let the programmer provide the baseline position manually.
Test Apps and Unit Tests
In the solution, there are some test app and unit test projects. The most important is Manufatura.Controls.VisualTests. This is a unit test project that renders some predefined scores (provided in MusicXml format) as bitmaps. It compares the created bitmaps with previously created bitmaps and marks all the differences in red so you can easily track regression cases between commits.
This picture shows sample visual tests result. Some slur corrections have been made between two versions of code. The differences are marked in red:
Additional Resources
You can read other articles and tutorials about Manufaktura.Controls here:
This is a Bitbucket repository:
Two articles about the old projects which started everything:
Fields that Still Need to be Improved
Proper Xamarin Implementation
There is a Xamarin.Forms implementation with renderer for Android. It’s still in beta state and I don’t have time to develop this.
Rendering Pipeline
Rendering mechanism is not optimized for rendering many pages. It first renders all notes and musical symbols and then draws all lines. There also should be some kind of virtualization to only render the part of score that is visible on the screen.
Support for More Music Notation Concepts
Some music notation symbols are still not supported. For example, the library can’t render crescendo and decrescendo marks.
Client-side Rendering in HTML
Currently, all web implementations (Manufaktura.Controls.AspNetMvc, Manufaktura.Controls.AspNetCore) use server-side rendering.
I tried to implement this using CSHTML5 (http://cshtml5.com/) but I did not have time to deal with it and CSHTML5 is still in development. There are three possibilities to implement client-side rendering:
- Continue CSHTML5 implementation project.
- Maybe use some library that is based on
WebAssembly, like Blazor. - Create a
ScoreRenderer that creates Vexflow (http://www.vexflow.com/) or Verovio (http://www.verovio.org/index.xhtml) scripts.
