Contents
Introduction
Perceptor is an artificially intelligent guided navigation system for WPF. Perceptor tracks a user's behaviour while he or she
interacts with the user interface. Changes to the DataContext of a host control indicate
user navigation behaviour, and induce the training of a neural network.
Knowledge acquired by the neural network is used to predict the IInputElements
to which a user may intend to navigate. This accelerates interface interaction, improves user efficiency,
and allows for dynamic and evolving business rule creation.
Background
Last year (2008), I was asked to implement some business rules for an ASP.NET application.
Part of this application was designed to allow high volume data entry, and used a tabbed interface
with which staff would navigate and manually validate and amend information.
The rules I implemented were designed to streamline this process.
At the time it struck me that hardwiring the behaviour of a user interface,
based on business procedures, was too rigid. The way people work changes, and the way
an application is used varies from user to user. Moreover, refinement of such rules overtime leads
to increased maintenance costs, and to the retraining of staff to handle new improved application behaviour.
I envisioned a system where we could let the users define the behaviour by using it.
A system that could learn how to respond by itself.
To this end, this article and the accompanying code are provided as a proof of concept.
A Neural Network Driven Interface
Even though we have at our disposal terrific technologies such as WPF to build dynamic and highly reactive interfaces,
most interfaces, albeit rich, are in themselves not smart; they employ not even a modicum of AI when responding to user interaction.
Perhaps one may liken intelligent interfaces to the flying car; they are both much easier to do in sci-fi, are both
the next step in the evolution of the technology, and both take a lot of refinement to get right.
My wish is that I want the interface to know what I want, and to learn about me.
But I also want it to do this in a way that doesn't bother me by making poor assumptions,
and that is probably one of the biggest challenges.
If running out of petrol requires a crash landing, then I'd prefer to remain land bound.
We've all seen how artificial neural networks (ANN) can be used to do things such
as facial and optical character recognition. Indeed they work well at pattern recognition
where there exists well defined training data, and it appears that we are able to leverage
the same technology, albeit in a different manner, to recognize user behaviour as well.
There are, however, a number of challenges, such as dealing with temporal based progressive training,
because training data is not predefined; the network is trained as we go.
An advantage of using an ANN is that we are able to provide predictions
for situations that haven't been seen yet.
Perceptor uses a three layered neural network, which becomes associated
with a host ContainerControl and a DataContext type.
In this article we will not be looking at neural networks,
as there are already some very good articles here on CP. I recommend taking a look at
Sacha Barber's series of articles
on the topic if you are new to neural networks. I will mention though that during experimentation it was realised that
a future enhancement might be a Long Short Term Memory (LSTM) implementation.
In this prototype we retrain the neural network repeatedly with all inputs in order to learn progressively.
Building a flying car with WPF
Perceptor trains a neural network using the state of the DataContext
of a host control as input, and a list of IInputControls id's as output.
Prediction data, and the serialized neural network is saved locally when offline,
or remotely on the server when online.
Figure: Perceptor system overview.
Perceptor monitors the DataContext of a host control for changes.
By doing this, rather than tracking only the state of the controls, we are able to gather more information
about how the user is affecting the state of the system. We are able to make inferences based on not only user behaviour
but also system behaviour, as the system is capable of modifying the DataContext as a result of an internal or external event.
Put another way, if we were to merely track the controls, we would not be able to associate properties that didn't
have a visual representation in the interface.
By tracking the DataContext we can analyse the structure more deeply, and
we can even enhance how we generate the input for our neural network. We can, in effect, drill down into the DataContext
to improve the granularity and the quality of Perceptor's predictions.
Input for our neural network is generated by the NeuralInputGenerator.
This takes an object exposed by the DataContext property of a control,
and converts it into a double[], which can then be used to train or pulse
our neural network.
public double[] GenerateInput(object instance, bool newInstance)
{
ArgumentValidator.AssertNotNull(instance, "instance");
var clientType = instance.GetType();
if (lastKnownType == null || lastKnownType != clientType)
{
lock(lastKnownTypeLock)
{
if (lastKnownType == null || lastKnownType != clientType)
{
Initialize(clientType);
}
}
}
var resultSize = propertyCount + 1;
var doubles = new double[resultSize];
doubles[0] = newInstance ? trueLevel : falseLevel;
for (int i = 1; i < resultSize; i++)
{
var info = propertyInfos.Values[i - 1];
if (info.PropertyType == typeof(string))
{
var propertyValue = info.GetValue(instance, null);
doubles[i] = propertyValue != null ? trueLevel : falseLevel;
}
else if (info.PropertyType == typeof(bool))
{
var propertyValue = info.GetValue(instance, null);
doubles[i] = (bool)propertyValue ? trueLevel : falseLevel;
}
else if (!typeof(ValueType).IsAssignableFrom(info.PropertyType))
{
var propertyValue = info.GetValue(instance, null);
doubles[i] = propertyValue != null ? trueLevel : falseLevel;
}
}
return doubles;
}
Here we examine the provided instance's properties and, using some
rules based on whether a property is populated etc., populate the double[].
The input generated by this method provides us with a fingerprint of our DataContext,
and indeed a discreet representation of the interface model. There is an opportunity
to refine the NeuralInputGenerator, to increase its recognition of known field types,
and even add child object analysis.
Persistence
The ADO.NET entity framework is used to access a table of prediction
data associated with a user and a control id. When Perceptor is attached to a host control,
it will attempt to retrieve existing prediction data for the user and the particular host control id.
It does this by, firstly checking if the host control has assigned the Perceptor.PersistenceProvider
attached property. If so, Perceptor will attempt to use the provider for persistence.
This extensibility point for
persisting prediction data can be utilised by implementing the IPersistPredictionData
interface.
When the window of host control is closing, Perceptor will attempt to save its prediction data.
In the sample application we associate the prediction data with a user id. The following excerpt
from the sample demonstrates how this can be done.
public void SavePredictionData(LearningData predictionData)
{
log.Debug("Attempting to save prediction data." + predictionData);
if (Testing)
{
return;
}
var learningUIService = ChannelManagerSingleton.Instance.GetChannel<ILearningUIService>();
Debug.Assert(learningUIService != null);
learningUIService.SavePredictionData(testUserId, predictionData);
}
Sample overview
The download includes a sample application, which is meant to demonstrate how Perceptor can be used
to guide the user to input elements. It displays a list of employee names, and each when selected
populates the Employee Details tab and Boss panel of the application.
Figure: Opening screen shot of sample application.
Each time a field is modified, causing a modification to the DataContext,
the ANN is pulsed, and a candidate input prediction is taken. If the prediction's
confidence level is above a predefined threshold, the user is presented with the option
to navigate directly to the predicted input control.
An overview of Perceptor's learning process is illustrated below.
Learning Phase
Figure: Learning Phase
Once Perceptor has acquired enough knowledge to make confident predictions,
it can be used to navigate to predicted elements.
Predictive Phase
Figure: Predictive Phase
A feature of Perceptor is automatic expansion when the predicted
element happens to reside in an Expander. This expansion occurs
as soon as a confident prediction is detected.
In the sample application we can see how a confident prediction of an element
is highlighted.
Figure: Perceptor guides the user to the next predicted element.
Shifting Control Focus
Deterministic focus shifting in WPF can be tricky. When we call Focus() on a UIElement
there is no guarantee that the element will gain focus. That is why this method returns true if it succeeds.
In Perceptor we use the FocusForcer class to move focus within the user interface.
UIElement.Focus() returns false if either IsEnabled, IsVisible
or Focusable are false, and true if focus is shifted.
Yet when performed on the same thread that is handling e.g. PreviewLostKeyboardFocus
of the currently focused element ϑ, the call will return false as ϑ won't be ready to relinquish focus.
Thus we use our FocusForcer and an extension method to perform the change of focus in the background if required.
The following excerpt shows how FocusForcer attempts to focus the specified element.
static void FocusControl(UIElement element)
{
ArgumentValidator.AssertNotNull(element, "element");
Keyboard.Focus(element);
var focusResult = element.Focus();
if (focusResult)
{
return;
}
element.Dispatcher.Invoke(DispatcherPriority.Background, (Action)delegate
{
focusResult = element.Focus();
Keyboard.Focus(element);
if (!focusResult)
{
CommitFocusedElement();
focusResult = element.Focus();
Keyboard.Focus(element);
}
if (!focusResult)
{
log.Warn(string.Format("Unable to focus UIElement {0} "
+ "IsVisible: {1}, Focusable: {2}, Enabled: {3}",
element, element.IsVisible, element.Focusable,
element.IsEnabled));
}
});
}
When we initialize Perceptor we create an output neuron in the neural network
for each IInputElement of the container control.
void InitializeFromHost(FrameworkElement host)
{
ArgumentValidator.AssertNotNull(host, "host");
this.host = host;
host.DataContextChanged += OnHostDataContextChanged;
host.CommandBindings.Add(new CommandBinding(
NavigateForward, OnNavigateForward, OnCanNavigateForward));
host.CommandBindings.Add(new CommandBinding(
NavigateBackward, OnNavigateBackward, OnCanNavigateBackward));
host.CommandBindings.Add(new CommandBinding(
ResetLearning, OnResetLearning, OnCanResetLearning));
outputNeuronCount = 0;
inputElementIndexes.Clear();
var inputElements = host.GetChildrenOfType<IInputElement>();
foreach (var inputElement in inputElements)
{
inputElementIndexes.Add(inputElement, outputNeuronCount);
inputElement.PreviewLostKeyboardFocus -= OnInputElementPreviewLostKeyboardFocus;
inputElement.PreviewLostKeyboardFocus += OnInputElementPreviewLostKeyboardFocus;
outputNeuronCount++;
}
var window = host.GetWindow();
if (window != null)
{
window.Closing += window_Closing;
}
}
Consuming Perceptor
In order to have Perceptor monitor any container control, we use attached properties as shown in the following example.
<TabControl Name="tabControl_Main" Grid.Row="2" VerticalAlignment="Stretch" SelectedIndex="0"
LearningUI:Perceptor.Enabled="true"
LearningUI:Perceptor.PersistenceProvider="{Binding ElementName=rootElement}" />
The PersistenceProvider property is not necessary. But it exists so that we can customize how the user's
prediction data is saved between sessions. In the example download we use the window to transport the prediction data
to and from the ILearningUIService WCF service. As this it is a hybrid smart client, Perceptor allows
the user to work offline if the service is unavailable, and will fall back on persisting the prediction data
to the user's local file system if the PersistenceProvider is unavailable or raises an Exception.
The following excerpt shows the IPersistPredictionData interface.
public interface IPersistPredictionData
{
void SavePredictionData(PerceptorData predictionData);
PerceptorData LoadPredictionData(string id);
}
Perceptor exposes three routed commands, and they are:
NavigateForward
Is used to change focus to the next predicted UIElement.NavigateBackward
Is used to return to the UIElement that previously had focus.
When NavigateForward is performed, the current element
with focus is placed on a stack.ResetLearning
Is used to recreate the neural network, so that previous learning is forgotten.
Service Channel Management
In order to manage channels efficiently I have implemented a class called ChannelManagerSingleton.
In a previous article
I wrote a little about the Silverlight incarnation, so I won't restate things here. I will, however, mention that since then
I have produced a WPF version (included in the download) with support for duplex services.
Duplex services are cached using the callback instance and service type combination as a unique key.
In this way, we are still able to have centralised management of services, even though a callback instance is involved.
The following excerpt shows the GetDuplexChannel method in full,
and how duplex channels are created and cached.
public TChannel GetDuplexChannel<TChannel>(object callbackInstance)
{
if (callbackInstance == null)
{
throw new ArgumentNullException("callbackInstance");
}
Type serviceType = typeof(TChannel);
object service;
var key = new DuplexChannelKey { ServiceType = serviceType, CallBackInstance = callbackInstance };
duplexChannelsLock.EnterUpgradeableReadLock();
try
{
if (!duplexChannels.TryGetValue(key, out service))
{
duplexChannelsLock.EnterWriteLock();
try
{
var context = new InstanceContext(callbackInstance);
var channelFactory = new DuplexChannelFactory<TChannel>(context, "*");
service = channelFactory.CreateChannel();
var communicationObject = (ICommunicationObject)service;
communicationObject.Faulted += OnDuplexChannelFaulted;
duplexChannels.Add(key, service);
communicationObject.Open();
ConnectIfClientService(service, serviceType);
}
finally
{
duplexChannelsLock.ExitWriteLock();
}
}
}
finally
{
duplexChannelsLock.ExitUpgradeableReadLock();
}
return (TChannel)service;
}
Unit Testing WPF with White
Black-box testing can compliment your existing unit tests. One advantage of black-box testing that I quite like
is that we are testing functionality within a real running environment, and interdependencies are also tested.
Another advantage is that tests remain independent of any implementation. For example, during the development
of Perceptor I changed much of the implementation, yet I was able to leave my black-box tests alone.
In the past I have used NUnitForms for black-box testing.
This was my first foray into black-box testing in WPF, and I needed to find another tool because NUnitForms
doesn't support WPF. So I decided to give the White project a try.
White uses UIAutomation,
so can be used with both Windows Forms and WPF applications.
Getting started with White merely involves referencing the White assemblies
and starting an instance of our application in a unit test, as the following excerpt shows.
[TestInitialize]
public void TestInitialize()
{
var startInfo = new ProcessStartInfo("DanielVaughan.LearningUI.Wpf.exe",
DanielVaughan.LearningUI.App.TestingArg);
application = Core.Application.Launch(startInfo);
window = application.GetWindow(DanielVaughan.LearningUI.Window_Main.WindowTitle,
InitializeOption.NoCache);
}
In order to have Perceptor not attempt to use the WCF during the test,
we use an argument to let it know that it is being black-box tested. Once we start
the application we use White to get a testable representation of the application.
The test method uses the window instance to locate and manipulate UIElements.
Among other things, we are able to set textbox values, click buttons, and switch tabs.
It appears that some elements are not yet supported, such as the Expander
control. I was using the rather old release version, and others may be better of acquiring and building
the source via a subversion client.
[TestMethod]
public void WindowShouldLearnFromNavigation()
{
Assert.IsNotNull(window);
textBox_ApplicationSearch = window.Get<TextBox>("textBox_Search");
Assert.IsNotNull(textBox_ApplicationSearch);
var resetButton = window.Get<Button>("Button_ResetLearning");
Assert.IsNotNull(resetButton);
var tabPageSelection = window.Get<TabPage>("TabItem_SelectEmployee");
Assert.IsNotNull(tabPageSelection);
...
var forwardButton = window.Get<Button>("Button_Forward");
Assert.IsNotNull(forwardButton);
forwardButton.Click();
Thread.Sleep(pausePeriodMs);
Assert.IsTrue(tabItemDetails.IsSelected);
Assert.IsTrue(phoneTextBox.IsFocussed, "phoneTextBox should be focused.");
}
Another nicety of black-box testing is that we don't need to worry about creating mocks.
There are, of course, disadvantages to black-box testing compared to traditional white-box testing.
But there's no reason why we can't use both!
Figure: Test results for unit tests.
Possible Applications
A version of Perceptor could be used in Visual Studio to present the appropriate tool window when a particular designer,
with a particular state, is selected.
Perceptor could prove especially useful in areas such as mobile phone interfaces, where the user's ability
to interact with the interface is inhibited by limited physical input controls.
Likewise, people with certain disabilities, who have a limited capacity to manipulate the user interface may also benefit.
Perhaps this kind of predictive UI technology could be classified
as a fifth-generation user interface technology (5GUI).
This suggestion is based on the way in which programming language classification,
in particular 5GL, is defined.
The following is an excerpt from the Wikipedia entry.
While fourth-generation
programming languages are designed to build specific programs,
fifth-generation languages are designed to make the computer solve a given problem without the programmer.
This way, the programmer only needs to worry about what problems need to be solved and what conditions
need to be met, without worrying about how to implement a routine or algorithm to solve them.
Over time, Perceptor learns how the user interface should behave, removing the need for programmer intervention.
Thus the classification 5GUI.
Conclusion
In this article we have seen how Perceptor tracks a user's behaviour while he or she
interacts with the user interface, and induces the training of a neural network.
We also saw how Perceptor is able to save its prediction data, either locally or remotely.
Knowledge acquired by the neural network is used to predict the user's navigation behaviour.
This allows for a dynamic and evolving interface not encumbered by rigid, predefined business rules.
Through the application of AI to user interfaces we have a tremendous opportunity to increase
the usability of our software. The burden of hardwiring behaviour directly into our user interfaces can be reduced,
and rules can be dynamic and refined over time.
By combining the visual appeal and richness afforded to us by technologies such as WPF,
we are able to move beyond merely reactive UIs, to provide a new level of user experience.
I hope you find this project useful. If so, then I'd appreciate it if you would rate it and/or leave feedback below.
This will help me to make my next article better.
Future Enhancements
- Modify the neural network to use Long Short Term Memory
or an alternative progressive recurrent learning strategy.
History
March 2009
