Emulate Human Mouse Input with Bezier Curves and Gaussian Distributions

• Download source - 29.5 KB

Introduction

If one day you find yourself trying to write an application that pretends to be a human (i.e. moves the mouse and clicks buttons in a believable manner), this is one way to solve the problem.

I can think of very few (if any), legitimate reasons to use this code but perhaps Big Brother is watching you and instead you'd rather be down the pub?

The Problem

In .NET, it's very easy to send the mouse cursor to a position on screen:

 System.Windows.Forms.Cursor.Position = new System.Drawing.Point(x, y); 

We can even move it in straight lines:

for (int count = 1; count < 100; count++)
System.Windows.Forms.Cursor.Position 
            = new System.Drawing.Point(count, count); 

It's also relatively easy to call the Windows API to send mouse click events using the signature:

[DllImport("user32.dll")]
public static extern void mouse_event(int dwFlags, int dx, int dy, int dwData, int dwExtraInfo); 

But the average human wrist doesn't trace out straight lines and each line it does trace will be subtly different. Add to that a human is imperfect and doesn't always land the pointer accurately over the target first time. So if we're to automate our mundane clickety-click tasks and deceive Big Brother at the same time, then we have two separate problems to solve:

1. Generating a curved mouse movement
2. Distributing semi-accurate mouse clicks across a target

A Solution

1. Rather than a simple circular arc, we can reasonably argue that mouse movements are somewhat better represented by a Bezier curve. Here is a nice graphical representation of Bezier curves and for our purposes we will use a simple 2D quadratic Bezier spline. This ensures that the curvature of the arc varies smoothly along its length.

   The excellent article Bezier Curves Made Simple by Tolga Birdal presents the algorithm we will use to generate intermediate points between the mouse start position and the intended target.

2. To mimic inaccuracy in selecting the target, let's use a random normal distribution where the centre of the target co-ordinates are located at the top of the bell curve (i.e. the mean value).

The problem here is that the BCL Random class will generate a uniform distribution across the target. Not very realistic (), so let's transform it using the Box-Muller method as illustrated on stackoverflow and shown below:

Random gaussian = new Random();
bool haveNextNextGaussian = false;
double nextNextGaussian;  

public double NextGaussian()
        {
                double v1, v2, s;
                do
                {
                    v1 = 2 * gaussian.NextDouble() - 1;   
                    v2 = 2 * gaussian.NextDouble() - 1;
   
                    s = v1 * v1 + v2 * v2;
                } while (s >= 1 || s == 0);
                double multiplier = Math.Sqrt(-2 * Math.Log(s) / s);
                nextNextGaussian = v2 * multiplier;
                
                return v1 * multiplier;
        } 

So let's bring it all together in a function that will transform the intended X-Y co-ordinates of a move and click into an approximate list of points that we can then use to drive the mouse.

First off, let's modify the actual target:

public List<Point> MouseMoveAndClick(int x, int y)
{
int pointerAccuracy = 10;  
int targetX = x + Convert.ToInt32(pointerAccuracy * targetDistribution.NextGaussian());
int targetY = y + Convert.ToInt32(pointerAccuracy * targetDistribution.NextGaussian()); 

Then, we have to derive the control point (the co-ordinate that governs the curvature of our spline), which I've arbitrarily chosen to be a vector (of random length) normal from the midpoint between the start and the target:

//declare the original pointer position
int originalX = System.Windows.Forms.Cursor.Position.X;
int originalY = System.Windows.Forms.Cursor.Position.Y;

//find a mid point between original and target position
int midPointX = (x - targetX) / 2;
int midPointY = (y - targetY) / 2;

//Find a co-ordinate normal to the straight line between start and end point, starting at the midpoint and normally distributed
//This is reduced by a factor of 4 to model the arc of a right handed user.
int bezierMidPointX = Convert.ToInt32((midPointX / 4) * (midpointDistribution.NextGaussian()));
int bezierMidPointY = Convert.ToInt32((midPointY / 4) * (midpointDistribution.NextGaussian()));

With our start co-ordinates, our control points and our target points, we can generate the intermediate points:

BezierCurve bc = new BezierCurve();
double[] input = new double[] 
{ originalX, originalY, bezierMidPointX, bezierMidPointY, targetX, targetY };
                
int numberOfDataPoints = 1000;
double[] output = new double[numberOfDataPoints];
                
//co-ords are couplets of doubles hence the / 2
bc.Bezier2D(input, numberOfDataPoints / 2, output);

int pause = 0;

Then looping through the co-ordinates, we can modify the mouse position. For added fun, we can even slow the movement down as our human zeros in on his target:

List<System.Drawing.Point> points = new List<Point>();
for (int count = 1; count != numberOfDataPoints - 1; count += 2)
{
    Point point = new System.Drawing.Point((int)output[count + 1], (int)output[count]);
    points.Add(point);
    System.Windows.Forms.Cursor.Position = point;

    //we can vary when we pause between moving from point to point, but to replicate how a user's action will slow down prior to clicking:
    if ((count % 10) == 0)
        pause = 100 + ((count ^ 5) / (count * 2));

    System.Threading.Thread.Sleep(pause);                    
} 

Finally, send the left click events to the API:

    //Use the Win32 API to send a mouse down/ mouse up events separated by an arbitrary pause
    mouse_event((int)(MouseEventFlags.LEFTDOWN), 0, 0, 0, 0);
    System.Threading.Thread.Sleep(50);
    mouse_event((int)(MouseEventFlags.LEFTUP), 0, 0, 0, 0);

    return points;
} 

Conclusions

Of course, to make serious use of this method, you'll have to programmatically discover the locations of the targets your simulated human will click. Perhaps that will be the subject of a more extensive article, but for now feel free to tinker with the attached example.

History

• #1 Code and explanation