Reputationator - Part 1 of 4 (this article)
Reputationator - Part 2 of 4 [
^]
Reputationator - Part 3 of 4 [
^]
Reputationator - Part 4 of 4 [
^]
Introduction
As most of you are probably aware, CodeProject awards reputation points for various user activities and events that occur on the site. The purpose of this award system is to promote participation by the users. Observing a given user's reputation points can be used as an indicator as to how long the user has been a member, but more importantly, how much the user participates.
Since its inception, the reputation points system has been the subject of a large number of threads on the site, and I understand this to indicate that a high level of importance is placed on the system. This importance (real or imagined) is the impetus for the writing of this article.
I've never really been happy with the way a users reputation status is displayed. All we get is the current value, as well as a static graph that shows accumulation since the dawn of CodeProject time. Once you get up above 50,000 points, spread over (for me) 11 years of data, it's difficult to do more than just "ohh! ahh!" at the graph.
To be honest, following one's reputation scores here involves a certain level of narcissism, and since I'm currently "up there" in points (as of this writing, I'm 3rd in overall points among the now more than 8 million users), I'm being especially narcissistic. This is the entire reason for this apps existence.
General
This application is comprised of several assemblies:
- ReputationScraperSvc.EXE - a Windows service which scrapes your Codeproject profile web page for reputation scores on a regular (configurable) basis. The use of this service is optional, but highly recommended.
- ReputationLib.DLL - a library of classes used by one or more of the other assemblies in the solution.
- DBConnectionTester.EXE - an application that allows you to test your SQL Server connection string before installing the Windows service.
- RepSvcManager.EXE - an application used to manage the Windows service without requiring you to get your hands dirty on the Windows command line. (It requires admin privileges in order to be used.)
- Reputationator.EXE - The primary application, which displays charts and other fuunctionality related to the display of reputation points.
an optional Windows service that scrapes the specified user's reputation scores from their profile page, and a Windows Forms app that displays the data with a selection of graphs. The data is stored in a SQL Server database.
The downside of this app is that it can only show data from the point at which you start using it. Historical data is essentially impossible to determine beyond the reputation points you've already attained. In other words, the app is a little late to the reputation party, but it did show up with an assortment of party favors. Without further ado, let's get dirty in the code.
At this time, this application is only supposed to be used to track your own reputation points. More functionality might be added in the future, but like everything else in life, there are no guarantees. If we happen to survive the end of the Mayan Calendar, or we're able to weather the inevitable Zombie Apocalypse, maybe I'll add more stuff.
Technologies Used
The following techniques and technologies were used in the creation of the software described in this article series. However, these articles do not profess to explain theory and best practice regarding said techniques. The list below is merely provided for those looking for real-world examples. Remember, google is free to use and should be available in most countries. If you have general questions about the items below that have nothing to do SPECIFICALLY with this article, I recommend you post a question on the appropriate Codeproject forum.
- Windows Forms applications
- Windows services
- Interaction between a desktop application and a Windows service
- Multi-threading (with UI updates)
- SQL server connections
- LINQ (including Lambda expressions)
- Using the HttpWebRequest object (sort of)
- Parsing returned HTML documents
- Use of the Microsoft .Net Chart library
- Extension methods
- Static classes
- Generic Collections
- Custom events
- The Windows API and interop services
Topic Of Discussion
In this part of the series, we'll be discussing the Windows service, the common library used by both the service and the Windows Forms application itself, and the two utility applications described earlier.
IMPORTANT! - Preparation For Use - IMPORTANT!
IMPORTANT! You MUST create the SQL Server database used by this software BEFORE running applications or installing the Windows service. I've included a ReputationatorDB.sql file in the download file. To create the database:
- Open Sql Manager Studio (Express).
- Open the ReputationatorDB.sql file.
- Execute the loaded SQL file.
That will create the appropriate table and stored procedures used by this software. If, while using the software, you get exceptions about not being able to find the database, (see the next section discussing the DBConnectionTester).
ReputationScraperSvc - The Windows Service
It's hell getting old. On top of being pretty lazy, my short-term memory is getting worse every day. I can go days without doing something simply because I forgot I had to do it - even if it's necessary to do that thing every day. This is the reason I wrote a Windows service to handle the scraping chores.
It's a simple service that does one thing - it scrapes the user's profile page here on CodeProject on a regularly scheduled basis (as specified by the user). As it sits, there is very little that's actually *in* the windows service application. Here are the notable points.
If you're interested in the minutia, keep reading. Otherwise, download and install the service.
Installing/Uninstalling The Service
The steps to install the service:
- Make sure that SQL Server Express is installed (along with Management Studio)
- Run the SQL scripts provided in the solution ZIP file.
- Modify the config file to use YOUR CodeProject UserID
- Modify the config file to use the appropriate connection string (what's already there should be sufficient for most of you).
- Install the Windows service.
Included in the solution is a small utility application that allows you to test your database connection string BEFORE installing the service. You should run this utility AFTER running the SQL scripts that installs the Reputationator database into your SQL Server instance. Just copy the connection string into the provided TextBox, and click the Test button. If everything is okay, you'll see a big green "SUCCESS!!" on the form. Otherwise, you'll get a red "FAIL!!" displayed, along with the exception encountered to help you diagnose the problem. Example screenshots:
During development of the Service Manager application, I often came to a point where, for one reason or another, I had to uninstall the service through the Windows command line, and it became so tedious that I created two batch files - INSTALL.BAT and UNINSTALL.BAT - to avoid the hassles of the command line requirements for using installutil.exe. I've included those batch files in the solution, but be aware that you will have to change them to suit your own system's folder hierarchy.
Once installed, the service is configured to automatically start (that's why you have to do all that other stuff before installing the service).
Initialization
Settings used by the service are maintained in - you guessed it - the app.config file for the service. The settings are:
<applicationSettings>
<ReputationScraper.RepSettings>
<setting name="ScrapeTime" serializeAs="String">
<value>23:30:00</value>
</setting>
<setting name="ConnectionString" serializeAs="String">
<value>Server=.\SQLEXPRESS;Database=Reputationator;Trusted_Connection=True;</value>
</setting>
<setting name="UserID" serializeAs="String">
<value>7741</value>
</setting>
<setting name="Interval" serializeAs="Int32">
<value>4</value>
</setting>
</ReputationScraper.RepSettings>
</applicationSettings>
- UserID - This is YOUR CodeProject UserID. Mine is shown in the example above.
- ScrapeTime - This is the base time at which you want to scrape your info, and should be specified in 24-hour format (hours, minutes, and seconds). My advice is to set it for sometime late in the day (like the example above) so that the last scrape shows the most recent info for the current day.
- Interval - This value indicates the interval in hours between scrapes. This allows you to specify from 0 to 24 hours between service scrape events. using the values 0 and 24 will result in a once-per-day scrape at the time specified in the ScrapeTime setting. As you can see, I have my copy set to scrape every four hours.
- ConnectionString - This is the SQL Server connection string for your desired server. In my case, it's on the same machine I'm running the service. Yours may be different.
The Schedule Thread
The schedule thread is the heartbeat of the service, and it runs until you stop the service. The thread method itself starts out by determining the next (or first) scrape time:
private void RunSchedule()
{
DateTime scrapeOn = new DateTime(0);
scrapeOn = CalculateNextScrapeTime(scrapeOn, Globals.Interval);
The CalculateNextScrapeTime Method
This method determines the next scrape time based on a number of factors. The description was hard to come up with because I know what it's doing and why. I hope to convey that knowledge into something that makes sense to you - the as-yet uninitiated. For purposes of example, we'll be using the values shown above in MY settings file - a scrape time of 11:30 pm, and an interval of four hours between subsequent scrapes.
The parameters passed to this method are the last scrape time, and the desired interval (in hours) to the next scrape time. The very first thing we need to do is perform a sanity check on the interval. We need to make sure that the interval hours is greater than 0 (zero), but no more than 24.
private DateTime CalculateNextScrapeTime(DateTime lastScrape, int hours)
{
hours = (hours <= 0) ? 24 : Math.Min(hours, 24);
Next, we check the last scrape time. If it's got 0 ticks, we assume this is the first time the next scrape time is being calculated, so we set the last scrape time to the current time.
DateTime newScrape;
if (lastScrape.Ticks == 0)
{
lastScrape = DateTime.Now;
Then, we initialize the next scrape time to midnight plus the minutes specified in Settings.ScrapeTime.
newScrape = (lastScrape.Date).AddMinutes(Globals.TimeToScrape.Minutes);
The user can specify ANY hour to scrape, and this gives us our starting point for determining the first scrape event, but this doesn't tell us when the first scrape is supposed to actually take place. I've specified 11:30 PM as my time to scrape, so we have to determine WHEN we are in relation to that start time. The first thing we have to do is to determine the hour offset from midnight of the current day. It's calculated by taking the modulus of the interval (4 in our example) to the specified start hour (23 in my example data file). In our case, the offset will be calculated as "3".
int offset = Globals.TimeToScrape.Hours % hours;
Now, we're ready to actually calculate the next scrape time. Remember, we've already established the date and time as midnight of the current day plus the scrape time minutes (30 in our example). All we have to do is add hours until the new date/time exceeds the last date/time. Of course, it's not quite that simple.
Before adding hours to our new date time, we have to determine how many hours to add. If the current hour of the new date/time is 0, that means we haven't yet added our offset hours. So, we add those hours here. If, on the other hand, the hour is greater than zero, we add the specified interval hours.
while (newScrape <= lastScrape)
{
if (newScrape.Hour == 0)
{
newScrape = newScrape.AddHours(offset);
}
else
{
newScrape = newScrape.AddHours(hours);
}
}
}
If the last scrape date had more than 0 ticks, we assumed that we merely have to add the interval hours to the last scrape time, and we're done!
else
{
newScrape = lastScrape.AddHours(hours);
}
return newScrape;
}
As an example of the method's functionality, we will assume the following:
- A Settings.ScrapeTime of 23:30
- A Settings.Interval of 4 hours
- The service is installed and started at 11:15am
The first run through this method would produce a next scrape date/time of 11:30am, and subsequent scrape times would be four hours apart, at 3:30pm 7:30pm, etc.
Picking Up Where We left off in the Thread Method
Now that we've calculated our first scrape time, we enter the loop that runs until the service is stopped. Simply, if the current date/time is greater than the scheduled date/time a scrape process is started. Since the scrape process is also run in its own thread, we have to take steps to keep the scrape from starting again while another is in progress, so we immediately set the next scrape time, and then start the scraping process (this is handled in the common library).
int tick = 1000;
while (true)
{
if (DateTime.Now > scrapeOn && !m_scraping && this.m_serviceState == ServiceState.Running)
{
scrapeOn = CalculateNextScrapeTime(scrapeOn, Globals.Interval);
m_repScraper.ScrapeWebPage();
}
else
{
Thread.Sleep(tick);
}
}
}
ReputationLib - The Common Library
This library contains several classes utilized by both the Windows service, and the WinForms application. The following content describes the four most important classes in depth, and just touches on the remaining classes.
The RepItem Class
When the CodeProject user profile page is scraped, it retrieves whatever points values exist for each reputation category, and stores them as RepItem objects. there is not functionality contained in this object beyond being inheriting INotifyPropertyChanged. The reason in inherits this class is because I wanted to provide support for WPF data binding. In the interest of completeness, here's the code for the item (sans comments).
public class RepItem : INotifyPropertyChanged
{
#region INotifyPropertyChanged
public event PropertyChangedEventHandler PropertyChanged;
protected void OnPropertyChanged(string name)
{
PropertyChangedEventHandler handler = PropertyChanged;
if (handler != null)
{
handler(this, new PropertyChangedEventArgs(name));
}
}
#endregion INotifyPropertyChanged
private const int MAX_CPID_LENGTH = 12;
private int m_value;
private int m_cpUserID;
private DateTime m_timeScraped;
private RepCategory m_category;
private int m_changeValue;
public int ChangeValue
{
get { return m_changeValue; }
set { m_changeValue = value; OnPropertyChanged("ChangeValue"); }
}
public int Value
{
get { return m_value; }
set { m_value = value; OnPropertyChanged("Value"); }
}
public DateTime TimeScraped
{
get { return m_timeScraped; }
set { m_timeScraped = value; OnPropertyChanged("TimeScraped"); }
}
public RepCategory Category
{
get { return m_category; }
set { m_category = value; OnPropertyChanged("Category"); }
}
public int CPUserID
{
get { return m_cpUserID; }
set { m_cpUserID = value; OnPropertyChanged("CPUserID"); }
}
public RepItem()
{
Value = 0;
ChangeValue = 0;
TimeScraped = new DateTime(0);
Category = RepCategory.Unknown;
CPUserID = -1;
}
}
The RepItemCollection Class
This class represents the collection of reputation items. When I deal with collections, I like to create a class that inherits from List. This makes it easier to deal with in other parts of the code. This technique also allows me to abstract away code that would otherwise sit in code external of the collection, thus reducing the lines of code to call this functionality to just a line or two, keeping the outward facing code cleaner and easier to maintain.
This collection lives in the RepScaper class because RepScraper is responsible for updating the collection. The methods found in this class serve utility functions hat relate to the collection.
public bool GetData()
This method is responsible for retrieving existing items from the database. This method also controls how far back to go to retrieve data. Currently, this is set to 365 days because quite honestly I saw no use to go back any further at this time. HOWEVER, data older than 365 days is still kept in the database in case anyone wants to write code that retrieves this data.
The return value indicates success/failure of the retrieval process.
public bool GetData()
{
if (Globals.UserID <= 0)
{
throw new Exception("Codeproject UserID not specified.");
}
if (string.IsNullOrEmpty(Globals.ConnectionString))
{
throw new Exception("Connection string not specified.");
}
bool result = false;
SqlConnection conn = null;
SqlCommand cmd = null;
SqlDataReader reader = null;
DataTable data = new DataTable();
DateTime thePast = DateTime.Now.AddYears(-1);
try
{
conn = new SqlConnection(Globals.ConnectionString);
cmd = new SqlCommand("GetData", conn);
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.Parameters.AddWithValue("@user", Globals.UserID);
cmd.Parameters.AddWithValue("@date", thePast);
conn.Open();
reader = cmd.ExecuteReader();
if (reader != null && reader.HasRows)
{
data.Load(reader);
PopulateList(data);
result = true;
}
}
catch (Exception ex)
{
if (ex != null) {}
}
finally
{
if (conn != null)
{
conn.Close();
}
}
return result;
}
private void PopulateList(DataTable data)
This method populates the collection from the database. The DataTable parameter is passed from GetData. The reason I didn't just put the code into GetData is because the GetData method was becoming too large for my own comfort. Indeed, GetData is the only method that calls populate list, but it just felt better abstracting out the functionality.
private void PopulateList(DataTable data)
{
foreach (DataRow row in data.Rows)
{
int value = Convert.ToInt32(row["PointValue"]);
RepItem item = new RepItem()
{
Category = Globals.IntToEnum (Convert.ToInt32(row["Category"]), RepCategory.Unknown),
TimeScraped = Convert.ToDateTime(row["ScrapeDate"]),
Value = value,
CPUserID = Convert.ToInt32 (row["CPUserID"]),
ChangeValue = 0
};
this.Add(item);
}
FillDataHoles();
CalcChangedValues();
}
public void AddOrUpdate(RepCategory cat, DateTime date, int user, int points)
This method is called by the RepScraper object in order to add new rep items, or update existing items in the collection. The reason this method does both add and update is because the RepScraper class can be called upon to scrape your profile page from within the WinForms application via a button click. Since the collection only needs one set of data for a given day, scraping from the UI may be performed several times, so this method allows both operations.
public void AddOrUpdate(RepCategory cat, DateTime date, int user, int points)
{
List<RepItem> items = null;
items = (from item in this
where (item.Category == cat && item.CPUserID == user && item.TimeScraped == date)
select item).ToList();
if (items.Count == 1)
{
((RepItem)items[0]).Value = points;
}
else
{
int change = 0;
if (items.Count > 1)
{
change = points - items.Last().Value;
}
this.Add(new RepItem() { Category = cat,
CPUserID = user,
TimeScraped = date,
Value = points,
ChangeValue = change });
}
}
public void UpdateDatabase()
This method actually updates the database. It doesn't determine what to do in the database, because that's a job for the stored proicedure. All it does is populates some SqlCommand parameters, and calls the stored procedure. The rest is magic (although the stored procedures are all explained in the database section below).
public void UpdateDatabase()
{
RemoveExpiredData();
SqlConnection conn = null;
SqlCommand cmd = null;
string query = "";
int records = 0;
DateTime now = DateTime.Now.Date;
List<RepItem> items = (from item in this
where item.TimeScraped == now
select item).ToList();
foreach(RepItem item in items)
{
query = "AddOrUpdate";
try
{
if (conn == null)
{
conn = new SqlConnection(Globals.ConnectionString);
}
if (cmd == null)
{
cmd = new SqlCommand(query, conn);
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.Parameters.Add("@user", SqlDbType.Int);
cmd.Parameters.Add("@date", SqlDbType.DateTime);
cmd.Parameters.Add("@category", SqlDbType.Int);
cmd.Parameters.Add("@points", SqlDbType.Int);
}
cmd.CommandText = query;
cmd.Parameters["@user"].Value = item.CPUserID;
cmd.Parameters["@date"].Value = item.TimeScraped;
cmd.Parameters["@category"].Value = (int)(item.Category);
cmd.Parameters["@points"].Value = item.Value;
if (conn.State == System.Data.ConnectionState.Closed)
{
conn.Open();
}
records = cmd.ExecuteNonQuery();
if (records != 1)
{
throw new Exception("Insert did not succeed");
}
}
catch (Exception ex)
{
if (ex != null) {}
}
}
if (conn != null && conn.State == System.Data.ConnectionState.Open)
{
conn.Close();
}
}
private void CalcChangedValues()
This method calculates the amount changed from the current category's value to the prior day's value for that categor. It provides data for the Accumulated By Day column chart in the WinForms application.
private void CalcChangedValues()
{
foreach(RepCategory cat in Enum.GetValues(typeof(RepCategory)))
{
if (cat != RepCategory.Unknown)
{
List<RepItem> list = (from item in this
where item.Category == cat
select item).ToList();
for (int i = 1; i < list.Count; i++)
{
list[i].ChangeValue = list[i].Value - list[i-1].Value;
}
}
}
}
private void FillDataHoles()
During development of the WinForms application, I freqently had to uninstall the service so I could build the solution. After doing the build, I frequently forgot to reinstall the service, causing holes in the data (no data for a day, or more). This was causing problems in the Accumulated By Day column chart because the calculated changes were sometimes astronomical, thus adversely skewing the trend lines. So, after loading the data from the database, this method is called to fill in the missing data. Data holes are avoided by running the windows service.
private void FillDataHoles()
{
bool needSorting = false;
foreach(RepCategory cat in Enum.GetValues(typeof(RepCategory)))
{
if (cat != RepCategory.Unknown)
{
List<RepItem> list = (from item in this
where item.Category == cat
select item).ToList();
if (list.Count > 1)
{
for (int i = 1; i < list.Count; i++)
{
RepItem currItem = list[i];
RepItem prevItem = list[i-1];
TimeSpan span = currItem.TimeScraped.Date - prevItem.TimeScraped.Date;
if (span.Days > 1)
{
needSorting = true;
int value = prevItem.Value + (int)(Math.Ceiling((double)(currItem.Value - prevItem.Value) / 2));
this.Add(new RepItem(){ Category = cat,
TimeScraped = currItem.TimeScraped.AddDays(-1),
CPUserID = currItem.CPUserID,
Value = value,
ChangeValue = 0 });
}
}
}
}
}
if (needSorting)
{
this.Sort(new GenericComparer<RepItem>(new string[]{"TimeScraped", "Category"}, GenericSortOrder.Ascending));
}
}
private void RemoveExpiredData()
This method removes expired data (anything older than 365 days) from the collection, and is only called when a manual scrape has been performed by the Windows Forms app.
private void RemoveExpiredData()
{
DateTime now = DateTime.Now.Date;
var x = (from item in this
where TimeSpan.FromTicks(now.Ticks - item.TimeScraped.Ticks).Days > MAX_DAYS
select item);
foreach(RepItem item in x)
{
this.Remove(item);
}
}
private List<RepItem> GetItemsByDateAndCategory(DateTime dateStart, DateTime dateEnd, DisplayCategories categories)
This method is a helper for the various GetHighestXXX and GetLowestXXX methods described below, and merely returns a list of category items that have the same category as those in the specified display categories.
private List<RepItem> GetItemsByDateAndCategory(DateTime dateStart, DateTime dateEnd, DisplayCategories categories)
{
List<RepItem> list = (from item in this
from cat in categories
where (item.Category == cat && item.TimeScraped.Between(dateStart, dateEnd, true))
select item).ToList();
return list;
}
public GetHighestDailyValue (three overloads)
This method is called from the various charts and is used to determine the highest daily reputation value so we can properly adjust the Y-axis in the charts (in the Winforms application). The three overloads aren't really necessary, as they call each other in a cascade as the process progresses. The cascade starts with the first overload shown below (called from the chart classes). It should be fairly easy to follow it:
public int GetHighestDailyValue(DisplayCategories categories)
{
DateTime dateStart = new DateTime(0).Date;
DateTime dateEnd = DateTime.Now.Date;
return GetHighestDailyValue(dateStart, dateEnd, categories);
}
public int GetHighestDailyValue(DateTime dateStart, DateTime dateEnd, DisplayCategories categories)
{
int value = 0;
foreach (int i in Enum.GetValues(typeof(RepCategory)))
{
RepCategory category = Globals.IntToEnum(i, RepCategory.Unknown);
if (category == RepCategory.Unknown)
{
continue;
}
List<RepItem> list = GetItemsByDateAndCategory(dateStart, dateEnd, categories);
if (list != null)
{
value = GetHighestDailyValue(list);
}
}
return value;
}
public int GetHighestDailyValue(List<RepItem> list)
{
int value = list.Max((item) => item.Value);
return value;
}
public int GetLowestDailyValue (three overloads)
This method is called from the various charts and is used to determine the lowest daily reputation value so we can properly adjust the Y-axis in the charts (in the Winforms application). The three overloads aren't really necessary, as they call each other in a cascade as the process progresses. The cascade starts with the first overload shown below (called from the chart classes). It should be fairly easy to follow it:
public int GetLowestDailyValue(DisplayCategories categories)
{
DateTime dateStart = new DateTime(0).Date;
DateTime dateEnd = new DateTime(0).Date;
return GetLowestDailyValue(dateStart, dateEnd, categories);
}
public int GetLowestDailyValue(DateTime dateStart, DateTime dateEnd, DisplayCategories categories)
{
int value = 0;
foreach (int i in Enum.GetValues(typeof(RepCategory)))
{
RepCategory category = Globals.IntToEnum(i, RepCategory.Unknown);
if (category == RepCategory.Unknown)
{
continue;
}
List<RepItem> list = GetItemsByDateAndCategory(dateStart, dateEnd, categories);
if (list != null)
{
value = GetLowestDailyValue(list);
}
}
return value;
}
public int GetLowestDailyValue(List<RepItem> list)
{
int value = list.Min((item) => item.Value);
return value;
}
public int GetHighestDailyChangeValue (three overloads)
In the interest of brevity, suffice it to say that these methods are the same as the GetHighestDailyValue methods, except that they return the highest ChangedValue (the amount that a given category changed from one day to the next).
public int GetLowestDailyChangeValue (three overloads)
In the interest of brevity, suffice it to say that these methods are the same as the GetLowestDailyValue methods, except that they return the lowest ChangedValue (the amount that a given category changed from one day to the next).
public int GetLatestPointValue(RepCategory category)
This method gets the last value for the specified category, and is used for the pie chart in the Windows Forms application, as well as the statitics panel above the chart display.
public int GetLatestPointValue(RepCategory category)
{
int value = 0;
var repItem = (from item in this
where item.Category == category
select item).LastOrDefault();
if (repItem != null)
{
value = repItem.Value;
}
return value;
}
public int GetDailyAverage(RepCategory category)
public int GetDailyAverage(RepCategory category)
{
int value = 0;
double total = 0;
DateTime now = DateTime.Now.Date;
DateTime periodStart = now.AddDays(-((int)RepPeriod.Year));
List<RepItem> list = (from item in this
where item.Category == category && item.TimeScraped.Between(periodStart, now, true)
select item).ToList();
foreach (RepItem item in list)
{
total += item.ChangeValue;
}
value = (int)(Math.Ceiling(total / list.Count));
return value;
}
public GetStartPeriod(RepPeriod period)
This is a helkper method for the GetCurrentPeriodPoints methods, and simply calculates a date based on the specified reputation period parameter.
public DateTime GetPeriodStart(RepPeriod period)
{
DateTime periodStart = DateTime.Now.Date;
switch (period)
{
case RepPeriod.Week :
periodStart = periodStart.WeekStartDate();
break;
case RepPeriod.Month :
periodStart = periodStart.AddDays(-(periodStart.Day - 1));
break;
case RepPeriod.Year :
periodStart = periodStart.AddDays(-(periodStart.DayOfYear - 1));
break;
}
return periodStart;
}
public int GetCurrentPeriodPoints (two overloads)
This method is called from the Windows Form application for the purpose of populating part of the statistics panel shown above the chart.
public void GetCurrentPeriodPoints(RepPeriod period, RepCategory category, out int value, out int projected)
{
value = 0;
projected = 0;
DateTime now = DateTime.Now.Date;
DateTime periodStart = GetPeriodStart(period);
value = GetCurrentPeriodPoints(period, category);
int avg = GetDailyAverage(RepCategory.Total);
projected = value;
TimeSpan span = now - periodStart;
if (span.Days <= (int)period)
{
int daysLeft = (int)period - span.Days;
projected = value + (daysLeft * avg);
}
}
public int GetCurrentPeriodPoints(RepPeriod period, RepCategory category)
{
int value = 0;
DateTime now = DateTime.Now.Date;
DateTime periodStart = GetPeriodStart(period);
List<RepItem> list = (from item in this
where item.Category == category && item.TimeScraped.Between(periodStart, now, true)
select item).ToList();
foreach (RepItem item in list)
{
value += item.ChangeValue;
}
return value;
}
public int GetLastPeriodPoints(RepPeriod period, RepCategory category)
This method is supposed to be called from the Windows Form application, but at the time of this writing, it's apparently not used. :/
public int GetLastPeriodPoints(RepPeriod period, RepCategory category)
{
int value = 0;
DateTime now = DateTime.Now.Date;
DateTime periodStart = DateTime.Now.Date.AddDays(-((int)period));
List<RepItem< list = (from item in this
where item.Category == category && item.TimeScraped.Between(periodStart, now, true)
select item).ToList();
foreach (RepItem item in list)
{
value += item.ChangeValue;
}
return value;
}
The Service Manager Application
Development of the desktop application associated with this service involved a child form that was intended to be used to install/uninstall and otherwise manage the service. However, the act of stopping/starting/pausing/resuming the service required an elevation of priviliges, which would have forced me to reequire that the entire application be run as administrator. Since that's considered to be "evil by proxy", I had to break that code out into its own utility application.
It's a very simple little program. Its entire purpose in life is to make it easy to manipulate the service, mostly to allow you to change the config file, and without having to install/uninstall the service to realize those changes. You can run this app free and clear of the main UI application, and the first time you do so, it looks like this:
To install the service, simply click the Install button. The status will eventually change to "Running", and the button text will change to "Uninstall". Once you've installed it, you don't need to uninstall it until you want to install a new version.
The Stop/Start and Pause/Resume buttons can only be used if the service is installed. Once stopped or paused, you can make changes to the config file, and simply Start or Resume to make the new configuration take effect. Once the service is installed, the form should look like this:
General Architecture
The app automatically starts a BackgroundWorker object that monitors the status of the service. Once every second, the app tries to find the service and if found, retrieve/display its current status.
private void FormManageSvc_Load(object sender, EventArgs e)
The forms Load event handler handles the initialization. Below, you can see a standard BackgroundWorker setup.
private void FormManageSvc_Load(object sender, EventArgs e)
{
FindPaths();
m_worker = new BackgroundWorker();
m_worker.WorkerReportsProgress = true;
m_worker.WorkerSupportsCancellation = true;
m_worker.ProgressChanged += new ProgressChangedEventHandler(m_worker_ProgressChanged);
m_worker.DoWork += new DoWorkEventHandler(m_worker_DoWork);
m_worker.RunWorkerAsync();
}
The BackgroundWorker Object
There is actually very little for the background worker to do. It Waits the prescribed time interval (1 second), and reports progress, and the ReportProgress handler invokes a method that does the updates the UI.
private void m_worker_DoWork(object sender, DoWorkEventArgs e)
{
BackgroundWorker worker = sender as BackgroundWorker;
int interval = 1000;
while (!worker.CancellationPending)
{
worker.ReportProgress(0);
Thread.Sleep(interval);
}
}
private void m_worker_ProgressChanged(object sender, ProgressChangedEventArgs e)
{
DelegateUpdateForm method = new DelegateUpdateForm(ShowServiceStatus);
Invoke(method);
}
Utility Methods
private void ShowServiceStatus()
This method gets the service and if found, updates the formms controls accordingly.
private void ShowServiceStatus()
{
ServiceController service = GetService();
if (service != null)
{
this.labelStatus.Text = service.Status.ToString();
m_installed = true;
}
else
{
this.labelStatus.Text = "NOT INSTALLED";
m_installed = false;
}
m_service = service;
buttonInstallUninstall.Text = (m_installed) ? "Uninstall" : "Install";
if (m_service != null)
{
buttonStartStop.Text = (m_installed && service.Status == ServiceControllerStatus.Stopped) ? "Start" : "Stop";
buttonPauseResume.Text = (m_installed && service.Status == ServiceControllerStatus.Paused) ? "Resume" : "Pause";
}
buttonPauseResume.Enabled = (service != null && service.CanPauseAndContinue);
buttonStartStop.Enabled = (m_installed);
}
private void GetService()
This method looks for the appropriate service controller, and returns it if found.
private ServiceController GetService()
{
ServiceController[] controllers = ServiceController.GetServices();
var service = (from svc in controllers
where svc.ServiceName == "Reputationator_Service"
select svc).FirstOrDefault();
return service;
}
private void FindPaths()
Part of the initialization in the Load event handler is a call to the FindPaths method. This methd determines the path to the service executable file, as well as the latest installed version of the .Net INSTALLUTIL.EXE application. Finding the service assembly is a simple matter of retrieving the executing assembly's path. The location of INSTALLUTIL.EXE is a little more work, but is also trivial. Once this is performed, we are ready to manipulate the service.
private void FindPaths()
{
m_svcPath = Assembly.GetExecutingAssembly().Location;
m_svcPath = System.IO.Path.GetDirectoryName(m_svcPath);
m_svcPath = System.IO.Path.Combine(m_svcPath, "ReputationScraperSvc.exe");
string rootpath = System.IO.Path.Combine(System.IO.Path.GetDirectoryName(Environment.SystemDirectory), @"Microsoft.Net\Framework");
DirectoryInfo di = new DirectoryInfo(rootpath);
FileInfo[] files = di.GetFiles("installutil.exe", SearchOption.AllDirectories);
foreach(FileInfo fi in files)
{
if (fi.FullName.Contains(@"Framework\v4"))
{
m_installUtilPath = fi.DirectoryName;
break;
}
}
}
Button Events
buttonInstallUninstall_Click
This event is responsible for launching INSTALLUTIL.EXE so we can install and uninstall the service. The ExternalProcess class is explained in Part 2 of this series, and is essentially a static class that controls a Process object.
private void buttonInstallUninstall_Click(object sender, EventArgs e)
{
string param = (m_installed) ? "/u" : "/i";
m_installing = (param == "/i");
string args = string.Format("{0} {1}", (m_installed)?"/u":"/i", m_svcPath);
ExternalProcess.WorkingDir = m_installUtilPath;
ExternalProcess.SetWorkingDir = true;
ExternalProcess.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
ExternalProcess.Run("installutil.exe", args, true);
}
buttonStartStop_Click
This handler starts or stops the service, whichever is appropriate. The label on the button is also changed to indicate the action that will take place the next time it's clicked.
private void buttonStartStop_Click(object sender, EventArgs e)
{
Button button = sender as Button;
ServiceController service = GetService();
switch (service.Status)
{
case ServiceControllerStatus.Running :
if (service.CanStop)
{
service.Stop();
button.Text = "Start";
}
break;
case ServiceControllerStatus.Stopped :
service.Start();
button.Text = "Stop";
break;
}
}
buttonPauseResume_Click
This handler pauses or continues the service, whichever is appropriate. The label on the button is also changed to indicate the action that will take place the next time it's clicked.
private void buttonPauseResume_Click(object sender, EventArgs e)
{
Button button = sender as Button;
ServiceController service = GetService();
switch (service.Status)
{
case ServiceControllerStatus.Running :
service.Pause();
button.Text = "Resume";
break;
case ServiceControllerStatus.Paused :
service.Continue();
button.Text = "Pause";
break;
}
}
End of Part 1
Part 2 of this 3-part series describes the scraper and common library code. Don't feel bad if you think you might want to skip it. I certainly wanted to when I was writing these articles. :)
If you don't vote my articles with a 5, I won't shoot you with this bacon gun.
Reputationator - Part 1 of 4 (this article)
Reputationator - Part 2 of 4 [^]
Reputationator - Part 3 of 4 [^]
Reputationator - Part 4 of 4 [^]
Major Revision 1
CodeProject Caching
A couple of days ago, I noticed that the total category displayed on the daily change (column) chart seemed a bit - well - out of whack. It was almost twice what it should have been. Upon investigation, I found that it was in fact pulling the data on the page and saving it, but when calculating the changed amount (which it was also doing correctly), it was simply the wrong value. I had been bitten by the Codeproject caching issue where the numbers don't necessarily add up (even manually doing the math on the numbers scraped from the page was showing a variation in the mathematical total vs. the one showing on the web page. What to do?
My solution was to change the application to accept an optional command line parameter that causes the program to adjust existing Total RepItem objects, and if adjusted update the database. The changes were as follows:
- Reputationator.Program.cs - add an
args parameter to the Main method, as well as a switch stament to handle the possible arguments.
- ReputationLib.Globals.cs - add a variable that's set by main when the appropriate parameter has been specified.
- ReputationLib.RepCollections.cs - add code to looks for the
Globals variable when loading data from the database, and if true, executes a new method that will adjust the existing total values to be mathematically correct vs what the web page reports, and then update the database with the new values. Additionally, the method that updates the database was modified to head this problem off in future updates.
Retrieving Data Manually
The form has a button that allows you to scrape your data at any time. When performed, the chart should have been updated to reflect the new data. A Y-Axis position was in fact being added, but the data wasn't showing up. The workaround was to shut the app down, and start it back up again, allowing it to load the data from the database and thus properly represent the data in the chart. With this change, the workaround should no longer be necessary.
User-Reported Errors
The following problems were reported by Simon Bang Terkildsen, and have been addressed:
- When you enter another user id, then it's still the user id stored in the config file that is used.
- Scraping the web page was confusing
Author with Authority because of the similar spelling.
- Countries where comma is not the thousands separator was causing parsing issues in the stated goal points field.
New Feature!
I added a new panel to the top section on the right side labeled You Vs The Leader. What it shows is the approximate date at which you will overtake the current points leader. The longer you use Reputationator, the more accurate this number will become. For the first few days, you may notice fairly erratic differences, but as the average points earned by you and the leader evens out (which will happen over longer periods) the more stable this date will become.
WPF!
I added a mostly-completed WPF version of the Reputationator app, along with an associated Part 4 of this article series.
History
- 29 Aug 2011 : Posted updated article and a new download zip file with finished WPF code
- 21 Aug 2011 : I came across a bug that only happens on the first day of the week (Sunday for me, Saturday for some of you). You'll get a message about no data being found for the specified period, and if you click yes in the message box (to scrape for the data), the app will crash. This was reported by somebody yesterday, and I came across it today. Anyway, a new download is available.
- 20 Aug 2011 : Revision 1 (see section above)
- 14 Aug 2011 - Original article
