Note: Before downloading, please review the Requirements and Setup Instructions.
Introduction
MarketRecorder connects to Interactive Broker’s (IB) TWS API and captures live streaming events for a set of symbols, such as bid, ask, volume, highs, lows, recent trades, etc. This raw event data is stored directly to a new database record every second.
This tool can be used to download market data for different reasons. For myself, I use real-time data to predict stocks using AI but it could be used for just about anything. To use it though, either an Interactive Broker’s account is required or the free demo TWS. The demo has fake information but can be useful for testing.
This article is a write up for MarketRecorder, one of several applications that are loosely connected. This app mainly saves raw real-time tick data directly to a database. Saving this dirty mess of almost random events is important because it closely represents real world trading. There is a second project named BriefMaker that takes this data to the next level by converting this event data into a more familiar and practical time-interval summary such has a high/low/close/vol every 6-seconds. This type of data can be dropped into a spreadsheet with a fixed time interval on one axis. BriefMaker will be covered in another article soon.
Market Recorder just records events. The advantage to saving these streaming events is that events what makes up the market. Markets are a chaotic mess of different events happing almost randomly. This chaotic form of data is what the market is. As soon as this data is converted, say to 6-second summary snapshots by BriefMaker for example, a little bit of the market’s fidelity is lost and cannot be restored. Saving the raw event data allows apps, like BriefMaker, to be re-run with different parameters or code.
Collecting Data with the API
Below is a dataflow chart for MarketRecorder.
Market data first comes in via the internet or dedicated circuit directly to the Interactive Brokers(IB) TWS application. TWS has a built-in API that allows users to do almost anything from accessing market data to checking order status to placing trades.
To retrieve data, we must first establish a TWS API connection. For clarity, I have removed some non-essential lines:
logger.Info("Connecting to Interactive Brokers API.");
IBClient client = new IBClient();
Thread.Sleep(1000);
int retryCt = 5;
while (!client.Connected)
{
Thread.Sleep(2000);
if ((retryCt--) <= 0)
{
logger.Info("Tried to reconnect 5 times but failed - aborting.");
return; }
try
{
logger.Info("Connecting to TWS..." + "(Try " + (4 - retryCt) + " of 5)");
client.Connect(settings.IBHostToConnectTo, settings.IBPortToConnectTo, 1);
client.RequestIds(1);
}
catch (Exception ex)
{
logger.Info("IB Connecting Exception: " + ex.Message);
}
}
logger.Info("TWS Client is now Connected.");
Next, we register some tiker symbols that we are interested in receiving events for:
foreach (var symbol in symbols)
{
Contract item = new Contract(symbol.Symbol.Trim(), symbol.Market, symbol.securityType, "USD");
client.RequestMarketData(symbol.SymbolID, item, null, false, false);
}
Now the application waits for updates. Whenever there is an event from TWS, say a new best bid price, the API fires an event method in the MarketRecorder. This method then captures the data into a BinaryWriter (capturingWriter
):
void client_TickPrice(object sender, TickPriceEventArgs e)
{
int secondFraction = (int)(DateTime.Now.Ticks % TimeSpan.TicksPerSecond / (TimeSpan.TicksPerSecond/256));
lock (capturingWriterLock)
{
capturingWriter.Write((byte)secondFraction ); capturingWriter.Write((byte)e.TickType); capturingWriter.Write((byte)e.TickerId); capturingWriter.Write((float)e.Price); }
...
}
Writing Data to the Database
On each second, the capturingWriter
is uploaded to the StreamMoments
database table.
DateTime time = new DateTime((DateTime.Now.Ticks / TimeSpan.TicksPerSecond) * TimeSpan.TicksPerSecond);
BinaryWriter ProcWriter = capturingWriter;
lock (capturingWriterLock) { capturingWriter = manufacturedWriterForFuture; }
ProcWriter.BaseStream.Position = 0;
ProcWriter.Write(time.Ticks);
ProcWriter.Flush();
OpenAndSendWithWCF(ProcWriter);
byte[] data = ((MemoryStream)ProcWriter.BaseStream).ToArray();
StreamMoment briefToSave = new StreamMoment() { SnapshotTime = time, Data = data };
try
{
dc.StreamMoments.InsertOnSubmit(briefToSave);
dc.SubmitChanges();
}
catch (Exception ex)
{
logger.Error("Exception with SubmitChanges(): {0}", ex.Message);
}
logger.Debug("Snapshot complete ({0} bytes at {1})", data.Length, time.ToString("HH:mm:ss.fff"));
ProcWriter.BaseStream.Dispose();
ProcWriter.Dispose();
manufacturedWriterForFuture = new BinaryWriter(new MemoryStream(MemoryStreamReserveSpace));
Every second MarketRecorder starts a new row in the StreamMoments
table. After each second completes a new row is added to the StreamMoments
table. Each row has a SnapshotTime(DateTime2)
and Data(image)
column:
SnapshotTime: This is the time at the end of the data collection. It is rounded to the end of the second. To get the beginning of the snapshot time, just subtract 1 second.
Data: The data field is what holds the streaming data. The size of it depends on how much data came across the wire and how busy the market was. The first 8 bytes of this field is the DateTimeNow.Ticks
when the stream moment ended. The next 7 bytes is the first event, the next 7 bytes is the second event, and so on.
Each tick is saved in the following format: (7 bytes total):
Format |
Description |
Range |
Byte |
Current sub-second (size: 1/256 second) (~4 ms resolution) |
0 - 255 |
Byte |
Ticker Type - holds the tick data type: high, last, vol, bid, etc. |
0 - 57 |
Byte |
Ticker ID - the symbolID in the Symbols Table (Ex: AMD, INTC, ^DJI) |
0 - 255 |
Float |
Value - this is the actual data. (Ex: volume, price, etc.) |
(float) |
To get the sub-second time for each event, use something like the following:
EventDateTime = SnapshotTime - 1 + (subSecond/256);
The symbols that are downloads are located in the Symbols
Table. It is preloaded with examples.
Bytes were used for the sub-Second and TickerID
in order to save space. This does provide some limitations tough. The sub-second accuracy is limited to 1/256 of a second, or 3.9ms. Because of network connections jitter though, this precision may be good enough. The TickerID
is also a byte so the system, as-is, is limited to 256 tickers – but that is a lot to be downloading at one time. Also, IB has a 100 symbol limit for standard accounts so 256 is well beyond that amount.
Features
NLog logging – This is a popular logging library. Debugging real-time data is difficult because the market data does not stop and wait while we debug our code. Having a nice logger, such as NLog, lets us go back and track down issues.
High Performance – One of the main goals was to minimize latency. When working with trading algorithms, low latency is usually important. Where possible, data is made available as soon as possible for the database or a WCF transfer. To speed things up, a direct link between MarketRecorder and BriefMaker was setup using Window Communication Foundation(WCF). This bypassed a round trip hit to the database.
Built-in auto-reconnect – Sometimes different issues happen, such as a network outage or a problem with TWS, and the API needs to be re-initialized. MarketRecorder checks to make sure data is coming in and if there is no data it will restart the connection. Note: Sometimes when small amounts of data coming in, like afterhours, it will trip the auto-reconnect feature.
Command Line Shutdown – The program can be closed via the command line using “MarketMaker.exe /q
”. For myself, I used to have a scheduled task run each day to close MarketMaker. I now use TWSStart (http://twsstart.free.fr).
The Symbols Table
Below is what the Symbols
table looks like. The columns to the right of ‘Name
’ are not used in MarketRecorder.
SymbolID
: Contains the ID for each symbol. (must be 0-255)
Type
: Used by MarketRecorder when registering symbols.
Market
: Used by MarketRecorder when registering symbols.
Name
: The symbol for the data to retrieve.
FullName
, TradeEnabled
, TradeQuanity
, Value
, AvgVol
, MarketCap
, TickTypesToCapture
fields are not used in MarketRecorder.
Direct WCF Connection
MarketRecorder has built-in functionality to directly connect to other applications, such as BriefMaker. The connection uses a WCF (Windows Communication Foundation) network connection and is used to minimize latency. When each 1-second snapshot is completed it can be shipped directly to another application, bypassing the database. The database record is still written however for historical reasons.
User Interface and Its Functions
Delete All Button - This will erase all the contents in the StreamMoments
table.
Launch Web Demo Button – This opens up the website where users can download and connect to IB’s demo. This is useful for testing.
Future Wish List
MarketRecorder uses the krs.Ats.IBNet library. This has been an excellent library but not been updated since 2012. In the future, I would like to move it to the Interactive Brokers C# API.
A second improvement I would like to add, but am not sure if I’ll ever get around to it, is capturing additional events like level II data and news events. For news, it would be nice to even capture the meaning or mood in numerical values.
Setup Instructions
- Download and mount the starter MS-SQL database. To start out, set the name to “
Focus
”. The DB has several SQL tables but only two are used: Symbols
and StreamMoments
.
- Update the .config file:
- If the database name is not focus, then you will need to update this in the connection string in the .config file.
- Next, you will see
IBHostToConnectTo
and IBPortToConnectTo
... adjust only if needed. The port number can be found in TWS -> File -> Global Configuration -> API -> Settings.
- If needed, configure the
ShutDownTime
. This is the time the program will automatically close. I am in the PST time zone so that is why it is defaulted to 14:15(2:15PM).
- To use the API in TWS, you will first need to enable it. In TWS, go to File -> Global Configuration -> API -> Settings and check the box "Enable ActiveX and socket clients". You might also need to add 127.0.0.1 in the Trusted IP Addresses. Also note the port number. This port number should match the value in the .config file mentioned above.
- Start the MarketRecorder. It will connect to TWS and the DB automatically. If you are going to get an error, it will be within the first few seconds. If it does not connect, review the log details or debug it in Visual Studio.
- After getting everything setup and the test data seems to be working, then feel free to update the list of symbols. If you are going to use BriefMaker, then I would keep a total of 32 stock items (0-31) and avoid editing the Market Indexes (32-38).
Requirements
History
- 2011-11-17 Initially created
- 2012-08-31 Added
FixedStepDispatcherTimer
- 2013-02-11 Add single instance limitation, exit command line option, started WCF
- 2014-01-28 Finished WCF, improved stability
- 2016-03-12 Cleaned up for online posting / published code