An End-to-End Data Loading Solution

After a month-long effort, my work on the end-to-end loading of data into NinjaTrader was finally completed. I hope the time spent on developing the solution will be worthwhile in the long run as I will no longer be required to manually collect, validate, convert and archive the data on a daily basis.

For as little as 5 bucks a month, I have my automation script hosted on Pythonanywhere.com. Although the CPU-time available for my account is barely enough (especially towards month-end due to more data to crunch), the Python host offers the most value for money as compared to other pricier hosts.

The automation script is scheduled to run everyday after market closes. It collects instrument data, validates the data against a secondary source, converts the data to NinjaTrader format, synchronises the converted data to Dropbox and finally archives the raw data into tarballs by month. The Dropbox client on my machine will then synchronise with the server and retrieve all the converted data. The final step of loading the data into NinjaTrader is still manual which I have to perform routinely.

The solution is not perfect yet. I have some ideas on how to improve it and will find time to implement them in the future.

Improvement Plans

• Automate the loading of data into NinjaTrader (the final step).
• Automate the addition and removal of instrument definitions.
• Generate trade summary such as the most active instruments, total traded, etc.
• Send email alerts on exceptional events from the remote script.
• ...

Blank NinjaTrader Instrument Database

If I remember correctly, NinjaTrader is pre-loaded with the definitions for about 17k instruments which are mainly stocks, futures and currency pairs for the US markets. I am running NinjaTrader in a VM environment. Sometimes the Instrument Manager will freeze momentarily when I add or remove an instrument. Since my focus has always been on Malaysian instruments, I embarked on a mission to get rid of all the pre-loaded instruments irrelevant to me.

If you have all the time in the world, you can delete each and every instrument one at a time in the Instrument Manager. It is a shame we cannot select multiple instruments for deletion. If you have some basic knowledge of SQL, a faster way to remove unwanted instruments is to update the NinjaTrader database directly.

NinjaTrader 7 Database Path:
C:\Users\{username}\Documents\NinjaTrader 7\db\NinjaTrader.sdf 

I will not go into details how I purged the database. If you would like to build your own instruments database from scratch, you may download the blank [1] NinjaTrader database file here [2]. Simply replace the file in the above-mentioned path and you are good to go.

---

1. The database is not really blank. It still contains the instrument definitions for the major currency pairs.
2. Download blank NinjaTrader Instrument Database.

The VOL (Volume) Indicator

All indicators in NinjaTrader will not plot the first 20-bar data. There is no exception whatsoever even though what the indicator might do is just plotting the input data as is. The VOL is one such indicator which plots the volume data without any sophisticated calculations. It is quite off-putting for a newbie like me when the basic VOL indicator "does not" work the way I've been accustomed to.

The "fix" is easy and it only requires one additional line in the VOL indicator source file [1].

protected override void Initialize()

{

BarsRequired = 0;

Add(new Plot(new Pen(Color.Blue, 2), PlotStyle.Bar, "Volume"));

Add(new Line(Color.DarkGray, 0, "Zero line"));

}

Even though the "fix" is a one-liner change, it has not been incorporated into the latest NinjaTrader release, i.e. v.7.0.1000.22. Hence, one will have to manually update and re-compile the indicator. Since one cannot save changes made to system indicators from within NinjaTrader, I would suggest one to edit and save the indicator using any text editors. Indicator source files are located in C:\Users\{username}\Documents\NinjaTrader 7\bin\Custom\Indicator.

---

1. Reference: NinjaTrader Support Forum.