Introduction
In this article, I will show an example of how to add IronPython to existing enterprise .NET software systems, and why I think it useful to do so. This isn't a new idea, but hopefully it brings something new to the table.
Background
From Python.org
"Python is a programming language that lets you work more quickly and integrate your systems more effectively. You can learn to use Python and see almost immediate gains in productivity and lower maintenance costs."
From IronPython.net
"IronPython is an open-source implementation of the Python programming language which is tightly integrated with the .NET Framework. IronPython can use the .NET Framework and Python libraries, and other .NET languages can use Python code just as easily. "
Below are some IronPython resources if you are interested in learning more about Python / IronPython.
Got Python?
You don't need to know Python coming into this article. IronPython weaves Python and .NET together very nicely. For C# developers, the learning curve isn't too steep. Here's a peek at some IronPython, and the C# equivalent.
IronPython
from System import DateTime, String
blurb = String.Format("{0} {1}", "Hello World! The current date and time is ", DateTime.Now)
print blurb C#
using System;
namespace IronPython.Example
{
class Program
{
static void Main(string[] args)
{
string blurb = String.Format("{0} {1}", "Hello World! The current date and time is ", DateTime.Now);
Console.WriteLine(blurb);
}
}
} Besides the obvious code differences, the key difference between these code examples is how they are executed. IronPython scripts are executed using the IronPython interpreter, whereas C# code must be compiled then run through the .NET runtime.
For a long time, I've been developing .NET applications in the traditional write, compile, test, fix, recompile, retest, rinse, repeat way. Scripting (such as Python, Perl, etc.) was always something I wanted to learn, but I never found (or took) the time to do it. IronPython was my bridge into Python scripting.
IronPython is bit of a change of pace for .NET developers who haven't been exposed to it. But if you have been coding in C# for years, a change of pace is a good thing.
Why Script?
Here are a few reasons why scripting was a good choice for me.
- I wanted to provide other developers the ability to query the data store using the existing business logic and data access libraries, with the convenience of writing SQL queries.
- I wanted a way to quickly test out design changes to an existing API as I went along quickly. Having to write ad hoc C# code to test the API is very inefficient, and you end up with a bunch of throw away code. Writing scripts interactively seems much more productive to me, and I don't even need to save the query to disk.
- Having other developers writing queries against the API would help with the design of the API, as new ideas for the types of queries are thought up. If the API doesn't support the features that the developers who will use it need, then those features can be added as you go.
- I wanted to play around with a running application. I wanted to be able to access the internal objects, query their properties, maybe even swap in new logic on the fly. When designing applications, it's impossible to conceive of every way users may want to use the applications you build. While I doubt non developers would be writing IronPython scripts, I think that other developers would be able to use this to diagnose production issues quicker if such an interface existed.
- I didn't want to have build a user interface to interact with the API, whether that be a web page, desktop or console application. For me, using a UI is too constraining for testing an API redesign. Scripting allows for not bothering to write UI code to test the API.
- Adding scripting via a web page, developers wouldn't need to use Visual Studio to write code to test the API. Instead, they could just enter their code in a web browser. Good for when you don't have access to a development machine, and good for demonstration purposes.
Disclaimer
I want to state upfront that adding a script interface probably shouldn't be done in a production environment, especially public facing web applications. There's just too big of a chance that someone could do something very very bad. However, on your own development machine or on locked down staging servers, this might be acceptable.
Python Tools
When working with IronPython scripts, it's handy to install the Visual Studio Python Tools. I was initially writing IronPython in Notepad++, but I find the IntelliSense in Visual Studio makes me more productive, and then it saves me from having another editor open.
Configuring IronPython
First, you'll need to install IronPython from here (The current version is 2.7.3). Then you'll need to add references to Microsoft.Scripting.dll, Microsoft.Dynamic.dll, and IronPython.dll from the Iron Python installation folder.
Below is a partial code sample that creates a Python engine and executes a script.
using System;
using System.IO;
using IronPython.Hosting;
using Microsoft.Scripting;
using Microsoft.Scripting.Hosting;
string script = "";
var engine = Python.CreateEngine();
var scope = engine.CreateScope();
var source = engine.CreateScriptSourceFromString(script, SourceCodeKind.Statements);
var compiled = source.Compile();
var result = compiled.Execute(scope);
There is support for multiple engines per application, but for this article, I'm just initializing it once during application startup.
That's enough code to get an application to support Iron Python scripts. Of course, you'll need some way of getting the scripts into the Python engine, which you might do by providing a text input in an application. The code in this article provides a web interface and a Winforms interface of doing just that.
Oh yeah, by the way, this also works in ASP.NET applications too.
IronPython / .NET Integration
The feature I'm most interested in with IronPython is the ability to manipulate .NET objects inside of my applications. To get at the internal objects quickly, I'm just adding the main objects as variables to the ScriptScope, as follows:
var scope = engine.CreateScope();
string variableName = "myObject";
object myObject = new Object();
scope.SetVariable(name, value);
After adding the variable in this way, the variable can be accessed from the IronPython directly. Assuming "myObject" was registered with the ScriptScope as shown above, you access it in the IronPython script by doing something like:
def getMyObject():
return myObject;
obj = getMyObject()
print obj.ToString() For developers new to IronPython (such as myself), this should be enough to get started. My thought was to register the abstract factories to create the Unit of Work / Repositories, maybe the main business logic objects. While these entities can be created from IronPython scripts just by invoking their constructors, the scripts could get cluttered up with the initialization code, especially if you have lots of small classes that need to be wired up (which typically happens when you design loosely coupled code using SOLID methods). Registering the main objects from your application can help reduce the lines of IronPython code you need to write.
Redirecting Script Output
If you want to display the results of your IronPython scripts, you have to redirect the output of the script engine. There are a couple of ways of doing this.
outputStream = new MemoryStream();
outputStreamWriter = new StreamWriter(outputStream);
engine.Runtime.IO.SetOutput(outputStream, outputStreamWriter);
Another way is to redirect output to the console, then redirect the console as follows:
var textWriter = null;
engine.Runtime.IO.RedirectToConsole();
Console.SetOut(TextWriter.Synchronized(textWriter));
Initially, I used the second approach, when the application only supported editing a single Iron Python script at a time. After adding support for having multiple scripts, each with its own output window, I decided to switch to the first option.
Filtering Data
When working with .NET ORM's, you write Linq queries something like:
var query = Customers.Where(c => c.Id > 10);
This is a very simple query, but there is a lot going on here. Below is a sample class that wraps calls to the Where method. There are two overloads to the GetCustomers method, which we will discuss after you have a quick look over the code.
public class CustomerRepository : Repository<Customer>
{
private DbSet<Customer> Customers { get; set; }
public IQueryable<Customer> All()
{
IQueryable<Customer> customers = from c in Customers select c;
return customers;
}
public IQueryable<Customer> GetCustomers(Expression<Func<Customer, bool>> predicate)
{
IQueryable<Customer> customers = All().Where(predicate);
return customers;
}
public IEnumerable<Customer> GetCustomers(Func<Customer, bool> predicate)
{
IEnumerable<Customer> customers = All().Where(predicate);
return customers;
}
} Say you write a lambda expression in IronPython to filter a list of customers, something like:
repo = CustomerRepository()
customers = repo.GetCustomers(lambda c: c.Id > 10)
The overload of GetCustomers that accepts a Func<Customer, bool> as an argument will be invoked. As far as I can tell, IronPython does not support converting lambda expressions to Expression Trees. There is an open issue on the IronPython CodePlex site, asking to support this feature.
This has some implications that you might not be immediately aware of (at least I wasn't).
The first overload of GetCustomers accepts an Expression Tree as an argument.
Expression<Func<Customer, bool>> predicate
The LINQ Provider translates the expression tree into a SQL statement, does the filtering on the database side, then only brings back data that matches the Expression Tree.
The second overload of GetCustomers accepts a simple delegate (Func) as an argument.
Func<Customer, bool> predicate
Since it's just a function, the LINQ Provider cannot generate SQL that does the filtering on the database side. Instead, the entire dataset needs to be enumerated, and the Func<Customer, bool> predicate is used to only include records that match.
Depending on the size of your data set, this may or may not be an issue. But if you want efficient / quick queries through IronPython, then you might have to provide methods in your repositories that do common filtering, to leverage filtering on the database side.
The Code
The code behind this article is on my personal GitHub repository. It's a VS 2010 / .NET 4.0 solution. There are several projects in the solution:
jterry.scripting.api - The business logic and data access library jterry.scripting.api.script.app - WinForms app that only provides IronPyton script access to the API jterry.scripting.host - Class library containing Python engine wrapper classes jterry.scripting.host.editor - Class library containing reusable WinForms Script Editor -
jterry.scripting.web - ASP.NET WebForms front end for accessing the API jterry.scripting.winforms - Desktop WinForms application for accessing the API
The ASP.NET Script Editor uses codemirror to provide Python syntax highlighting in the browser. I found this on StackOverflow, and it looked interesting. I won't discuss codemirror usage in this article, but you can take a look at the code to see the basic usage.
Data Model
For this article, I've used the Chinook SQlite database, accessed using the Entity Framework via System.Data.Sqlite. I followed the steps from Brice's Blog to get this up and running.
Here's IUnitOfWork interface is used to hide the implementation details of the Entity Framework in the ChinookContext.
using System.Linq;
namespace jterry.scripting.api
{
public interface IUnitOfWork
{
IQueryable<Customer> GetCustomers();
IQueryable<Employee> GetEmployees();
IQueryable<Invoice> GetInvoices();
IQueryable<InvoiceLine> GetInvoiceLines();
IQueryable<Track> GetTracks();
IQueryable<Playlist> GetPlaylists();
}
}
ChinookContext implements the IUnitOfWork interface, and provides access to the Chinook SQLite database via the Entity Framework.
using System.ComponentModel.DataAnnotations.Schema;
using System.Data.Entity;
using System.Data.Entity.ModelConfiguration.Conventions;
using System.Linq;
namespace jterry.scripting.api
{
public class ChinookContext : DbContext, IUnitOfWork
{
public DbSet<Customer> Customers { get; set; }
public DbSet<Employee> Employees { get; set; }
public DbSet<Invoice> Invoices { get; set; }
public DbSet<InvoiceLine> InvoiceLines { get; set; }
public DbSet<Track> Tracks { get; set; }
public DbSet<Genre> Genres { get; set; }
public DbSet<MediaType> MediaTypes { get; set; }
public DbSet<Album> Albums { get; set; }
public DbSet<Artist> Artists { get; set; }
public DbSet<Playlist> Playlists { get; set; }
public DbSet<PlaylistTrack> PlaylistTracks { get; set; }
public ChinookContext()
{
Database.SetInitializer<ChinookContext>(null);
}
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Conventions.Remove<PluralizingTableNameConvention>();
var customerMap = modelBuilder.Entity<Customer>();
customerMap.ToTable("Customer");
customerMap.HasKey(c => c.Id);
customerMap.Property(c => c.Id)
.HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity)
.HasColumnName("CustomerId");
customerMap.HasOptional(c => c.SupportRep)
.WithMany().Map(x => x.MapKey("SupportRepId"));
var employeeMap = modelBuilder.Entity<Employee>();
employeeMap.ToTable("Employee");
employeeMap.HasKey(e => e.Id);
employeeMap.Property(e => e.Id)
.HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity)
.HasColumnName("EmployeeId");
employeeMap.HasOptional(c => c.ReportsTo)
.WithMany().Map(x => x.MapKey("ReportsTo"));
var invoiceMap = modelBuilder.Entity<Invoice>();
invoiceMap.ToTable("Invoice");
invoiceMap.HasKey(e => e.Id);
invoiceMap.Property(e => e.Id)
.HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity)
.HasColumnName("InvoiceId");
invoiceMap.HasOptional(i => i.Customer)
.WithMany().Map(x => x.MapKey("CustomerId"));
var invoiceLineMap = modelBuilder.Entity<InvoiceLine>();
invoiceLineMap.ToTable("InvoiceLine");
invoiceLineMap.HasKey(l => l.Id);
invoiceLineMap.Property(l => l.Id)
.HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity)
.HasColumnName("InvoiceLineId");
invoiceLineMap.HasRequired(l => l.Invoice)
.WithMany(i => i.Lines).Map(x => x.MapKey("InvoiceId"));
invoiceLineMap.HasOptional(l => l.Track)
.WithMany().Map(x => x.MapKey("TrackId"));
var trackMap = modelBuilder.Entity<Track>();
trackMap.ToTable("Track");
trackMap.HasKey(t => t.Id);
trackMap.Property(t => t.Id)
.HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity)
.HasColumnName("TrackId");
trackMap.HasOptional(t => t.Genre)
.WithMany().Map(x => x.MapKey("GenreId"));
trackMap.HasOptional(t => t.Album)
.WithMany().Map(x => x.MapKey("AlbumId"));
trackMap.HasOptional(t => t.MediaType)
.WithMany().Map(x => x.MapKey("MediaTypeId"));
var artistMap = modelBuilder.Entity<Artist>();
artistMap.ToTable("Artist");
artistMap.HasKey(a => a.Id);
artistMap.Property(a => a.Id)
.HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity)
.HasColumnName("ArtistId");
var genreMap = modelBuilder.Entity<Genre>();
genreMap.ToTable("Genre");
genreMap.HasKey(g => g.Id);
genreMap.Property(g => g.Id)
.HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity)
.HasColumnName("GenreId");
var albumMap = modelBuilder.Entity<Album>();
albumMap.ToTable("Album");
albumMap.HasKey(a => a.Id);
albumMap.Property(a => a.Id)
.HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity)
.HasColumnName("AlbumId");
albumMap.HasOptional(a => a.Artist)
.WithMany().Map(x => x.MapKey("ArtistId"));
var mediaTypeMap = modelBuilder.Entity<MediaType>();
mediaTypeMap.ToTable("Media");
mediaTypeMap.HasKey(m => m.Id);
mediaTypeMap.Property(m => m.Id)
.HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity)
.HasColumnName("MediaTypeId");
var playlistMap = modelBuilder.Entity<Playlist>();
playlistMap.ToTable("Playlist");
playlistMap.HasKey(p => p.Id);
playlistMap.Property(p => p.Id)
.HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity)
.HasColumnName("PlaylistId");
var playlistTrackMap = modelBuilder.Entity<PlaylistTrack>();
playlistTrackMap.ToTable("PlaylistTrack");
playlistTrackMap.HasKey(p => p.Id);
playlistTrackMap.Property(p => p.Id)
.HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity)
.HasColumnName("PlaylistTrackId");
playlistTrackMap.HasOptional(p => p.Playlist)
.WithMany(p => p.Tracks).Map(x => x.MapKey("PlaylistId"));
playlistTrackMap.HasOptional(p => p.Track)
.WithMany().Map(x => x.MapKey("TrackId"));
base.OnModelCreating(modelBuilder);
}
public IQueryable<Customer> GetCustomers()
{
var query = from c in Customers select c;
return query;
}
public IQueryable<Employee> GetEmployees()
{
var query = from c in Employees select c;
return query;
}
public IQueryable<Invoice> GetInvoices()
{
var query = from c in Invoices select c;
return query;
}
public IQueryable<InvoiceLine> GetInvoiceLines()
{
var query = from c in InvoiceLines select c;
return query;
}
public IQueryable<Track> GetTracks()
{
var query = from c in Tracks select c;
return query;
}
public IQueryable<Playlist> GetPlaylists()
{
var query = from c in Playlists select c;
return query;
}
}
}
Sample IronPython Script
Below is a sample Iron Python script that illustrates how to call .into NET code. You should read the IronPython .NET documentation to learn about the IronPything .NET integration.
import clr
import System
clr.AddReference("System.Core")
clr.AddReference("System.Windows.Forms")
clr.ImportExtensions(System.Linq)
from System import String
def TestIronPython():
CountEntities()
TestCustomers()
TestInvoices()
TestPlaylists()
TestSearchTracks()
def GetAllCustomers():
return unitOfWork.GetCustomers()
def GetAllEmployees():
return unitOfWork.GetEmployees()
def GetAllInvoices():
return unitOfWork.GetInvoices()
def GetAllPlaylists():
return unitOfWork.GetPlaylists()
def GetAllTracks():
return unitOfWork.GetTracks()
def CountEntities():
WriteLine(String.Format("Found {0} customers", GetAllCustomers().Count()))
WriteLine(String.Format("Found {0} employees", GetAllEmployees().Count()))
WriteLine(String.Format("Found {0} invoices", GetAllInvoices().Count()))
WriteLine()
def TestCustomers():
WriteLine("Testing Customers...")
customer = GetAllCustomers().FirstOrDefault()
rep = customer.SupportRep
reportsTo = rep.ReportsTo
WriteLine(String.Format("Customer: {0} {1}", customer.FirstName, customer.LastName))
WriteLine(String.Format("Support Rep: {0} {1}", rep.FirstName, rep.LastName))
if (reportsTo != None):
WriteLine(String.Format("Reports To: {0} {1}",
reportsTo.FirstName, reportsTo.LastName))
else:
WriteLine("Employee has no boss")
WriteLine("Testing Customers complete");
WriteLine()
def WriteLine(line = ""):
print line
def TestPlaylists():
WriteLine("Testing Playlists...")
playlist = GetAllPlaylists().FirstOrDefault()
WriteLine(playlist.Name)
WriteLine("Testing Playlists complete")
WriteLine()
def TestInvoices():
WriteLine("Testing Invoices...")
invoice = GetAllInvoices().FirstOrDefault()
WriteLine(String.Format(
"Invoice Id: {0} Date: {1}", invoice.Id, invoice.InvoiceDate))
lines = invoice.Lines
WriteLine(String.Format("Found {0} tracks", lines.Count))
for l in lines:
WriteLine(String.Format(
"Invoice Line Id: {0} Track Id: {1} Track Name: {2} Price: {3}",
l.Id, l.Track.Id, l.Track.Name, l.Track.UnitPrice))
WriteLine("Testing Invoices complete")
WriteLine()
def TestSearchTracks():
WriteLine("Testing Search Tracks...");
SearchTracks("AC/DC")
WriteLine("Testing Search Tracks Complete");
WriteLine()
def SearchTracks(composer):
WriteLine(String.Format("Searching for tracks by '{0}'...", composer))
tracks = GetAllTracks().Where(lambda t: t.Composer == composer)
WriteLine(String.Format(
"Found {0} tracks by '{1}'", tracks.Count(), composer))
for t in tracks:
WriteLine(String.Format("Track Id: {0} Name: {1}", t.Id, t.Name))
WriteLine("Search complete")
TestIronPython()
Tabbed Script Editor
After working with a simple Python Editor that was just a single form with a RichTextBox, the user experience didn't even come close to what you would expect in a full fledged code editor. To make it more developer friendly, I added support for having multiple tabs open at once, a toolbar with all the action buttons right at the top, a simple detection of changes in the code file so the * gets appended to the file name if it's been changed,
I would have liked to added support for Python syntax highlighting (something equivalent to codemirror for WinForms), but I couldn't find a quick and easy solution.
Edit - I found the editor I have been looking for. Pavel Torgashov posted an awesome article "Fast Colored Text Box" that I've replaced the RichTextBox with. It took no time at all to wire in the new control.
Since this control supports code folding, I've hooked up the editor to use #< and #> as the code folding markers.
Web Script Editor
The web script editor still only supports editing a single script at a time.
Conclusion
We know that software will change, but typically we don't know when or why or how. We keep our designs flexible, but user interfaces are usually not as flexible as the code. Providing scripting extensions to your applications might help in providing a little bit of extra flexibility.
History
- June 3, 2013 - Initial release
- June 4, 2013 - Added brief description of IP / .NET integration, minor code fixes and article cleanup
- June 10, 2013 - Added discussion of filtering data
- June 11, 2013 - Added Chinook SQLite database
- June 14, 2013 - Added tabbed script editor
- June 15, 2013 - Editorial changes
- June 19, 2013 - Added Fast Colored Text Box
- June 20, 2013 - Added IronPython primer
- June 24, 2013 - Added brief IronPython background
