Building an ASP.NET Core web app for a Linux VPS: Part 2

Introduction

In Part 1 of this series we got .NET up and running on a Debian Virtual Private Server (VPS). Time to build a simple ASP.NET Core 1.0 web app that will greet unnamed sailors surfing by. But first things first: we need to harden our Linux box, which is still running out in the wild with a default setup. Then we're ready to set up NGINX and build that fancy app in C#.

Firewall

Remember how excited I sounded about VPSes because they let me build on a full-fledged system for peanuts? If you go for a standard, dumbed-down IIS hosting plan, you don't get any of that, but in exchange a bunch experts will be taking care of the entire system's safety, and they'll be doing a much better job at it than you or I ever could. The price of your freedom with a VPS is that it's your responsibility to ensure a minimum level of security.

There are apparently two ways to go about setting up a firewall in Linux. The first is for those who truly, really know what they are doing, and/or are willing to spend days learning. You can learn about that way if you Google for iptables. My way is the Uncomplicated Firewall, or ufw for short. Here's how you set it up from the command prompt.

gabor@debian-512mb-fra1-01:~$ sudo apt-get install ufw
#
# Many lines of output omitted
#
gabor@debian-512mb-fra1-01:~$ sudo ufw status
Status: inactive
gabor@debian-512mb-fra1-01:~$ sudo ufw default deny incoming
Default incoming policy changed to 'deny'
(be sure to update your rules accordingly)
gabor@debian-512mb-fra1-01:~$ sudo ufw default allow outgoing
Default outgoing policy changed to 'allow'
(be sure to update your rules accordingly)
gabor@debian-512mb-fra1-01:~$ sudo ufw allow ssh
Rules updated
Rules updated (v6)
gabor@debian-512mb-fra1-01:~$ sudo ufw allow http
Rules updated
Rules updated (v6)
gabor@debian-512mb-fra1-01:~$ sudo ufw allow https
Rules updated
Rules updated (v6)
gabor@debian-512mb-fra1-01:~$ sudo ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup

(If you didn't read the first part: I'm highlighting your own commands in bold for better readability.)

Let's unpack that:

• apt-get installs ufw
• With ufw status you check that the firewall is inactive. (It should be, right after install.) This one matters quite a bit. If you start disabling all connections, you might end up losing the very SSH session you're using, and then you have a fully autonomous server somewhere that no one can ever access again remotely.
• The next two commands are self-explanatory: they disable all incoming connections, and allow outgoing ones. There's a school that would be much more paranoid about outgoing connections too, but then you have to work extra to download updates or modules, not to mention to call services from your own application.
• Having established that our baseline is "no incoming connections," the following commands re-enable specific things, such as the ports for SSH, HTTP and HTTPS. The most crucial one is SSH, or you won't be able to get back in again.
• Finally, ufw enable turns on the firewall with the rules we just set up.

Point a domain name at your server

If you already own a domain, I don't need to waste your time here. If you don't, then you might do what I did: get a free one in about five minutes. There are several top-level domains (TLDs) that let you do this; I went for a .tk domain. I'll be using vps-net-core.tk in the rest of these articles. In your own domain's management interface, point an empty "A" record at your VPS's IP address; optionally, also point a second record for the www subdomain. In my case, the setup looks like this:

Officially, that could take up to a day to propagate through the internets, but in practice I've never had to wait more than a few minutes. You can check the status by pinging your server from the Windows command prompt:

C:\>ping vps-net-core.tk

Pinging vps-net-core.tk [46.101.137.211] with 32 bytes of data:
Reply from 46.101.137.211: bytes=32 time=41ms TTL=52
Reply from 46.101.137.211: bytes=32 time=27ms TTL=52
Reply from 46.101.137.211: bytes=32 time=62ms TTL=52
Reply from 46.101.137.211: bytes=32 time=26ms TTL=52

Ping statistics for 46.101.137.211:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 26ms, Maximum = 62ms, Average = 39ms

Pro tip: If you really, really don't want to blow a throwaway email address for a free domain, you can mess with the hosts file that resides in C:\Windows\System32\drivers\etc. The comments in that file are self-explanatory; the only trick is allowing Windows and your local firewall to modify it. Hint: copy onto your Desktop; edit there; then copy-paste back and click through Mother Doze's threatening confirmation dialogs. You do need to be an Admin for this, of course.

Optional: Midnight Commander

This is about the point where I typically lose patience with a barefoot command line. If you're familiar with Norton Command or FAR Manager in the Windows universe, then Midnight Commander will be your friend. By now you may well guess the exact incantation below; follow it to see what I see (image below) in your PuTTY session.

gabor@debian-512mb-fra1-01:~$ sudo apt-get install mc
gabor@debian-512mb-fra1-01:~$ mc

Serving pages: NGINX

Time to start serving pages! We will set up NGINX, but before we begin, a short language lesson is in order. I spent months of my life pronouncing it "enjinx" and trembling at the thought of the inevitable bad luck that was about to follow. Then I found out that it's really "Engine X." I wish I could recondition myself now.

Now, NGINX is kinda like Apache, only it's a lot easier to tame, and also much less hungry. Remember, we're after a tiny server that still blazes as fast as a SpaceX rocket, right?

But why do we need a web server at all? Ain't ASP.NET Core's web component, Kestrel, all about serving HTTP requests? Well, partly. It does exactly that, for sure, but it also doesn't do much else. Multiple sites (via multiple domain names) served by the same engine? Nope. Secure connections over HTTPS? Negative. As far as I'm concerned, that's also the right thing to do: let every component deal with a single responsibility, but deal with it well. Nuff said. Let's magick up NGINX on our box.

gabor@debian-512mb-fra1-01:~$ sudo apt-get install nginx

Now go to your favorite browser and admire NGINX's default welcome page:

Here's a super-cool and super-concise overview of NGINX for you, by @carrotcreative: http://carrot.is/coding/nginx_introduction

Happy reading.

A truly simple ASP.NET Core wep app

Download the first code sample, WCA.zip, and fire it up in Visual Studio. It's a slightly reduced version of what you get if you create a "blank" ASP.NET Web Application, except my definition of blank is apparently different from Microsoft's. Also, I changed a few crucial lines in Startup.cs and Program.cs to make it serve static HTML files from the wwwroot subfolder, but I'll get into the actual web development part later on in the series.

OK, so it works on Windows. Now just unzip the sample to any local folder and upload it into your own Debian home directory - I'm using work/WCA for this. You can do this through WinSCP. If you don't know what that is and you skipped the first part, now is a good time to read up.

Go to the command prompt in your home directory, and fire it right up like so:

gabor@debian-512mb-fra1-01:~$ cd work/WCA/src/WCA/
gabor@debian-512mb-fra1-01:~/work/WCA/src/WCA$ dotnet restore
log  : Restoring packages for /home/gabor/work/WCA/src/WCA/project.json...
log  : Writing lock file to disk. Path: /home/gabor/work/WCA/src/WCA/project.lock.json
log  : /home/gabor/work/WCA/src/WCA/project.json
log  : Restore completed in 2501ms.
gabor@debian-512mb-fra1-01:~/work/WCA/src/WCA$ dotnet run
Project WCA (.NETCoreApp,Version=v1.0) will be compiled because expected outputs are missing
Compiling WCA for .NETCoreApp,Version=v1.0

Compilation succeeded.
    0 Warning(s)
    0 Error(s)

Time elapsed 00:00:02.1747842

Hosting environment: Production
Content root path: /home/gabor/work/WCA/src/WCA
Now listening on: http://0.0.0.0:5000
Application started. Press Ctrl+C to shut down.

Ha! You've got a web app listening on port 5000 right there. But... how do you get to see the pages it serves? That's a bit difficult, because you just told ufw not to let you in through that port. If you're in the mood to play around, you can do one of two things:

1. Tweak ufw to let you in after all.
2. Install a text-based web browser (!) like Lynx through a different session, and check out localhost:5000.

Alternatively, you can move straight on to the next section.

Reverse proxy

The idea is to have Kestrel, the .NET Core "web server," running confined to localhost, answering calls only locally through port 5000. But we also configure NGINX to act as a reverse proxy, which really is just a fancy way of saying that instead of serving incoming requests directly, it should forward them straight on to localhost:5000 to be answered by your app.

The configuration files of NGINX reside in /etc/nginx, and in particular, two subfolders: /nginx/etc/sites-available and /nginx/etc/sites-enabled. The basic idea is that you keep a file in sites-enabled for every domain that your machine serves. The extra twist is that you actually keep those files in sites-available and only point a symlink at them from sites-enabled. This way you can juggle multiple different config files for the same domain: e.g., one that defines the website when it's working properly, and another one that serves a "maintenance in progress" page. You can swap between the two by pointing the symlink to one or the other.

In my case, I've created

<span style="font-size:11.0pt;line-height:107%;
font-family:"Calibri",sans-serif;mso-ascii-theme-font:minor-latin;mso-fareast-font-family:
SimSun;mso-fareast-theme-font:minor-fareast;mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";mso-bidi-theme-font:minor-bidi;
mso-ansi-language:EN-US;mso-fareast-language:ZH-CN;mso-bidi-language:AR-SA">/etc/nginx/sites-available/vps-net-core.tk</span>

with the content below. (Tip: I like to start Midnight Commander with root privileges through sudo mc, and then use its built-in editor to modify simple files like this.)

server {
  listen 80;
  server_name vps-net-core.tk;

  location / {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://127.0.0.1:5000;
  }
}

Now create a symlink to this file, remove NGINX's default config file, test if everything's OK, and restart NGINX:

gabor@debian-512mb-fra1-01:~$ ln -s /etc/nginx/sites-available/vps-net-core.tk /etc/nginx/sites-enabled/vps-net-core.tk
gabor@debian-512mb-fra1-01:~$ rm /etc/nginx/sites-available/default
gabor@debian-512mb-fra1-01:~$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
gabor@debian-512mb-fra1-01:~$ sudo service nginx restart

The third command, nginx -t, is a neat little trick. It parses the current configuration and gives you a heads-up if something's wrong. NGINX ignores whatever is in the config files while it's running and only reads them when you restart. If you mess them up, you'll end up with NGINX down until you manage to fix them.

And we're basically... there! If you still have the .NET app running in a different terminal session, you can navigate to your website (your equivalent of vps-net-core.tk) and see "Hello, sailor." If your app is not running, you'll see this:

There's more good reading about the nitty-gritty of hooking up Kestrel with NGINX at these links:
https://mjomaa.com/computer-science/frameworks/asp-net-mvc/141-how-to-combine-nginx-kestrel-for-production-part-i-installation
http://druss.co/2015/06/asp-net-5-kestrel-nginx-web-server-on-linux/

Publish and deploy as a service

We have just cobbled together a system that serves pages as long as you run a .NET Core app from the command line, compiled on the spot - but that's a far cry from a production system. For one, you don't want to be uploading source code but want to compile locally and deploy binaries; second, you want something service-like to be running on the Linux box, not a command line program.

Publish from Visual Studio

In your Visual Studio project, go to Build / Publish and walk through the wizard to create a new "profile." This is essentially a handful of settings telling VS how to compile your code and what to do with the output. Let's just go for the simplest mechanics: put the output into the local file system on your PC, and copy manually (using WinSCP) for deployment. In the wizard's first step, select "Custom" for the publishing target:

In the second step, after you've named your profile, stay with "File System" and choose a suitable folder. I like to keep this folder outside my project's folder hierarchy; it is, after all, not part of my code, just a kind of springboard to deploy from. I hate those braindead \bin\Release\PublishOutput defaults.

The rest is obvious, and you can use this "profile" again every time you are publishing a version of your web app. I have uploaded a sample compiled output as WCA-published.zip. Check what's inside; it contains your project's output, plus all the dependencies it needs to run.

The way I deploy it is to:

1. Upload the raw (unzipped) files into a subdirectory of my Linux user's home, using WinSCP
2. In the Linux shell, copy this directory's contents to /opt/WCA-app/app. You first need to create the directories in /opt, and you need root privileges (sudo) to be allowed to do both that and the copying.

Now change to the target directory and run your dotnet app:

gabor@debian-512mb-fra1-01:/opt/WCA-app/app$ dotnet WCA.dll

Notice that we didn't use dotnet run like we did before: that compiles and executes code from a project directory. This one's different: we're executing a compiled binary. Come to think about it, this is cool. Like, way out there cool. We just copied a whole bunch of binary DLLs from a Windows box and executed them "natively" on Linux! Pinch me.

Set up as a service

There are three things to think about when you lay out the production environment:

1. What's the right directory structure to use?
2. What user will run the service, and with what privileges?
3. How do you wrap up your app to act as a service, starting up on system startup and responding to standard commands?

The directory structure. I wish I had found any best practices for this, but there doesn't seem to be a whole lot of folks running .NET Core web apps on Linux just yet. So it's down to common sense. Read all of this with a critical eye, and let me know in the comments if you have any quibbles or better suggestions.

• I like to keep a dedicated directory for the site's compiled output, hence the /app subfolder. The point is, the app should not be able to write anything here, and I should be able to copy the full published package blindly into this space.
• Where should the whole app, with data, config files and all, go in the first place? I opted for a subdirectory under /opt, assuming that's in line with Linux etiquette for placing your own stuff. After all, dotnet installs there too, and so does Let's Encrypt (come back to read more about that in the next part.)
• Besides the executables and static content, my web apps typically need at least two other kinds of spaces. One is for system-specific config files, such as the database connection string. You want to have an environment set up on your development box, and a production environment in your VPS. You don't want to be cross-pollinating the config files of the two, so they need to stay out of the way when you deploy compiled output. This is why I have the /app subfolder, with space to spare for a neighboring /config folder later.
• Finally, your app may well need its own safe stomping ground where it can write and delete stuff too: e.g., temporary files for uploads, logs etc. This can all go to one or more additional, neighboring subdirectories where you grant the executing user write privileges. But these folders should never, ever be available through a direct URL from your site's public view.

Executing user. We'll see in a bit how to execute your app as a "service" on Linux. It seems that unless you do extra work, such a service will run as root by default, which I'm not at all comfortable with. Below is the solution I came up with that works with Linux's standard privileges scheme. There may be a more elaborate way using ACLs, but I lost my enthusiasm seeing the sheer amount of incantations needed, and how even those will vary from one system to the other, depending on the file system you are using.

gabor@debian-512mb-fra1-01:~$ sudo groupadd wca-users
gabor@debian-512mb-fra1-01:~$ sudo adduser --system --no-create-home wca-app
Adding system user `wca-app' (UID 108) ...
Adding new user `wca-app' (UID 108) with group `nogroup' ...
Not creating home directory `/home/wca-app'.
gabor@debian-512mb-fra1-01:~$ sudo usermod -a -G wca-users wca-app
gabor@debian-512mb-fra1-01:~$ sudo chown -R wca-app:wca-users /opt/WCA-app
gabor@debian-512mb-fra1-01:~$ sudo chmod -R 550 /opt/WCA-app
gabor@debian-512mb-fra1-01:~$ sudo chmod -R g+s /opt/WCA-app
gabor@debian-512mb-fra1-01:~$ sudo -u wca-app dotnet /opt/WCA-app/app/WCA.dll

In English:

• groupadd wca-users creates a group called, yes, "wca-users".
• adduser --system --no-create-home wca-app creates a system user (cannot log in interactively) that has no home directory, and is called "wca-app".
• usermod -a -G wca-users wca-app adds the new user to the "wca-users" group.
• chown -R wca-app:wca-users /opt/WCA-app changes the ownership of our application folder to the new user and the new group.
• chmod -R 550 /opt/WCA-app grants read and execute permissions to our app directory, recursively, for its owning user and group, but grants no permission for anyone else.
• chmod -R g+s /opt/WCA-app  is the crucial bit. To quote Wikipedia on what it does: it "causes new files and subdirectories created within it to inherit its group ID, rather than the primary group ID of the user who created the file (the owner ID is never affected, only the group ID)." In other words, if other users create files in my app's directory, those will inherit the parent directory's group ID. In Windows terms, inherited permissions. That's apparently not possible for users in Linux, only groups: that's why we needed the group in the first place. And the reason I need all of this, of course, is that I'll be wanting to copy deployed files around as root in the future, but I want my permissions architecture to hold.
• sudo -u wca-app dotnet /opt/WCA-app/app/WCA.dll, finally, executes the web app in the name of the wca-app user, just to verify that it all worked out.

Shell script. As the last ingredient, we need a shell script that's able to start the web app as a bacground process running in the name of wca-app, stop it, and restart it, on demand. We can then strategically place the script in a magic folder, /etc/init.d, and tell Linux to start our service at system startup. For this, I relied heavily on druss's script, but had to modify it for two reasons. One, his script is for a pre-release version of .NET Core, and the command you need has changed to dotnet since then. Second, his script did not work with my permissions scheme. Here's the script I ended up with:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          wca-srv
# Required-Start:    $local_fs $network $named $time $syslog
# Required-Stop:     $local_fs $network $named $time $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Description:       Script to run asp.net 5 application in the background
### END INIT INFO
 
# Author: Ivan Derevianko aka druss <drussilla7@gmail.com>
# Modified by: Gabor L Ugray
 
SRVUSER=wca-app
MODULEROOT=/opt/WCA-app
APPROOT=$MODULEROOT/app
APPDLL=WCA.dll
PIDFILE=$MODULEROOT/service/service.pid
LOGFILE=$MODULEROOT/service/service.log
 
# fix issue with DNX exception in case of two env vars with the same name but different case
TMP_SAVE_runlevel_VAR=$runlevel
unset runlevel
 
start() {
  if [ -f $PIDFILE ] && kill -0 $(cat $PIDFILE); then
    echo 'Service already running' >&2
    return 1
  fi
  echo 'Starting service...' >&2
  #su -c "start-stop-daemon -SbmCv -x /usr/bin/nohup -p \"$PIDFILE\" -d \"$APPROOT\" -- \"$DNXRUNTIME\" kestrel > \"$LOGFILE\"" $WWW_USER
  su -c "start-stop-daemon -SbmCv -x /usr/bin/nohup -p \"$PIDFILE\" -c \"$SRVUSER\" -d \"$APPROOT\" -- /usr/bin/nohup dotnet \"$APPROOT/$APPDLL\" > \"$LOGFILE\" 2>&1"
  echo 'Service started' >&2
}
 
stop() {
  if [ ! -f "$PIDFILE" ] || ! kill -0 $(cat "$PIDFILE"); then
    echo 'Service not running' >&2
    return 1
  fi
  echo 'Stopping service...' >&2
  start-stop-daemon -K -p "$PIDFILE"
  rm -f "$PIDFILE"
  echo 'Service stopped' >&2
}
 
case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart)
    stop
    start
    ;;
  *)
    echo "Usage: $0 {start|stop|restart}"
esac
 
export runlevel=$TMP_SAVE_runlevel_VAR

The crucial line is the one responsible for starting the process running dotnet running your app. Below is druss's original, as well as my modified version.

#su -c "start-stop-daemon -SbmCv -x /usr/bin/nohup -p \"$PIDFILE\" -d \"$APPROOT\" -- \"$DNXRUNTIME\" kestrel > \"$LOGFILE\"" $WWW_USER
su -c "start-stop-daemon -SbmCv -x /usr/bin/nohup -p \"$PIDFILE\" -c \"$SRVUSER\" -d \"$APPROOT\" -- /usr/bin/nohup dotnet \"$APPROOT/$APPDLL\" > \"$LOGFILE\" 2>&1"

The key difference is that his version already invokes start-stop-daemon in the name of the service user, while mine invokes start-stop-daemon still as root, and tells it to execute dotnet as a background process in the service user's name.

Another thing this script (or, specifically, start-stop-daemon) does is create a "pidfile" or process ID file when the service starts, containing, well, the running process's ID. It's used as an indicator to know if the service is running, and to kill the right process when stopping it. I chose to keep this file outside my app's deployment directory proper, putting it inside /opt/WCA-app/service.

To deploy the script:

• Upload to your home folder. I called it wca-srv.sh.
• Test it: ./wca-srv.sh start
• Copy to init.d (you'll need sudo; omit the .sh)

Then:

gabor@debian-512mb-fra1-01:~$ sudo chmod 755 /etc/init.d/wca-srv
gabor@debian-512mb-fra1-01:~$ sudo /etc/init.d/wca-srv start
Starting service...
Service started
gabor@debian-512mb-fra1-01:~$ sudo update-rc.d wca-srv defaults
gabor@debian-512mb-fra1-01:~$ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.9  28580  4676 ?        Ss   Jul09   0:19 /sbin/init
root         2  0.0  0.0      0     0 ?        S    Jul09   0:00 [kthreadd]
#
# Many lines omitted
#
wca-app    601  0.0 12.0 7077296 60776 ?       SLl  Jul09   0:37 dotnet /opt/WCA-app/app/WCA.dll
#
# More lines omitted
#

Unwrapped:

• chmod 755 /etc/init.d/wca-srv makes it executable
• /etc/init.d/wca-srv start starts the service
• update-rc.d wca-srv defaults sets up the service to be started when the system boots up. (To disable that, you can use update-rc.d -f wca-srv remove later.)
• ps aux lists all the processes running on your system, and the user associated with them. I use this to verify that it's really running as wca-app as intended.

Load up your site in a browser to verify the .NET Core app is running. Restart the system to verify it starts up on boot.

A few useful links about setting up a .NET Core app as a service, and about configuring permissions:

http://druss.co/2015/06/run-kestrel-in-the-background/
http://xmodulo.com/how-to-automatically-start-program-on-boot-in-debian.html
http://www.yolinux.com/TUTORIALS/LinuxTutorialManagingGroups.html

Conclusion

This was a bit of a marathon! But now you have a simple .NET Core app up and running as a service in Linux, and you have the fundamentals of a build-and-deploy process in place that will allow you focus on actual development. Except... we still don't have HTTPS. Check back for Part 3 to see how you can set up a free certificate from Let's Encrypt.

History

07/15/2016 - Initial version