Introduction
Google Cloud Platform opens up Google's massive fast and vast networks and supporting infrastructure to 3rd party developers, Google App Engine is one of the primary pillars that enpowers developers with great productivity and efficiency in the cloud. This article discuss the background thinking of selecting a cloud platform in a start-up project case, also provides the plumbing and sample code to bootstrap your own GAE project.
This article can be viewed as a seed starter full stack project on Google App Engine, and it's also an implementation of Service Oriented HTML Application (SOHA) on GAE.
Background
When starting a new n-tier cloud based project about 6 months ago, I was facing variety options for development, hosting and operations that offered by different cloud platforms. Initial selection scopes were quickly narrowing down to three options: Amazon Web Services, Windows Azure and Google Cloud Platform. For cost effective, productivity sensitive and quick iterative projects, which one is most suitable?
Without making pros and cons comparisons and analysis among these popular cloud computing platforms, to cut to the chase directly, I'd like to talk a bit about the greatest benefits of selecting Google App Engine after experimenting and evaluating on all three platforms: developer productivity and operation efficiency.
GAE provides great developer productivity through fully managed Platform-as-a-Service (PaaS) using profuse built-in services that runs on the same infrastructure that Google uses to run their own offerings. Developer can focus more on their own code and project, off-load the works of virtual machine management, database admin, system patches, library updates, server configuration, sharding and load balancing to Google. All developer needs to do is to use GAE SDK developing and deploying project to GAE, and Google will take care of the rest. When your application becomes a hit, more concurrent users generate more loads on servers, GAE will handle auto-scaling at runtime automatically and dynamically.
Additionally, GAE makes developers more productive by eliminating boilerplate code, also enables easy integrating with plenty managed services such as Users, Maps, Geocoding, Gmail, Docs API together with task queues, memcache and NoSQL data stores. Up and running on GAE is super easy, just download Google App Engine SDK and start your favorite IDE.
In addition to great developer productivity, Google Cloud Platform console provides abundant easy to use tools and dashboards for application operation tasks. Cloud datastore dashboard shows all entity statistics whenever you log in, and the console also allows you to view, query, update and create entities right in the browser with zero configuration. Comparing the classic database admin and schema configuration model, GAE provides excellent operation efficiency improvements by eliminating configurations and admin set up for database (cloud datastore).
The operation efficiency doesn’t stop with zero configuration. Server side logs monitoring is also available in browser window, even though monitoring dashboards and alerts are not released to the public yet, App Engine dashboard offers sophisticated tracking statistics for free, including request count/sec chart, VM instance (QPS, latency, memory usage, start time, etc.), concurrent loads, server errors, memcache keys and status, task queues and quota details.
GAE’s support for multi-tenancy is powerful and flexible, it enables application compartmentalization with ease, and with traffic splitting, application A/B testing for different live versions are very well supported in the platform level.
Google App Engine has even more benefits over other cloud platform (like cost effective, etc.), I’ll stay within the scope of technical for now. Google managed environments are fairly developer friendly to grasp and its built-in operational dashboards and statistics/tracking/monitoring/db querying come as a bonus.
GAE’s productivity and efficiency are the #1 reason for me to select it over others for full stack development, and it’s also the compelling reason for me to write this article. In terms of software engineering, what Google makes you to give up is the freedom to select your programming language and operational tools. Currently, GAE runtime or SDK only supports four languages, Java, Python, Go and PHP. If Node.js or Ruby or other languages are preferred, you may have to go with Compute Engine, or AWS EC2, or Windows Azure, but then you also have to start manage VM, OS and configure scalability. To me, it’s a trade-off I can ignore, I started off with knowing nothing about Python and became comfortable with it in about few days.
This article will provide code snippet and sample project in Python 2.7. The sample project is a full stack web application, what we show case here is how productive and efficient you are to build modern web application and services with separation of concerns and great scalability and manageability.
What we’re building
We’re going to build a full stack modern web application in SOHA architecture on Google App Engine. On a high level, our web application consists of 3 major tenets as in SOHA:
- Presentation agnostic web services or APIs, it implements business logics and data access layer to expose functionalities through RESTful endpoints. It usually runs on app server with database access, in our GAE implementation, our APIs are part of the project and deployed onto the same GAE while still keeping agnostic about any presentation or user interaction logics;
- Front-end static files are separated from API and served through HTTP. Usually this layer is served by a web server, like Apache or Nginx, since there is no concept of “web server” on Google App Engine, we’re going to implement it in YAML, no web server configuration either;
- HTML5 user interface that interacts with backend APIs through Ajax. Your favorite JavaScript framework plays well here, either Angular, Backbone, Ember or any other, either Single Page App or Multiple Page App with secure navigation all work.
The sample project does two main things, saving and fetching data. It implements one endpoint that takes and validates a HTTP POST request, then save the data to Google Cloud Datastore - Google's version of NoSQL cloud data storage. It also provides a generic data retrieval mechanism, shows the support of ndb.Model JSON serialization, response packaging, sorting, paging and query for property with great flexibilities, it not only query property with actual value, also supports 'OR' operation via dictionary literal in the query string, and Range operation through tuple literal in the query string as well, more details can be found at GitHub.
Get Started
Quick and easy: get a project id from Google, activate Google Cloud Datastore, start up Google App Engine App Launcher, and you are ready to go.
What it’s built with
HTML5 user interface will be built with following frameworks, libraries and tools:
As for the API layer, there are many frameworks to built RESTful web service/APIs in Python on Google App Engine. I've evaluated Django, Django REST framework, Google Cloud Endpoints, since all I need is a lightweight, easy to use and plug-able RESTful-only framework, I eventually landed on the following stack:
Flask-RESTful
Flask-RESTful provides a lightweight abstraction for building RESTful service; it's an extension to Flask and has all the bells and whistles to quickly get a RESTful service up and running on Google App Engine.
All dependencies of Flask-RESTful, and dependencies of dependencies are packaged up in /lib/ directory, including:
- aniso8601
- flask
- pytz
- werkzeug
- itsdangerous
RESTful API building blocks
Although Flask-RESTful has provided an excellent starting point for all sorts of RESTful service building blocks, like useful decorators, Resource base class, easy API routes, request parsing and validation, error handing, etc., there always are opportunities to add more utilities and helper functions to make it more suitable to your specific project. To be as much as generic as possible, I also include the following building blocks in the api/common directory:
- response.py: two base classes to structure all response in a unified form
- session.py: utilize Google App Engine's memcache to validate a session
- util.py: lots of utilities methods mainly around data types and ndb model processes
- validators: validator types that works with ReqParse
CORS Support
The built-in implementation of CORS comes with Flask Restful has issues to automatically handle HTTP OPTIONS calls, it returns 404 with the correct setups for methods decorators. I made two additions to CORS support in Flask Restful (look for "MQZ" pre-fixed comments in lib/flask_restful/utils/cors.py):
- Default HTTP OPTIONS handler:
# MQZ. 08/12/2014: without this, will get 405 responses for OPTIONS request
f.required_methods = ['OPTIONS']
- Default on Cookie support:
# MQZ. 8/26/2014: Make sure cors support for cookies (with_credentials), default on
def crossdomain(origin=None,
methods=None,
headers=None,
max_age=21600,
attach_to_all=True,
automatic_options=True,
with_credentials=True):
Then in line 43:
if with_credentials:
h['Access-Control-Allow-Credentials'] = True
Leave me a message in GitHub if you see issue on CORS.
Source Code Structure
Since Flask-RESTful is not included in Google App Engine by default, we need to structure our project to make it 'Cloud Ready'. All the necessary libraries, source trees and scripts/yaml are included:
lib: this directory includes all the 3rd party Python libraries that Flask-RESTful requires, no special installation commandline to run, just clone or copy the entire directory, it's ready to goapp.yaml: Google App Engine requires app.yaml to run your project, since this project is a backend example of SOHA architecture, this file is essentially empty: it only includes two other yaml files to ensure the front-end is separated from back-end, these two other yaml files are:
web_app.yaml: this is a fork of yaml that enables hosting a static site on Google App Engine. The only update I made to this yaml file is to tell GAE looking for ‘index.html’ underneath /app/ folder, all other frontend static files, either Angular or Backbone based project, should put in /app/ folder. Otherwise, you can modify the rules list in the yaml file to match your customized folder structure;web_api.yaml: this yaml tells Google App Engine which Python class to load to run RESTful service if the request path matches /api/*
appengine_config.py: this Python file is to tell Python runtime to use Python modules underneath /lib/ directorydata.datastore: optional data file for local testing, it requires --datastore_path=<Your Clone Path>/data.datastore commandline arguments when run the project on localhost.api/config.py: some API level configuration constants and variables are defined here, including:
- session time span
- datastore version string
- api base url for version management
- application version, etc.
Flexible Data Retrieval
As discussed before, this project provides a generic data retrieval mechanism (defined in modelX.retrieve_list as generic static method), it supports ndb.Model JSON serialization, response packaging, sorting, paging and query for property with great flexibilities. It can not only query property with actual value, but also supports 'OR' operation via dictionary literal in the query string, and 'Range' operation through tuple literal in the query string as well, here are some query examples via HTTP GET:
default query (with default limit set to 20: it returns up to 20 entities):
curl -isv http:
order by name (ASC):
curl -isv http:
order by name (DESC):
curl -isv http:
sort and limit:
curl -isv "http://localhost:8080/api/v1.0/calllogs/?order=-name&limit=2" notice "more_cursor" and "more_url" in the response
paging (with cursor value returned in earlier calls)
curl -isv "http://localhost:8080/api/v1.0/calllogs/?cursor=E-ABAOsB8gEEbmFtZfoBChoISGVrdGhvbjXsAYICNWodZGV2fmNhbGwtbG9nLTIxMS1kYXRhLXNlcnZpY2VyFAsSB0NhbGxMb2cYgICAgID0jgoMiAIBFA%3D%3D&limit=2&order=-name"
list all (default limit) entities whose name is "Hekthon5":
curl -isv http:
OR: list all (default limit) entities whose name is either 'Hekthon5' or 'Hekthon4':
curl -isv -G --data-urlencode "name=['Hekthon5', 'Hekthon4']" http:
Range: list all (default limit) entities which is created between '2014-08-01 00:00:00' and '2014-08-04 23:59:59' (inclusive)
curl -isv -G --data-urlencode "created=('2014-08-01 00:00:00', '2014-08-04 23:59:59')" http:
Run and deploy the project
Google has excellent documentation on how to run on local development server, and also how to deploy and manage your app on App Engine. You can also reference Python tutorial for more details.
Wrap it up
Google App Engine provides cost effective and developer friendly way to build modern web apps, apis and mobile back ends. Its fully managed runtime environment does the heavy lifting for auto-scaling, dynamic computing resourcing, critical monitoring and tracking, together with other build-in tools and dashboards (database CRUD console in browser, etc.), has moved the needle for great productivity and efficiency. The sample project discussed in this article further extends what Google offers out-of-the-box, the code structure, yaml and appengine_config.py are cloud production ready, hope it helps to make your full stack development on GAE even more flexible and powerful.
