This article discusses the importance of creating isolated environments in Python to ensure smooth execution of programs, covering topics such as multiple Python interpreters, package installers, search paths, virtual environments, installing packages, using Jupyter, integrating with Visual Studio Code, Flask and Gunicorn, load balancing with Nginx, and Python classes and object orientation.
Background
Python is claimed to be an easy to learn and easy to use language, but this note is not about the Python language. It is about creating an isolated environment where we can run Python programs with certainty.
In this note, I attached a few simple Python files. I will use them to show you the benefit to have an isolated environment to run the Python programs. By the way, if you use Ubuntu flavored Linux, this link https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa is a great link for you.
How Many Pythons?
To run a Python program, we typically need three components on the computer.
- A Python interpreter
- An instance of Python Package Installer (pip) to install the dependency packages
- A search path that the interpreter looks for the dependency packages to run our programs
Multiple Interpreters
One of the Python claimed advantages is that it is readily available on all the computers. This also means that we may have a version of Python interpreter without our knowledge. In my Linux Mint 17.2 Cinnamon 64-bit, I have at least 3 Python interpreters.
When we install an operating system, it comes with certain pre-installed Python interpreters. These Python interpreters are important for the operating system to function. We should not remove them without knowing exactly what we are doing.
Multiple Package Installers
To make things more complicated, we actually have more than one instance of pip on the computer.
Multiple Search Paths
When we run a Python program, we need to make sure that the correct instance of the interpreter is used. Let us take a look at the python-search-path.py file.
import sys
for item in sys.path:
print(item)
If we run the program by the following command, we can find the package search path used by the python
interpreter.
python python-search-path.py
If we run it by the python3.4
interpreter, we will see a totally different path.
python3.4 python-search-path.py
Now we know that the Python versions on a computer are complicated. What makes it more unstable is that we may be using the same version of Python that is also used by the operating system. If we update a certain package that is also used by the operating system, we may cause a problem to the operating system. In the worst case, it may not even start. I have personally encountered problems with the OS, after playing with some versions of Python.
The Virtual Environment
We can work diligently to make sure that we are using the desired Python interpreter. We can also work diligently to install the packages into the correct search path to make a Python program run smoothly. But we can also create a virtual environment to easily run a Python program with certainty.
- A virtual environment makes sure that we use the desired Python interpreter.
- A virtual environment makes sure that we use the desired Python Package Installer (pip).
- A virtual environment makes sure that we install the packages in the environment and look for them in the environment when running a program.
Physically, a virtual environment is a folder. This folder contains almost everything to run a Python program. When we install a package, it is also installed in this folder.
Create a Virtual Environment
To create a virtual environment, we need to install the virtualenv tool. In my computer, I use the following command to install the virtualenv
.
sudo python3 -m pip install virtualenv
We can then validate the successful installation.
It looks like that the virtualenv
is independent from any specific version of Python. With the virtualenv
installed, we can then create a virtual environment by the following command:
virtualenv -p /usr/bin/python3.4 environment-3.4
- When creating the virtual environment, we specified the execution path to a Python interpreter, it will be the Python interpreter when we use the virtual environment.
- When creating the virtual environment, we specified a folder name. It is the physical folder to hold the virtual environment.
Activate a Virtual Environment
To activate a virtual environment, we can issue the following command:
source environment-3.4/bin/activate
When a virtual environment is activated, we can then check the execution path to Python and pip.
python python-search-path.py
With the virtual environment activated, we can see that all the python, pip, and the search path are located inside the virtual environment folder. We can deactivate the virtual environment by the following command:
deactivate
If we want to completely remove the virtual environment, we can simply delete the environment-3.4 folder.
Install Packages in a Virtual Environment
For experimental purposes, I created a small flask application in the python-flask-api.py file.
from flask import Flask, request, jsonify
app = Flask(__name__)
@app.route('/')
def message():
name = 'Song Li'
return jsonify(
username = name,
email = 'song.li@email.com',
)
if __name__ == '__main__':
app.run(host='0.0.0.0', port=3000)
This application requires the flask
package. With the virtual environment activated, we can use the following command to install the package:
python -m pip install flask
With flask
installed, we can then run the python-flask-api.py file.
python python-flask-api.py
We can further verify that the flask
package is installed in the virtual environment by running the python-inspect-path.py file.
import inspect
from flask import Flask
print(inspect.getfile(Flask))
Another way to create a virtualenv is to use venv
. This is a good document on venv. In Ubuntu, we need to manually install the python3.9-venv
(or the corresponding version of venv).
apt-cache policy python3.9-venv
We can then create the virtualenv with a simple command.
python3.9 -m venv "virtualenv-directory-name"
We can reference the https://docs.python.org/3/library/venv.html for more advanced usages of venv.
The PIP Freeze
After working on a virtual environment for some time, you may have installed many packages. If you want to let another person run your program, you need to tell them these packages and the versions. With the virtual environment activated, you can issue the following command:
python -m pip freeze > requirements.txt
It creates a file named requirements.txt that has all the information about the packages in the virtual environment.
Click==7.0
Flask==1.0.4
itsdangerous==1.1.0
Jinja2==2.10.3
MarkupSafe==1.1.1
Werkzeug==0.16.1
We can use the following command to install all the packages in a new virtual environment:
python -m pip install -r requirements.txt
We can also remove all the packages in the requirements.txt file by the following command:
python -m pip uninstall -r requirements.txt -y
The pip freeze
command takes a --path
parameter, it limits the requirements.txt file to include the packages installed in a certain path.
python3 -m pip freeze --path . > requirements.txt
The pip install
command take a -t
or --target
parameter to install the packages to the path specified.
python3 -m pip install -r requirements.txt -t .
Jupyter
It is very convenient to use Python. Here is the document on Jupyter. We can install Jupyter in a virtual environment.
python3.9 -m venv env-3.9
source env-3.9/bin/activate
pip install pip -U
The above command creates a virtual environment and upgrades pip
in it. We can verify the virtual environment by the following command:
which python
pip --version
We can install Jupyter with the following command:
pip install jupyterlab
We can start Jupyter by the following command:
jupyter-lab
If we want to convert a ipynb file to a regular python file, we can use the following command:
python -m nbconvert --to script --no-prompt Untitled.ipynb
We can check if nbconvert is installed by the following command:
pip list | grep nbconvert
The VSC Integration
The Visual Studio Code is a nice IDE for Python programs. If you are not familiar with VSC, you can take a look at my earlier note. To support Python, I only need to install the Python extension by Microsoft.
To be able to run and debug the program with the Python interpreter by the virtual environment, we can simply create a launch.json file like the following. It points the pythonPath
to the virtual environment.
{
"version": "0.2.0",
"configurations": [
{
"name": "python-search-path.py",
"pythonPath": "${workspaceFolder}/environment-3.4/bin/python",
"type": "python",
"request": "launch",
"program": "${workspaceFolder}/python-search-path.py",
"console": "integratedTerminal"
},
{
"name": "python-inspect-path.py",
"pythonPath": "${workspaceFolder}/environment-3.4/bin/python",
"type": "python",
"request": "launch",
"program": "${workspaceFolder}/python-inspect-path.py",
"console": "integratedTerminal"
},
{
"name": "python-flask-api.py",
"pythonPath": "${workspaceFolder}/environment-3.4/bin/python",
"type": "python",
"request": "launch",
"program": "${workspaceFolder}/python-flask-api.py",
"console": "integratedTerminal"
}
]
}
If you like, you can actually add the path to the python executable in the settings.json file.
{
"files.exclude": {
"**/.git": true,
"**/.gitignore": true,
"**/environment-3.4": true
},
"python.pythonPath": "${workspaceFolder}/environment-3.4/bin/python3.4"
}
In this case, whenever you open the terminal, the virtual environment is automatically activated.
Flask & Gunicorn
The Flask and Django are the web frameworks in Python environment. The python-flask-api.py is a simple flask application.
import sys, os
from flask import Flask, jsonify
app = Flask(__name__)
@app.route('/')
def message():
return jsonify(pid = os.getpid())
if __name__ == '__main__':
app.run(host='0.0.0.0', port=3000)
We can launch the application by the following command:
python python-flask-api.py
But when we start the application, we see the following warning.
We need to find a good server to host the application. We have many options, but Gunicorn is a popular one. We can actually install Gunicorn in the virtual environment.
python -m pip install gunicorn
After installing Gunicorn, we can run it in the virtual environment.
gunicorn --bind 0.0.0.0:8000 python-flask-api:app
If everything goes well, we can now access the application through the 8000
port number. We can also add the -w
option to specify the number of worker processes. Typically, the number of worker processes are 2-4 per core in the server.
gunicorn --bind 0.0.0.0:8000 -w 4 python-flask-api:app
It is important to know that Flask can on multi-thread by itself (Reference).
if __name__ == '__main__':
app.run(threaded=True)
or
if __name__ == '__main__':
app.run(threaded=False, processes=3)
Load Balancing with Nginx
The Nginx is a popular load balancer. It is common to use Nginx to load balance multiple Flask instances. To install Nginx on my Linux Mint computer, we can use the following commands.
apt-cache policy nginx
sudo apt-get install nginx
After installation, we can use the following commands to manipulate the Nginx server.
sudo service nginx start
sudo service nginx restart
sudo service nginx reload
sudo service nginx stop
service nginx status
Because I am experimenting Nginx on my personal computer, I do not want it to start when I start the computer, I use the following command to disable the service.
sudo update-rc.d nginx disable
To set up the load balancing, I created a file named loadbalance.conf.
upstream pythonweb {
server localhost:4000;
server localhost:3000;
server localhost:2000;
}
# This balances all the requests
# It also disable caching
server {
listen 80;
location / {
proxy_pass "http://pythonweb/";
add_header Cache-Control
'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
}
}
# nginx.conf file
# Comment out this line - include /etc/nginx/sites-enabled/*;
#include /home/song/sandbox/p-virtualenv-excercise/loadbalance.conf;
In my computer, I need to "include" this file in the /etc/nginx/nginx.conf file and comment out the default /etc/nginx/sites-enabled/*.
#include /etc/nginx/sites-enabled/*;
include /home/song/sandbox/p-virtualenv-excercise/loadbalance.conf;
With the configuration being ready, we can restart or reload the Nginx server. We need also start 3 instances of the Flask application on the port numbers 2000/3000/4000 in different terminals. The following is the command to start an instance on the port number 2000.
gunicorn --bind 0.0.0.0:2000 python-flask-api:app
If everything goes well, we can then access the application by port number 80 and the requests are balanced among the 3 Flask instances.
Python Classes & OO
Python does support classes and object orientation. In this section, we will be using a few examples to give a quick summary on Python classes & OO.
Example 1 - Python Empty Classes and Dynamic Attributes
class EmptyClass:
pass
obj_1 = EmptyClass()
obj_1.attribute1 = "The attribute 1"
obj_1.attribute2 = "The attribute 2"
print(obj_1.attribute1 + ' - ' + obj_1.attribute2)
print('The __dict__:')
print(obj_1.__dict__)
del obj_1.attribute2
print(obj_1.__dict__)
print()
Unlike classes in the other languages, a Python class does not need any implementation.
- The instance attributes can be added to a Python object dynamically;
- The instance attributes (including the
__dict__
attribute) can be also removed from the object dynamically; - The
__dict__
attribute maintains all the information of the Python object.
Running the above program, we can see the following result:
The attribute 1 - The attribute 2
The __dict__:
{'attribute1': 'The attribute 1', 'attribute2': 'The attribute 2'}
{'attribute1': 'The attribute 1'}
Example 2 - Python Constructor and Instance Methods
class AClass:
def __init__(self, attr_1, attr_2):
self.attr_1 = attr_1
self.attr_2 = attr_2
def print_attributes(self):
print("{} - {}".format(self.attr_1, self.attr_2))
obj_1 = AClass('obj_1_a1', 'obj_1_a2')
obj_2 = AClass('obj_2_a1', 'obj_2_a2')
obj_1.print_attributes()
obj_2.print_attributes()
print()
Although we can add dynamic attributes to Python objects, the most common way to initiate attributes is by a constructor.
- A Python constructor is named as
__init__
, the first parameter to the constructor is the reference of the object; - A Python instance method also takes the first parameter as the reference of the object. By convention, this parameter is named as
self
. - When the constructor and the instance methods are called, the
self
parameter is passed in implicitly by Python.
Running the above program, we can see the following result:
obj_1_a1 - obj_1_a2
obj_2_a1 - obj_2_a2
Example 3 - The Class Attributes
class AClass:
a_class_attr = 'The class attribute'
def print_class_attribute_by_self_reference(self):
print(self.a_class_attr)
def print_class_attribute_by_class_name(self):
print(AClass.a_class_attr)
obj_1 = AClass()
obj_1.print_class_attribute_by_self_reference()
obj_1.print_class_attribute_by_class_name()
print()
obj_1.a_class_attr = 'This can override the class attribute'
obj_1.print_class_attribute_by_self_reference()
obj_1.print_class_attribute_by_class_name()
print()
del obj_1.a_class_attr
obj_1.print_class_attribute_by_self_reference()
obj_1.print_class_attribute_by_class_name()
print()
Besides the instance attributes, a Python class can have class level attributes.
- A class attribute can be access by the object reference and the class name;
- If the object has a instance attribute with the same name as the class attribute, and if the attribute is accessed by the object reference, the instance attribute overrides the class attribute.
Running the above program, we can see the following result.
The class attribute
The class attribute
This can override the class attribute
The class attribute
The class attribute
The class attribute
Example 4 - Class Methods and Static Methods
class AClass:
a_class_attr = 'The class attribute'
def __init__(self):
self.a_instance_attr = 'The instance attribute'
def an_instance_method(self):
print("From the instance method - {}".format(self.a_instance_attr))
@classmethod
def a_class_method(cls):
print("From the class method - {}".format(cls.a_class_attr))
@staticmethod
def a_static_mathod():
print('Print from a static method')
obj_1 = AClass()
obj_1.an_instance_method()
obj_1.a_class_method()
obj_1.a_static_mathod()
print()
AClass.a_class_method()
AClass.a_static_mathod()
A Python class can have class methods and static methods.
- Annotating a method by
@classmethod
makes it a class method. When the method is called, the reference to the class is implicitly passed in as the first parameter. By convention, this parameter is named cls
; - Annotating a method by
@staticmethod
makes it a static
method. When the method is called, neither the object reference nor the class reference is passed in implicitly.
Running the above program, we can see the following result.
From the instance method - The instance attribute
From the class method - The class attribute
Print from a static method
From the class method - The class attribute
Print from a static method
Example 5 - Inheritance
class ParentClass:
def __init__(self):
self.parent_attr = 'Parent Attribute'
def print(self):
print("Print from parent class - {}".format(self.parent_attr))
class ChildClass(ParentClass):
def __init__(self):
super().__init__()
self.child_attr = 'Child Attribute'
def print(self):
print("Print from child class - {}".format(self.child_attr))
parent_obj = ParentClass()
parent_obj.print()
child_obj = ChildClass()
child_obj.print()
print()
print('Test polymophism')
objects = [parent_obj, child_obj]
for obj in objects:
obj.print()
Same as other OO languages, Python supports inheritance.
- The
super()
method can be used to access the method in the parent class; - A method is overridden if the child class implements the same method;
- Python objects show polymorphism behavior similar to other languages.
Running the above program, we can see the following result:
Print from parent class - Parent Attribute
Print from child class - Child Attribute
Test polymophism
Print from parent class - Parent Attribute
Print from child class - Child Attribute
Spark & PySpark
Points of Interest
- This is a note on Python & Virtualenv & VSC & Miscellaneous.
- I hope you like my posts and I hope this note can help you one way or the other.
History
- 28th January, 2020: Initial version