One Hour PNG Development - Image Encoding in Python
In this post, we look at rapidly prototyping a PNG image encoder in Python.
The aim is to create a PNG encoder that does not rely on any code not
available within the Python standard library, and in doing so learn a bit about
the internal structure of the PNG format.
You should already be familiar with
what a PNG file is
and with basic image transmission concepts.
All of the code required is given in this post, however a repository of the
complete program can be
found on GitHub.
Format Summary
First let's summarise some information on the PNG format specification,
mostly found by reading the standard at
www.w3.org/TR/PNG/
Multi-byte integers are stored big endian (aka network order)
Compression algorithm is mandatory (but easily implemented with Python 3)
First 8 bytes of PNG (signature) [1] are - 137 80 78 71 13 10 26 10
After the signature, the file is divided into chunks [2]
The minimum mandatory chunks are a header (chunk type "IHDR"),
image data ("IDAT"), and an image trailer ("IEND"), appearing in that order.
Chunk and Data Formats
Table 1. Chunk Format
Offset
Length
Field
0
4
Length - number of bytes in data field
4
4
Chunk Type - four ASCII alphabetic characters
8
n
Data Field
n+8
4
CRC check value
IHDR Header Chunk
Table 2. Data field contents for IHDR chunk
Offset
Length
Field
0
4
Width - image width in pixels
4
4
Height - image height in pixels
8
1
Bit Depth - bits per colour plane per pixel - 1, 2, 4, 8 or 16
9
1
Colour Type - 2 = Truecolor RGB with 8 or 16 bits per colour per pixel
The data field contains the compressed image data as output by the
compression algorithm.
IEND Image Trailer Chunk
This marks the end of the file and has an empty (zero length) data field.
Compression
PNG uses an LZ77 based compression algorithm called Deflate.
Working in Python 3 allows us to hand-wave over the details of the
compression algorithm [4],
as Python provides an implementation called zlib within
it's standard library. [5]
Conveniently, the zlib library is even able to calculate
CRC values for all of our chunks for us.
Writing an Encoder
We will be following the common code pattern defined by use of a main()
function.
Since we are using the zlib library we are sure to import it at the top of our
file. Our first piece of code creates a bytes object holding the signature
values defined in the standard.
Following tables 1 and 2 above, we create a function to take any data
and build it into a chunk, and another to generate the data for an IHDR header
chunk.
def make_chunk(type, data=b""):
b = len(data).to_bytes(4, byteorder="big") # Length of Data
b += bytes(type, "ascii") # Chunk Type
b += data # Data Field
crc = zlib.crc32(bytes(type, "ascii"))
crc = zlib.crc32(data, crc)
b += crc.to_bytes(4, byteorder="big") # CRC
return b
def make_header(width, height):
b = width.to_bytes(4, byteorder="big") # Width
b += height.to_bytes(4, byteorder="big") # Height
b += (8).to_bytes(1, byteorder="big") # Bit Depth
b += (2).to_bytes(1, byteorder="big") # Color Type = Truecolor
b += (0).to_bytes(1, byteorder="big") # Compression Method
b += (0).to_bytes(1, byteorder="big") # Filter Method
b += (0).to_bytes(1, byteorder="big") # Interlace Method = None
return b
We can now add a call to these functions into main() to show them working.
Running this will output a string of gobbledygook, but if you examine it
closely you will find it corresponds to the data structures defined above - note
the two instances of "\x04" denoting the width and height, and "\x08\x02" for
the bit depth and colour format bytes.
In this article I will look at creating a minimal custom user model in
Django.
The aim is to create the simplest possible user model, providing only a
username field and a password field, that will allow logging in, logging out and
testing authorisation with the standard Django auth interface.
Being as small as possible is the goal at all costs. The user model we create
here will not be suitable for accessing the Django admin site, as the admin
requires fields that we will be leaving out.
This user model is unlikely to be usable as it stands, but provides a clean
slate on which a customised user model could be built, without clutter in the
background.
Research - AbstractBaseUser
Online sources including
the official Django documentation
suggest we may want to build our custom user model on top of the
AbstractBaseUser class. We will begin there, and it is instructive to first
examine the code in the libary for AbstractBaseUser.
Examining AbstractBaseUser in the django library, a password field is
already provided for us, along with a last login field, a fixed is_active
property, and the REQUIRED_FIELDS empty list)
Coding Our Custom User Model
In this example, we have named our custom user app 'customusers'.
We will be writing our classes in 'customusers/models.py', building on
BaseUserManager and AbstractBaseUser, so we include these at the top of
the file.
customusers/models.py
from django.db import models
from django.contrib.auth.models import BaseUserManager, AbstractBaseUser
The User Manager
Our custom user model will require a custom user manager, whose only required
purpose is to handle creation of users. The user manager is set as the 'objects'
property on the user model, and inherits from BaseUserManager, which in turn
inherits from models.Manager, therefore inheriting the usual database management
methods.
Django passwords are stored as a hash and therefore need to be set using the
set_password function (provided by the AbstractBaseUser class).
The create_superuser method is required so `manage.py createsuperuser` will
work.
customusers/models.py
class CustomUserManager(BaseUserManager):
def create_user(self, username, password):
user = CustomUser.objects.create(username=username)
user.set_password(password)
user.save()
return user
def create_superuser(self, username, password):
user = self.create_user(username, password)
return user
Python 3 Venv - Local and Central Virtual Environments
Since version 3.3, Python includes the module venv which allows
you to create isloated environments, where you can install
any packages, libraries etc. that your project may need, without cluttering
or affecting other projects on your system - you can have as many separate
environments as you like.
Since venv comes as part of Python it is available to use at all
times without requiring installation.
Using Venv
First I will show you how to create a virtual environment in your project
folder, as most tutorials do. But please then read on as other
possibilities abound!
The following assumes you are working on MacOS or Linux using the bash shell.
Starting in the folder for your python project:
Create a new virtual environment in this folder with
$ python3 -m venv .venv
This creates a new subfolder for the virtual environment files named
.venv (this is idiomatic but you can name it whatever you want)
Activate your virtual environment (for the terminal you are working in)
with $ source .venv/bin/activate - when activated, by default
your shell prompt will be modified with the environment name in brackets
eg. (.venv)$
You can now install/do stuff that won't impact
the system-wide python installation, eg. pip install Django
To deactivate, and return to the system default environment, run $ deactivate
If you usually have to call Python with the command `python3`,
while the virtual environment is active you are able to use just `python`
instead.
You can also run your virtual version of Python without activating the
virtual environment, by referencing the binary in the .venv directory
directly, eg.
$ .venv/bin/python
Centralised Virtual Environments with Venv
Tutorials I have seen on venv only show you how to create a virtual
environment in the project folder you are working on, but this does not need to
be the case at all.
You can just as easily create a virtual environment in a centralised location
and use it while working on multiple projects. For example, if you have several
Django projects you're working on, you don't need to have a virtual environment
in each separate project, all with the same version of Django installed;
just create one environment in a convenient location.
You might for example choose to keep central virtual environments in a
subfolder of your home directory such as ~/.venv/
Following the same instructions as above
(and naming our environment django as an example):
Create a new virtual environment with
$ python3 -m venv ~/.venv/django
This creates a new subfolder under your home directory
~/.venv/django/
Activate your virtual environment with
$ source ~/.venv/django/bin/activate - and your shell prompt changes to
the environment name (django)$
Install stuff and operate as with a project-based environment, eg.
pip install Django
To deactivate, run $ deactivate
You can also run this virtual environment without activating using direct
reference, eg.
The BBC Micro was a
Microcomputer made by
Acorn Computers; introduced at the end of 1981 as part of the
BBC Computer Literacy Project, it was succesful both as home computer and in
the educational market, with most schools in the UK using the machine for
computing education. During the 1980s and 1990s, using a BBC Micro in primary
school was the first experience of a computer for many people -
including the author.
The Fault, Testing and Research
This example, despite being in immaculate physical condition, really wasn't
happy when turned on; it seemed to put out a very bad video signal. Sometimes it
appeared it was trying to work, but the picture would judder wildly.
Testing with an oscilloscope revealed the video signal looked ok, but on
closer examination the vertical sync rate was 80Hz, obviously not to the liking
of the 50Hz monitor!
A little research and head scratching later I figured out this is
symptomatic of a RAM fault, causing the CPU to read garbage values that then
get written into the 6845 video controller during boot, resulting in the weird
output frequency. Further I came across a trick of moving a jumper on the
motherboard (S25) to disable
half the RAM, and this indeed worked, the machine operating happily and
reporting itself with 16k of memory. [1]
The Arduino Uno is a readily available development board for the
Amtel ATmega328 microcontroller, allowing the user to easily experiment and
iterate with their design. For use in an actual product, or to make a one off
gadget even smaller, it is advantageous to use the microcontroller chip
independent of the Arduino board. These notes look at operating the ATmega328
by itself on a breadboard, and how to program the chip in situ using an Arduino.
Advantages of Bare Microcontroller
Running with a slower clock will save power (when high performance not required).
Using the internal clock will eliminate need for crystal (when crystal-accurate timing not required).
More flash memory available by not using Arduino bootloader code.
Faster start up time - Arduino bootloader causes a delay on start up before
user code starts running.
In this post I will look at the basics of making alert dialogs
appear on Android.
An example Android Studio project demonstrating the code shown here is
available on GitHub.
Overview
Android provides the AlertDialog class for creating dialog boxes with
a simple structure of a title, content (typically a text-based message), and
up to three action buttons.
An instance of AlertDialog is created (using the provided Builder class)
and various setter methods are used to create our design, which can then
be shown with the show() method.
Methods of AlertDialog
Builder(Context)
Nested class, who's create() method
returns a new AlertDialog object. Context parameter should be the current
activity object, eg. MainActivity.this
PHP has a really useful feature, which is a built in web-server, great for
local testing of a php website. It's all installed and available by default
on MacOS and some versions of Linux. To use,
cd to the html root directory and run:
$php -S localhost:8888 testserver.php
This will run the file testserver.php for each request received from the
local machine at port 8888.
Example testserver.php file:
<?
/* remove get parameters */
$file_uri = explode('?',$_SERVER['REQUEST_URI'],2)[0];
/* remove leading / to make relative unix path */
if ($file_uri[0] == "/") $file_uri = substr($file_uri,1);
/* run controller to handle non-existent uri */
if (! is_file($file_uri)) {
include 'controller.php';
} else {
/* all other files served automatically by php server */
return False;
}
?>
The KHP-30 is a high voltage probe with a built in meter, measuring up to
30 kV DC. It is meant primarily for testing the anode voltage on CRTs.
The box indicates this was made in Japan, and based on
some of the materials used in the probe I would say it dates from the 1970s.
This example is branded 'Eagle Products', but I have seen otherwise identical
looking probes with different brand names online.
Specifications
Input Impedance 600 Meg
Current Draw 50 uA at 30 kV
Schematic
All resistors have 5% tolerance. The adjustable leg with R3 and the preset
resistor takes about 10% of the current and can be adjusted to compensate for
the tolerances of the resistors and the meter movement.
My meter movement was full scale at 43uA and measured 3,200 ohms. The preset
resistor was set for a resistance of about 57k. For details of the "ground ring"
see the constructional information and photographs below.
I reckon the probe is built to withstand continous operation, a full scale
reading would only dissipate 1.5W, however with my example after 5 minutes or so
measuring 22kV the reading slowly rises about 5 percent, as the large resistors
do warm up a little. The reading returns to normal after giving it a few minutes
to cool.
