Square and Non-Square Pixels

By Chris Pirazzi. Thanks to Bob Williams for much of the information. Thanks to Charles Poynton for pointer to RP 187 and other info.

What Are They?

Pixels in the graphics world are square. A 100 pixel vertical line is the same length as a 100 pixel horizontal line on a graphics monitor.

Pixels in an ITU-R BT.601-4 digital video signal (also known as Rec. 601 and formerly CCIR 601) are non-square. A 100 pixel vertical line may be longer or shorter than a 100 pixel horizontal line on a video monitor, depending on the video standard. See below for details.

The term which describes this is pixel aspect ratio. In SGI libraries it is specified as a fraction of vertical (y) pixel size divided by (x) horizontal pixel size. The pixel aspect ratio for square pixels is 1/1.

This term is different than frame aspect ratio, which is a fraction of total vertical (y) frame size over total horizontal (x) frame size, for a given definition of "frame."

Why Do I Care?

Many image processing operations assume a certain pixel aspect ratio. For example, a blur operation with a radially symmetric kernel may look bad on non-square pixels.

Any rendering of geometry-based graphics to an image must take the pixel aspect ratio into account.

An analog video signal consists of discrete video lines, each of which is a continuous analog signal that represents the luma and chroma for that line of the image. So an analog video signal has no "pixels." When a video board samples a video line, it can sample whatever part of the line it pleases, and it can sample the line at whatever interval it pleases. For a given video signal format (525/625), a board may choose to sample the line according to one of two rules: square sampling or non-square sampling.

A Rec. 601 digital video signal is non-square. Some SGI video boards have horizontal resampling hardware which will give you square pixels in memory even if you have a Rec. 601 signal.

Therefore, if you get a buffer of pixels from a video board or give one to a video board, you need to know whether your pixels are square or non-square. Among other things, this will determine the image dimensions of your buffer.

How Do I Tell Which Pixel Aspect Ratio I Have?

Any drawing to a graphics monitor using OpenGL and IRIS GL is in square pixels. You can also use SGI graphics boards to process non-square pixels, on-screen or off-screen, by drawing to a framebuffer or pbuffer and then reading back the pixels using glReadPixels() (or lrectread()).

Images in the DM (dmIC, etc.), MV, or CL libraries may be square pixel or non-square pixel, depending on context. There is a DM parameter DM_IMAGE_PIXEL_ASPECT which specifies the pixel aspect ratio (y/x) in movie files and other situations. Be careful: some of the movie capture and conversion programs shipped with SGI systems even as recently as IRIX 6.3 create movies where this parameter is always 1.0, or is an x/y value instead of a y/x value. These are known problems that will get fixed, but movies may exist with the bogus pixel aspect ratios (see Motion JPEG on SGI Systems for more on that).

For our VL and compression devices,

vino data from analog input jacks is always square pixel.
vino data from the IndyCam is always square pixel.
vino data from the Miranda Mindy 601 Dongle is always non-square pixel.
ev1 may decode/encode analog video squarely, nonsquarely, or both, depending on the board. Check out ev1 and cosmo1 for the specifics.
ev1 data from the IndyCam is always square pixel.
ev1 always inputs and outputs Rec.601 video non-square pixel.
cosmo1 will use whatever kind of video data the connected ev1 feeds it.
ev3 is a totally non-square pixel board. it has no analog ins or outs.
cosmo2 always encodes/decodes analog video squarely. it can also compress or decompress non-square data from memory or from a connected ev3 board.
sirius lets you choose square or non-square encoding/decoding of analog video signals
sirius always inputs and outputs Rec.601 video non-square pixel.
mvp has horizontal resampling hardware, so you can have either square or non-square pixels in memory, regardless of the video signal format. Note: mvp's 625-line square pixels are not completely square. See below.
divo is a totally non-square pixel board. it has no analog ins or outs.

What Is the Pixel Aspect Ratio of Non-Square Pixels?

For 525-line Rec.601 video, the pixel aspect ratio is exactly 11/10 (y/x).
For 625-line Rec.601 video, the pixel aspect ratio is exactly 54/59 (y/x).

Example dmParams calls to set the params:

525-line non-square (Rec. 601):

  dmParamsSetFloat(userImgParams, DM_IMAGE_PIXEL_ASPECT,
                   ((double) 11.0 / (double) 10.0));

625-line non-square (Rec. 601):

  dmParamsSetFloat(userImgParams, DM_IMAGE_PIXEL_ASPECT,
                   ((double) 54.0 / (double) 59.0));

Contrary to popular belief, the pixel aspect ratios are not 4:3, nor are they 720/640 and 768/720! The actual ratios are defined purely in terms of the pixel sampling frequency of each video standard:

Rec. 601 digital video is always sampled at 13.5 million pixels per second (for both 525 and 625).
If you have a 525-line analog NTSC (ANSI/SMPTE 170M-1994) video signal which you want to sample square pixel, the industry standard is to sample at exactly 12 + 27/99 million pixels per second.
If you have a 625-line analog PAL (Rec. ITU-R BT.470-3) video signal which you want to sample square pixel, the industry standard is to sample at exactly 14.75 million pixels per second.
Therefore, we can derive from this that:
525-line Rec.601 pixel aspect ratio = 13.5 / (12 + 27/99) = exactly 11/10 (y/x)
625-line Rec.601 pixel aspect ratio = 13.5 / (14.75) = exactly 54/59 (y/x)

Wait, How Can That Work?

"OK smarty," you may now be saying, "if the pixel aspect ratio is not 720/640 for 525-line or 768/720 for 625-line, then how can I resolve this with the fact that my images are 720, 640, or 768 pixels wide?" The answer to this is that a 720-pixel-wide non-square pixel image and a 640- or 768-pixel wide square pixel image do not represent the same horizontal range of the underlying signal! Here is an illustration which shows you how to convert between square and non-square pixel images which have the standard dimensions:

When the diagram says "pad," this means you need to pad the image out to 720 non-square pixels. Usually this means that you start with a greater number of square pixels than 640 or 768. Sometimes those pixels are not available, and you have to improvise.

When the diagram says either "pad" or "crop," it is best to do the padding or cropping so that the image stays centered.

A video board which is encoding or decoding an analog signal will never need to pad or crop: it can measure or synthesize the continuous analog signal for each video line any way it wants. Each analog video line is long enough to accomodate either horizontal range.

An mvp Quirk

Internally, mvp always encodes or decodes analog and digital video non-squarely. Then, if you have requested square pixel data in memory, mvp digitally resamples each line as it goes into memory or comes out of memory. mvp uses the correct resampling ratios shown above for 525-line video, but for 625-line video, mvp uses 11/12 and 12/11 instead of 54/59 and 59/54, as shown here:

As a result, the 768-pixel-wide images you get from mvp for video-to-memory operations will be horizontally compressed by 1 7/33 pixel, and the 768-pixel-wide images you provide to mvp for memory-to-video operations will be horizontally expanded by 1 5/59 pixel on the video signal (where one pixel is defined as a Rec.601 luminance sampling instant). This quirk only affects square-pixel 625-line data; square 525 and non-square 625 and 525 work correctly.

This problem may be fixed in a future release of the mvp video hardware or system software. Unless you really need to account for this quirk now, we recommend that you assume the mvp hardware implements the correct ratios.

Square Pixel Sampling Frequencies and SMPTE RP 187

Lurker's Guide Trivia Contest! If you know a way to derive the industry standard square pixel sampling frequencies above or you know some historical clues that could lead to a derivation, send us some mail! The prize is your name in lights on the top of this page. Well, ok, your name on the top of this page.

In theory, we could derive the square pixel sampling frequencies from the frame aspect ratio and other statistics from each analog signal spec. Unfortunately, while both ANSI/SMPTE 170M-1994 and Rec. ITU-R BT.470-3 state that the active picture has a 4:3 frame aspect ratio, neither spec defines "active picture" well enough to use in deriving an accurate square pixel sampling frequency. So the square pixel sampling frequencies in use today are industry conventions, not stated in any spec.

In 1995, SMPTE created RP 187, "Center, Aspect Ratio, and Blanking of Video Images," to address exactly that problem. Unfortunately, instead of recommending the values above, which were de facto industry standards even in 1995, RP 187 recommended a pixel aspect ratio of 177/160 (y/x) for 525-line video and 1035/1132 (y/x) for 625-line video. In addition to conflicting with pretty much every piece of square-sampling equipment in existence, these numbers are highly impractical to implement. They result in a non-integral numbers of luma and chroma sampling instants per line, which makes hardware people jump out of high windows. At a time when the industry so badly needed to standardize pixel aspect ratio and sampling frequencies, it's really too bad that RP 187 chose such impractical values: RP 187 will probably create more problems than it solves. Strangely, RP 187's informative annex A.4 talks about square to non-square conversion of computer images, and suggests resampling the image by a factor of 11:10---thus using the industry standard values and contradicting itself!

Pixel Aspect in MPEG I and MPEG II

The sequence header of MPEG I video bitstreams contains an enumerated field identifying the pixel aspect ratio of the video images. In rev 1 of ISO/IEC DIS 2-11172, the MPEG I committee assigned two of the 16 tokens for the aspect ratios of 525- and 625-line video. Unfortunately, the values behind these tokens were not the 11/10 or 54/59 above. The MPEG committee derived the values from a fixed 4:3 frame aspect ratio times a canonical image height and width in pixels which the committee seems to have made up. The pixel aspect ratio does not and should not depend on the video bitstream image size in pixels. The pixel aspect ratio is determined by the sampling grid of the device capturing the video; you can capture many different image sizes from the same grid.

In rev 2 of ISO/IEC DIS 2-11172, the committee changed the pixel aspect ratios in the table, but sadly they still were not 11/10 or 54/59. The committee only changed the made-up canonical image dimensions to other made-up canonical image dimensions.

Even more bizarrely, in both drafts, the committee decided to fill in the 12 open slots in the pixel aspect ratio table with linear interpolations and extrapolations of the 525- and 625-line values! They topped it off with this conundrum:

"It is evident that the specification does not allow all possible pel aspect ratios to be specified. We therefore presume that a certain degree of tolerance is allowable. Encoders will convert the actual pel aspect ratio to the nearest value in the table, and decoders will display the decoded values to the nearest pel aspect ratio of which they are capable."

Hopefully from this data you have learned not to trust the pixel aspect ratio encoded in an MPEG I video bitstream.

The MPEG II committee discarded the MPEG I pixel aspect table. But unfortunately it retained the notion that pixel aspect ratio depends on encoded image size. Pixel aspect of an MPEG II video bitstream sequence can be 1.0, or it can be specified in terms of a frame aspect ratio times a ratio of the image height and width in pixels. The frame aspect ratio can be a fixed value (4:3 or 16:9) or it can be encoded in the bitstream as any two integers. Unlike MPEG I, it's possible to twist the MPEG II paramers around in order to make them say what they should originally have said: that the pixel aspect is 11/10 or 54/59. It's too bad that the committee didn't just offer these as constants.

So I would watch out for the pixel aspect ratios encoded in all kinds of MPEG. If you know that data came from non-square-sampled video, use your own judgement.