Note: This document uses the O2 VL DMbuffer API instead of the cross-platform VL DMbuffer API (see What Are the SGI Video-Related Libraries? for more info). It refers to IRIX 6.3 man pages and documentation.

This page describes how to render graphics onto a live video image and send the results out as video. There are several different approaches to doing this: you can render directly onto the video frame, you can render graphics to the screen and then blend the results onto a video frame, or you can copy the video to and from the screen. The thing that all of these approaches have in common is the use of the OpenGL DMbuffer extension to allow video frames to be used as drawables and readables in OpenGL.

There are example programs that illustrate all of these techniques. They are included in the dmedia_dev.src.examples subsystem in "IRIX 6.3 for O2 including R10000" and later releases for O2. The sources are installed in "/usr/share/src/dmedia/video/O2". Each of the programs illustrates several of the techniques described below:

• clock
  • Rendering directly on video
  • Rendering in YCrCb
  • Using stencils to handle fields
• copypixels
  • Copying video to and from the screen
• fieldblend
  • Blending graphics from the screen onto video
  • Using pixel zoom to handle fields

Setting up the Video Path

When using video with graphics, the video input and output paths are set up in the normal way, except that the "graphics" image layout must be used, and the pixel format must match the format in the pbuffer being used.

The image layout is set using vlSetControl on the memory node to set VL_LAYOUT to VL_LAYOUT_GRAPHICS. The pixel format is also set on the memory node, setting VL_PACKING to VL_PACKING_ABGR_8 (assuming an 8-bit-per-component pbuffer).

Setting up Graphics

Setting up OpenGL is also pretty straightforward. A digital media pbuffer must be created that is exactly the same size as the video frames, and with the same pixel depth. When using 8-bit-per-component video, an select an FB config that has exactly 8 bits each of R, G, B, and A. It can also include depth and stencil buffers as needed. (The examples below use a stencil buffer to control field rendering.)

Rendering Fields

There are basically three approaches to handle fields:

1. Ignore the problem. When the graphics being rendered is not moving, or is not moving very fast, treating video as frames can be good enough. Video is captured as VL_CAPTURE_INTERLEAVED, graphics are rendered once per frame, and the results are combined.
   
2. Render fields. For some types of graphics, each field can be rendered independently. In this case, the video fields are captured separately (VL_CAPTURE_NONINTERLEAVED). Graphics are rendered at half height, and then combined with the fields. This approach does not work well when drawing diagonal lines. Aliasing artifacts are exacerbated by the facte that each pixel being rendered is effectively two pixels in the resulting video.
   
3. Render a full frame for each field. This approach provides the best quality, at the expense of rendering twice as many pixels. For each field, a full frame is rendered, but only every other line is kept and sent to video output.

The first two are easy. Below are two tricks that can be used to implement approach number 3.

Rendering Fields using Stencils

You can use OpenGL's stencil test to selectively draw to alternate lines. Using this, the flow of control looks like this:

• Set up a stencil that alternates lines
• For each video frame
  • Get a frame from video input
  • Activate stencil to draw to field 1
  • Render a full frame at the time of field 1
  • Activate stencil to draw to field 2
  • Render a full frame at the time of field 2
  • Send the results to video output

Setting up the stencil just means drawing into half of the lines in the buffer:

    glClearStencil( 0x0 );
    glClear( GL_STENCIL_BUFFER_BIT );

    glEnable( GL_STENCIL_TEST );

    glStencilFunc ( GL_ALWAYS, 0x1, 0x1 );
    glStencilOp( GL_REPLACE, GL_REPLACE, GL_REPLACE );

    glColor3f( 1.0, 1.0, 1.0 );

    glBegin( GL_LINES );
    {
        int line;
        for ( line = 0; line < height; line += 2 ) {
            glVertex2f( 0.0, line + 0.5 );
            glVertex2f( width, line + 0.5 );
        }
    }
    glEnd();

    glDisable( GL_STENCIL_TEST );

    glClearColor( 0, 0, 0, 0 );
    glClear( GL_COLOR_BUFFER_BIT );

This leaves a stencil with 1s is half of the lines and 0 in the other half. The two fields can then be rendered like this:

    glEnable( GL_STENCIL_TEST );

    glStencilFunc( GL_EQUAL, 0x1, 0x1 );
    glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );
    renderField( field_1_time, width, height );

    glStencilFunc( GL_NOTEQUAL, 0x1, 0x1 );
    glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );
    renderField( field_2_time, width, height );

    glDisable( GL_STENCIL_TEST );

Selectively Copying Fields with glPixelZoom

By setting glPixelZoom to (1.0,0.5), glCopyPixels can be used to copy every other line of a frame to produce a field. Assuming that you are rendering 60 full frames per second, this technique can be used to copy out individual fields and send them to video output. To do this, the graphics buffer should be the size of a video frame, while the video output path should be set for fields (VL_CAPTURE_NONINTERLEAVED).

Rendering in the YCrCb Color Space

Normally, an OpenGL buffer contains pixels represented as red, blue, green, and alpha components. The color in a video signal is represented as Y, Cr, and Cb components. In order to display video on a grahics screen, you must convert from RGBA to YCrCb. To send images from the graphics screen to video output, the inverse conversion, from YCrCb to RGBA, must be performed.

For the most part, the operations performed by OpenGL are correct for any orthogonal color space, including YCrCb. Goraud shading, texturing, and blending all work just as well in YCrCb. Lighting effects don't work, though.

If the images you are rendering can be rendered off-screen and don't using lighting, then they can be rendered directly in YCrCb. The trick is to capture and output the video using the VL_PACKING_AUYV_4444_8 packing. This brings in the video, packing the pixels in the order: Cb, Y, Cr, A.

When drawing, though, you need to pass CbYCrA values to glColor, because those are the components stored in the buffer. This function can be used to convert from RGBA to CbYCrA:

void rgbaColor(
    float r,
    float g,
    float b,
    float a
    )
{
    /* This is from Video Demystified, 2nd edition, page 43 */
    float y  = ( 255.0 * ( 0.257 * r + 0.504 * g + 0.098 * b ) +  16 ) / 255.0;
    float cr = ( 255.0 * (-0.148 * r - 0.291 * g + 0.439 * b ) + 128 ) / 255.0;
    float cb = ( 255.0 * ( 0.439 * r - 0.368 * g - 0.071 * b ) + 128 ) / 255.0;

    glColor4f( cb, y, cr, a );
}

Rendering Directly onto Video

On O2, the most efficient way to add graphics to a stream of live video is to use DMbuffers with OpenGL to render directly onto a video frame, without having to copy the pixels to or from a pbuffer. The sequence of events is this:

• For each video frame:
  • Get a frame (in a DMbuffer) from video input
  • Associate the DMbuffer with a pbuffer
  • Render to the pbuffer (which renders to the DMbuffer)
  • Send the DMbuffer to video output

Blending onto Video

If you have an image in a pbuffer or on the screen that includes alpha values, it can be blended onto a video frame in the same way that geometry can be rendered onto video. (See above.)

Blending must be enabled, and then the pixels can just be copied from the screen to the video buffer:

    glXMakeCurrentReadSGI( dpy, videoDrawable, screenDrawable, context );
    glEnable( GL_BLEND );
    glBlendFunc( GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA );
    glRasterPos2i( 0, 0 );
    glCopyPixels( 0, y, width, height, GL_COLOR );
    glDisable( GL_BLEND );