Hints for Vid-to-Mem Applications

By Nelson Bolyard.
Updated for mvp by Chris Pirazzi.

This document supplements the material in the "IRIS Media Libraries Programming Guide" (hereinafter, "the Guide"). It adds information about the VL that I think is missing or wrong in the Guide. It doesn't attempt to answer all remaining questions about the VL, or to be a complete reference. This is merely the information that I figured out while making InPerson work on vino, ev1, ev3, and sirius video for IRIX 5.3 and 6.2. Chris updated this document for mvp and IRIX 6.3. This information applies only to those VL video devices, and not to any other present or future VL video devices (it does not yet completely reflect cosmo2 and divo, for example).

This document does not represent SGI, and is not an official SGI publication. The entire contents of this document are my personal observations, and/or opinions, and all errors and opinions in it are mine alone. SGI makes no guarantees about anything in this document. Neither do I. You use it at your own risk. Additional disclaimers below.

Some topics NOT discussed in this page include:

Video source node controls

VL_FORMAT

 The VL_FORMAT control on the video source node is usually set using the Video Control Panel. It is often of no concern to a vid-to-mem application, except with sirius video, where it is used to determine color space conversion. See the discussion of the VL_PACKING and VL_FORMAT control on the memory drain node (below) for more details.

VL_TIMING

The VL_TIMING control on the video source node is usually set using the Video Control Panel. It is often read by vid-to-mem apps to determine if the input source timing is PAL or NTSC, and if the pixels will be square or not. It also affects the value returned by the VL_SIZE control on the video source node. The knowledge of PAL vs. NTSC timing is useful to the vid-to-mem app for setting the VL_RATE control on the memory drain node. For sirius Video, it is also used to determine the value for the VL_TIMING control on the memory drain node. See the discussion of the VL_TIMING control on the memory drain node (below) for more details.

VL_SIZE

There are two separate VL_SIZE controls that a vid-to-mem app must deal with. One is a control on the video node, the other is on the memory node. They have different meanings, different units, and do different things. The most common problem in discussing the VL_SIZE control is people forget to specify which one they're talking about, or are confused about that. This section discusses the VL_SIZE control on the video source node.

The VL_SIZE control on the video source node is a read-only control. The x and y values returned by this control are affected by the setting of the VL_TIMING control on the video source node. The x and y values of this control are not, in general, affected by the settings of any controls in the memory drain node, including VL_ZOOM, VL_SIZE, and VL_CAP_TYPE.

The x component value of this control reveals the width, in pixels, of the unzoomed, unclipped video input images (fields or frames). The meaning of the y component value of the video source node's VL_SIZE control depends on the video device.

Memory drain node controls

VL_PACKING (plus VL_FORMAT or VL_COLORSPACE)

A vid-to-mem app chooses: On all devices except sirius and divo, an application specifies both of these with VL_PACKING on the memory node.

On sirius, an application uses VL_PACKING on the memory node to specify the packing, and VL_FORMAT on the memory node to specify colorspace. If necessary, sirius will do colorspace conversion. Sirius is the only device where VL_FORMAT even applies to memory node.

On divo, an application uses VL_PACKING on the memory node to specify the packing, and VL_COLORSPACE on the memory node to specify colorspace. If necessary, divo will do colorspace conversion. This is a new API which should be inherited by all new VL devices from divo and beyond.

For complete information on pixel packings and colorspaces, check out The Pixel Rosetta Stone: Packings and Colorspaces.

The VL_PACKING_YVYU_422_8 packing is the only packing that is natively supported in hardware (requiring no software conversion) on all VL video devices. The packing always uses 4:2:2 sampling, and the default colorspace for this packing is always headroom-range. See The Pixel Rosetta Stone: Packings and Colorspaces for definitions of these terms.

VL_TIMING

On all VL video devices except sirius, the VL_TIMING control applies only to the video (not memory) nodes. The timing on the video source node is usually set through the Video Control Panel. With sirius video, the VL_TIMING control must be set properly on the memory drain node. An easy way to set the VL_TIMING value for the memory node is to read the value of the VL_TIMING control from the video source node, and then set that value into the VL_TIMING control for the memory node.

VL_CAP_TYPE

On all VL video devices except sirius, this control is settable by the application. Its setting determines whether the images in the buffers returned by the VL are individual fields, or interleaved frames, or pairs of non-interleaved fields.

On sirius, this control is read-only, and is permanently set to VL_CAPTURE_NONINTERLEAVED. Each captured buffer contains exactly one field, unclipped, unzoomed.

The possible values are:

 Vino has a bug that affects capturing with VL_CAPTURE_INTERLEAVED. The first buffer returned after the call to vlBeginTransfer() may have only one field in it. The other field may contain all zero values. For continuous transfers, this may go unnoticed, but for VL_TRANSFER_MODE_DISCRETE captures (e.g. of a single image), this will often result in an image in which every other line is black, or green, depending on the color space in use. A missing field can be detected by examining the first pixel or two in several successive pixel rows. The workaround for this problem is to examine the image for the presence of one black field, and one non-black field, and recapturing another buffer if one field is black. Another solution worth exploring is to capture two buffers, instead of one, and then throw the first one away. This might suffice.

There is no single VL_CAP_TYPE that is available, and implemented in the same way, on all VL video devices. VL_CAPTURE_NONINTERLEAVED is available on all devices, but has different meanings on different platforms. VL_CAPTURE_INTERLEAVED, VL_CAPTURE_EVEN_FIELDS, and VL_CAPTURE_ODD_FIELDS are available and common to all VL video devices except sirius.

VL_RATE

In the section entitled "Setting Drain Node Controls for Data Transfer", the Guide states, incorrectly, that the VL_RATE control is "not applicable" to memory drain nodes. For vid-to-mem applications, the memory drain node is precisely where video rate is controlled.

On all VL video devices except sirius, this control sets the target rate (upper bound) of image buffers per second that will be captured and returned to the application. That is, the VL will not deliver more buffers per second than the rate you specify, but it might deliver less than you specify, especially on ev1.

VL_RATE is in units of buffers per second. That may be frames or fields per second depending on whether each buffer entry has a frame or field in it, which is determined by VL_CAP_TYPE. See VL_CAP_TYPE above for more info.

VL_RATE is a fraction. Both the numerator and denominator must be specified. The usual value for the denominator is 1. Some devices, such as ev1, internally convert the fraction to a whole integer (truncating, not rounding) of images per second, so using values that are equivalent to integer values is the safest thing to do.

Because VL_RATE is a fraction, vlGetControlInfo() cannot be used to obtain the minimum or maximum values for VL_RATE.

Acceptable values are determined from the following table for non-sirius devices:

VL_CAPTURE_NONINTERLEAVED
(all devices except ev1 and sirius) 		525 All multiples of 10 and 12 between 10 and 60
625 All multiples of 10 between 10 and 50
VL_CAPTURE_NONINTERLEAVED
(ev1) 		525 All multiples of 5 and 6 between 5 and 30
625 All multiples of 5 between 5 and 25
VL_CAPTURE_INTERLEAVED, VL_CAPTURE_EVEN_FIELDS, and VL_CAPTURE_ODD_FIELDS 525 All multiples of 5 and 6 between 5 and 30
625 All multiples of 5 between 5 and 25

With sirius video, this control is read-only. Its value is determined by the setting of the VL_TIMING control on the memory node:

VL_CAPTURE_NONINTERLEAVED 525 60 fields per second
625 50 fields per second

Vino's VL_RATE cannot be set to a value less than 5/1.

VL_ZOOM

The table of acceptable values for VL_ZOOM in the Guide is incorrect. Below is a corrected table of acceptable values for vid-to-mem applications. The only value of VL_ZOOM that works on all VL devices is 1/1.

Vino has hardware bugs that affect decimation of images at these decimation factors: 1/4, 1/5, 1/6, 1/7, and 1/8. These bugs include:

To work around these bugs, the vino driver implements decimation by 1/4 and 1/6 by decimating in hardware by 1/2 or 1/3, and then decimates by an additional factor of 1/2 in software. This produces acceptable looking images, but at significant cost in CPU time. The three other VL_ZOOM factors, 1/5, 1/7, and 1/8 all exhibit the green image problem described above.

Vino:
1/1, 1/2, 1/3Implemented in hardware, looks OK.
1/4, 1/6Implemented partially in hardware, partially in software. Looks OK, but slower and uses roughly 10% of an R4600 CPU.
1/5, 1/7, 1/8implemented in hardware. Looks bad. Green shift.

ev1:
1/1, 1/2, 1/4, 1/8Works OK for vid-to-mem
1/3, 1/5, 1/7Works only for vid-to-screen, not vid-to-mem, and only with VL_CAPTURE_INTERLEAVED.
2/1, 4/1Works only for vid-to-screen, not vid-to-mem.

sirius and ev3:
1/1sirius and ev3 don't zoom.

VL_SIZE and VL_OFFSET

There are two separate VL_SIZE controls that a vid-to-mem app must deal with. One is a control on the video node, the other is on the memory node. They have different meanings, different units, and do different things. The most common problem in discussing the VL_SIZE control is people forget to specify which one they're talking about, or are confused about that. This section discusses the VL_SIZE control and the VL_OFFSET control on the memory drain node.

The VL_SIZE control on the memory drain node determines the number of rows of pixels, and the number of pixels in each row, in each image buffer (field or frame) that the VL returns to the application. If zooming (decimation) is being done, the VL_SIZE control on the memory drain node specifies the size of the image after it has been decimated.

The VL_SIZE control on the memory drain node can be used to "clip" a region out of an image by setting the X and/or Y components to values that are smaller than the size of the captured (and decimated, if applicable) image.

When the (possibly decimated) image is being clipped, the clipped region does not have to come from the upper left hand corner of the (possibly decimated) source image. The VL_OFFSET control on the memory drain node determines the number of top pixel rows to skip and the number of leading pixels to skip in each row to find the first pixel in the (possibly decimated) image to place in the image buffer, the first pixel of the clipping region.

The Guide says, incorrectly, that "VL_OFFSET operates on the unzoomed image; it does not change if the zoom factor is changed.". The truth is that when zooming (decimation) is being used, VL_OFFSET is always in coordinates of the zoomed image. It is as if the entire source image is decimated down, and then the clipping function is applied to the decimated image. The hardware rarely works that way, and usually clips before decimating, but the VL API always specifies the VL_OFFSET in the coordinates of the decimated (virtual) image.

On all VL devices except sirius, the vertical (Y) component of VL_OFFSET may be specified with a negative value. This causes the clipping region to include row of samples taken before the top of the image, e.g. rows from the vertical blanking interval. This feature is usually used with VL_ZOOM of 1/1, since the information in the vertical blanking interval is generally destroyed by a zooming operation.

Because VL_OFFSET is useful for parsing VITC and other vertical interval information, it is useful to know which video line you get at the top of your F1 and F2 buffers for a particular Y value of VL_OFFSET, when you have selected VL_CAPTURE_NONINTERLEAVED and VL_ZOOM=1/1. A VL_OFFSET Y value of 0 gives you F1 and F2 buffers which begin on the following video lines:

Video Standardvino F1vino F2ev1 F1ev1 F2mvp F1mvp F2ev3 F1ev3 F2
525-line212842128321283??
625-line233362333623336??

This table uses the line numbers and definitions from Definitions: F1/F2, Interleave, Field Dominance, and More. Note that all the cases except for 525-line F2 fields with vino conform to the diagrams "analog 525, practical digital 525, analog 625, and digital 625" shown in that document.

On vino, ev1, mvp, and ev3, any other VL_OFFSET Y value (positive or negative) slides your buffer up or down the video signal by that many lines. The minimum allowed Y value is device-dependent, but is negative enough to pick up VITC on at least the higher of the recommended lines found in ANSI/SMPTE 12M.

On ev1, when you select VL_CAPTURE_NONINTERLEAVED, each of your buffers contains an F1 and an F2 field butted together. The Y coordinates of VL_OFFSET have strange units which behave like "picture lines" even though this doesn't make sense. Basically, if you would like to shift N lines, set VL_OFFSET's Y coordinate to 2*N.

This document does not cover what ev3's units are because we haven't tested it yet.

The VL imposes these requirements on the values of VL_OFFSET and VL_SIZE:

When an attempt to set either one of these controls would violate either of the rules above, the call to vlSetControl() fails with the vlErrno VLValueOutOfRange, and the offending component (horizontal or vertical) is set to the largest non-negative value that does not violate the rule, or to zero if no such non-negative value exists.

VL_OFFSET and VL_SIZE cannot be both set in one atomic operation. A change in either component of either control could violate one of the rules above (or below), especially after VL_ZOOM is set to a smaller fraction. It may be necessary to alternately and repeatedly set VL_OFFSET and VL_SIZE until no VLValueOutOfRange errors are reported. See vlSetControlLoop.txt for a code sample that does this.

Every VL video device places additional limitations on the range of acceptable values of VL_SIZE and VL_OFFSET. Each device has different limitations.

Hints about VL Events

VLStreamPreempted

If your application has setup a VL path with "shared" usage of controls and the data stream (e.g. vlSetupPaths(..., VL_SHARE, VL_SHARE), then whenever another application attempts to setup the same path as yours, or a path that shares nodes with your path, your program will be preempted. To handle preemption properly, your program must select VLStreamPreempted and VLStreamAvailable events in its call to vlSelectEvents().

Sirius doesn't support preemption. ev1 ellows preemption in some "sync modes" but not in others. When preemption is not allowed, the VL behaves as if the path was in mode VL_LOCK instead of VL_SHARE; preemptions simply don't happen.

When preemption occurs, your program will receive a VlStreamPreempted event. This means the following things:

No immediate action is required on the part of a program that receives a VLStreamPreempted event. The transfer has been ended, so you don't need to call vlEndTransfer(). The ring buffer in use by your path at the time of its preemption is neither deregistered nor destroyed by the preemption. There may still be one or more image buffers available. Calls to vlGetNextValid() or vlGetLatestValid() may continue to return previously unprocessed images. Non-NULL values returned from vlGetNextValid() or vlGetLatestValid() should not be construed to mean that the transfer is still in progress. It is probably a good idea to process and free all image buffers that can be obtained from calls to vlGetNextValid() or vlGetLatestValid().

VLStreamAvailable

When all the nodes of a path that is being held in VL_READ_ONLY mode are released, all programs holding the path in this mode will receive a VLStreamAvailable event. This will happen when the program that was last using the path called vlDestroyPath(), or released the path by setting the stream usage to VL_READ_ONLY or VL_DONE_USING in a call to vlSetupPaths().

The VLStreamAvailable events create a race between the preempted programs. The outcome of the race depends on the actions taken by the programs, and also on the device's implementation. Generally, as long as all the programs are attempting shared use of the path, the last caller of vlSetupPaths() wins.

When a program receives the VLStreamAvailable event, to regain the usage of the path's data stream, it should call vlSetupPaths(..., VL_SHARE, VL_SHARE) again. This call to vlSetupPaths() to regain access to the path's data stream can succeed or it can fail. The VLStreamAvailable event does not guarantee that it will succeed. Let's look at these cases separately.

If the call to vlSetupPaths() succeeds, the application should set all the controls for the path to the values they had before it was preempted. Your application, that is now regaining the use of the stream, should not assume that the controls are as they were before the preemption. The preemption caused the usage of the path's controls to become VL_SHARE, so the preempting application could have changed any or all of those controls.

Since preemption does not deregister or destroy the ring buffer used by your path, you may resume usage of that same buffer, provided that all the path's controls are returned to the same values they had before the preemption (especially VL_SIZE and VL_PACKING). However, you might want to be doubly sure that the ring buffer is empty before restarting the transfer with a call to vlBeginTransfer().

If the call to vlSetupPaths() to regain access to the path's data stream fails (returns a negative value), the path should remain in the same mode in which it was before the failed call to vlSetupPaths(). If your program hasn't changed the path since it was preempted, and if the call to vlSetupPaths() failed with vl error VLPathInUse, the path should still be in the mode in which it was placed by the original preemption, namely as if vlSetupPaths(..., VL_SHARE, VL_READ_ONLY) had been done on this path. The path should continue to behave exactly as when it was preempted, and another VLStreamAvailable should be generated, if and when the path becomes available again.

VLDefaultSource

Each video device may have multiple video source nodes, each of which serves one or more physical input jacks on the video board. The VL supports the concept of a "default input source node" or "default video source node" for each device. There is no system-wide default input source node, nor is there a default input device. An application on a system with multiple video devices must choose one of the devices according to its own methods.

A VL application that wishes to use the default input source node on a given device calls vlGetNode(vlServer, VL_SRC, VL_VIDEO, VL_ANY).

Each device's default input source node may be changed via a control. The Video Control Panel supports this, and is usually used to make this choice. When the default input source node on any VL video device is changed, (e.g. between a digital input source node and an analog input source node), the VL sends a VLDefaultSource event to all programs that have selected this event, regardless of whether they are using the device whose default input changed, or another VL device.

Each video source node may serve multiple jacks. The choice of active input jack on a given source node is done with some VL controls, VL_FORMAT or VL_MUXSWITCH, as described in Choosing an Input Jack. A change in input jack between jacks connected to the same video source node usually results in a VLControlChanged event, not a VLDefaultSource event.

If you wish your application to track the default input node, you must manually tear down and rebuild your video path each time you receive a VLDefaultSource event, like this:

If you wish to track VL_FORMAT and VL_MUXSWITCH controls on a given node you are using, this may or may not happen automatically depending on the device. See Choosing an Input Jack.

Other Miscellany

More About vlSetupPaths()

If a first attempt to gain access to a path via vlSetupPaths(..., VL_SHARE, VL_SHARE) fails, and vlGetErrno() returns VLPathInUse, then it should be possible to call vlSetupPaths(..., VL_SHARE, VL_READ_ONLY) to put the path in the same state as it would be if it had been preempted. Likewise, a program that wishes to voluntarily release the path, and then later regain it, as if it had been preempted should be able to put the path info VL_READ_ONLY mode.

While the path's stream usage is in VL_READ_ONLY mode, a VLStreamAvailable event should be generated when the data stream for the selected path becomes available. That is, a program should be able to put itself into a "voluntarily preempted" mode by putting the path's stream usage is in VL_READ_ONLY mode. Whether this actually works or not depends on the device, and on the OS release. It has worked on ev1 at one time. sirius does not support putting a path into VL_READ_ONLY mode. sirius also does not support preemption.

An application should be able to transition the stream usage of a path from any one to any other of the four modes (VL_SHARE, VL_READ_ONLY, VL_LOCK, VL_DONE_USING) as many times as it wishes. VL_DONE_USING is not the "mode of no return". The principal difference between VL_READ_ONLY and VL_DONE_USING seems to be the receipt of VLStreamAvailable events in the VL_READ_ONLY mode.

Since sirius doesn't support VL_READ_ONLY, an application cannot use VL_READ_ONLY to put itself into a mode where it will be notified when another device has finished with the path. A sirius app has no choice but to go to VL_DONE_USING mode when relenquishing the path, and to use some external or manual means to know when to attempt to reaquire the path.

Premption vs. Cooperative Management

Since pre-emption and VL_READ_ONLY modes don't work on all platforms, two or more programs that wish to cooperatively share the use of a VL device might be better off to use their own protocol to pass ownership of the device between them. The programs in the InPerson product used this approach succesfully.

IndyCam on ev1 Video Devices

IndyCam works on many of the boards in the ev1 family, as explained in Taxonomy of the ev1 and cosmo1 Boards. The images produced by IndyCam on ev1 tend to be very noisy in low light, much more so than the images produced by IndyCam on vino. This noise is seen as vertical and horizontal lines of gray in the image. Since the output of IndyCam is digital, this noise is probably induced via the power supplied to the IndyCam by the video board.

Additional Disclaimers:

This product is intended for educational purposes only. Not responsible for direct, indirect, incidental or consequential damages resulting from any defect, error, or failure to perform. NO OTHER WARRANTY EXPRESSED OR IMPLIED. Reproduction strictly prohibited. The trademarks mentioned in this product appear for identification purposes only. Your mileage may vary. Not for resale. Any resemblance to real persons, living or dead, is purely coincidental. Void where prohibited. This is not an offer to sell securities. Some assembly required. Batteries not included. Not to be taken internally. Contents may settle during shipment. Freshest if eaten before date on carton. Use only as directed. Avoid contact with skin. Apply only to affected areas. Use only in well-ventilated area. Keep away from fire or flame. Do not use while operating a motor vehicle or heavy equipment. If condition persists, contact your physician. Contains a substantial amount of non-tobacco ingredients. Not recommended for children under five. May be too intense for some viewers. No passes accepted for this engagement. All models over 18 years of age. Times approximate. Simulated picture. Edited for television. Processed at location stamped in code at top of carton. No user-serviceable parts inside. Subject to change without notice. For recreational use only. For off-road use only. Subject to approval. One size fits all. Colors may fade. See label for washing instructions. For office use only. Post office will not deliver without postage. Penalty for private use. Substantial penalty for early withdrawl. Nor for removal except by the consumer. List was current at time of printing. Lost ticket pays maximum rate. At participating locations only. Your canceled check is your receipt. Slightly higher west of the Mississippi. Employees and their families not eligible. You must be present to win. Contestants have been briefed on some questions before the show. Limited time offer. No purchase necessary. Price does not include sales tax. Slippery when wet. Breaking seal constitutes acceptance of agreement. Some equipment shown is optional. Replace with same type. No anchovies unless otherwise specified. Driver does not carry cash.