Video objects

 Top  Previous  Next

Whisker version required

 

Video support requires WhiskerServer v4.0 (summer 2011).

 

In brief, Whisker plays videos using DirectShow. This means it supports any video filetype for which a decoder is available and registered with DirectShow (with a very few caveats as below). WMV9 (Windows Media Video 9) is supported as the default filetype.

 

Like any other object, video objects are placed in display documents, which may then be shown on display devices.

 

To play audio tracks from a video file, the display device must additionally be associated with a (claimed) audio device.

 

Caveats

 

There are some important caveats, as follows.

 

•Video can be CPU-intensive. Electing to play audio adds to the CPU load.
•Flexibility/speed tradeoff, configured in advance. Because the server may have to show multiple synchronized copies of a given video (e.g. the subject's view and the server copy, and/or multiple subject views), and because it must be possible to add and remove views without pausing the video, even briefly, the server uses a system that requires it to know in advance the maximum number of possible audio and/or video "views" of the video file. This must be specified in Server – Video configuration. Unused potential connections use some CPU time; choose the minimum possible value. Because of these limits, DisplayAddObject (option: video) and DisplayShowDocument can now fail, if they would add too many video views.
•Timing. Internally, for each video object, the server has a single "source" filtergraph and potentially multiple "render" filtergraphs (whose maximum numbers must be known when the video objects are created). When a renderer is active, the source graph is bridged to the relevant render graphs. Note that (1) the synchronization between the subject's view and the server view is not guaranteed to be perfect; (2) when a video timestamp is requested, it is taken from the source graph; (3) the DirectShow and bridging systems do several things to try to ensure accurate synchronization between the notional timestamp and the audio/video renderers, but this is not an absolute guarantee.
•Video is incompatible (in its current incarnation) with DirectShow backbuffering; therefore, in Configure – Display Devices, do NOT tick "Use DirectDraw back buffer" if you wish to use video. (If you don't, go ahead and tick it, as it improves static video performance.)
•Audio streams from video devices must use DirectDraw-supported sound devices (so they won't work on the "Primary Sound Driver" or "fake" audio devices). Prior to Whisker v4.6, they could not be confined to just one audio channel, so could not be used with left/right "split" devices (see Configure – Audio Devices); this limitation is removed as of Whisker v4.6 (Dec 2014).
•Video objects are always "topmost". Don't try to place anything on top of video objects; they will be forced down (and if you try to overlap two videos, they will flicker as they each try to go topmost).

 

More on supported video file types

 

 THESE WORK:

   - video codec type Windows Media Video 9 (and audio codec type Windows Media Audio 9.2)

     ... known as WMV9

     ... uses the "WMVideo Decoder DMO" filter and works

   - things encoded with video codec "wmv2" and audio codec "wmav2" by ffmpeg (see below)

 SOME FILES WON'T LOAD, and say so. Convert them (see below).

 SOMETIMES AUDIO FAILS:

   - in which case the server log might say e.g. "Couldn't use bridge to create source graph - DirectX error in CSourceGraph::CSourceGraph: Cannot play back the audio stream: the audio format is not supported."

   - use the -noaudio flag in this case

 SOME FILES LOAD AND DON'T COMPLAIN BUT VIDEO PLAYS AS A BLANK SCREEN.

   - example: video codec type Windows Media Screen V7 (and audio codec type Windows Media Audio V8)

     ... fail because the VMR9 renderer insists on a colour space converter when used

                 from an infinite tee, and the "WMV Screen decoder DMO" filter used by this format

                 won't play through a CSC (as tested in GraphBuilder),

                 unlike the "WMVideo Decoder DMO" filter

 TO INSPECT WMV VIDEO CODE TYPES:

   - run Windows Media Player; load file; then File > Properties

 GENERAL PROCEDURE TO MAKE SOMETHING WORK:

   - Install ffmpeg.

     On a Ubuntu/Debian Linux box: sudo apt-get install ffmpeg

     On a Windows box: see http://www.ffmpeg.org/

   - ffmpeg -i INPUTVIDEO.XXX -vcodec wmv2 -acodec wmav2 OUTPUT.wmv

   - This also allows conversion from FLV, MPEG, etc.

   - Or with the newer avconv system:

    avconv -i INPUTVIDEO.XXX -c:v wmv2 -c:a wmav2 OUTPUT.wmv

   - In the case of a video with no audio, or to remove audio, you can do:

    avconv -i INPUTVIDEO.XXX -c:v wmv2 -an OUTPUT.wmv

 

Video configuration on the server

 

See Server – Video configuration

 

Video commands

 

Video objects are controlled using these conventional object-manipulation commands:

 

•DisplayAddObject (option: video)
•DisplayDeleteObject
•DisplaySetEvent
•DisplayClearEvent
•DisplaySetEventTransparency
•DisplayEventCoords
•DisplayBringToFront
•DisplaySendToBack

 

... and these video-specific commands:

 

•DisplaySetAudioDevice
•VideoPlay
•VideoPause
•VideoStop
•VideoSeekAbsolute
•VideoSeekRelative
•VideoGetTime
•VideoGetDuration
•VideoTimestamps
•VideoSetVolume