iFACE
Interactive Face Animation - Comprehensive Environment

Version 1.99

Interactive Media Technologies Inc. (iMediaTek)

iFACE Framework
    Installation
    Application Development
    Using iFaceStudio
    Configuration Files
Architecture Description
Geometry Hierarchy
iFACE Parameters
iFaceStudio User's Guide
 

iFACE Framework and iFaceStudio Application


Installation
Create the folder C:\iFACE, unzip the installation file in atemporary directory, and copy the unzipped content to C:\iFACE. Make sure that C:\iFACE includes files such as iFaceStudio.exe. Using another location is possible but may need modification of the provided data files. 

Run TestDotNet.exe (shows a dialog box titled Test .NET) and TestD3D.exe (shows a 3D teapot) to test if .NET framework and Managed DirectX are installed in your system. If they are not, or you receive a runtime error and can not run iFaceStudio.exe, try installing .NET framework and Managed DirectX v9.0c. 

    You can download installation files for .NET and DirectX from Microsoft or iFACE web sites.

Before running iFaceStudio, register the files ETCodeX.dll and SpeeX.dll using this command:
    regsvr32 <DLL-filename>
or run RegisterDLLs.bat.

Application Development
To develop new applications with iFACE framework, you can:
Add Wrappers.dll to a C# project References, Add Face3DControl to the Toolbox (Tools/Customize Toolbox/.NET), and insert an instance of the control to a form. Or
Add Helpers.dll, Geometry.dll, Behaviour.dll, and Speech.dll to the project References, and derive the form from Face3DForm. The necessary iFace namespaces must be added to the source code:
using iFace.Geometry;
using iFace.Behaviour;
using FaceHelpers;
using D3DHelpers;
using GeneralHelpers;
using SoundHelpers; 

Using iFaceStudio
iFaceStudio is authoring and testing app for iFACE. It includes two main 3D (Mesh) and 2D (Imager) windows, and the Control panel with following tabs:
iFACE , general iFACE functionality such as Animation and Defining KeyFrames.
Geometry , components and features in geometry hierarchy and their parameters.
MPEG-4 , access to MPEG-4 Face Definition Parameters (FDPs) and Face Animation Parameters (FAPs) .
Mesh Setup , load/save mesh objects and define their regions. 
Imager Setup , load/save image objects and define their regions.
View/Tools, set viewing properties of iFaceStudio and access general tools

Animation types are:
FML, XML-based scripting language (see FML v2.0 Specification )
KeyFrame, as defined by iFaceStudio
Wave, either a wave file (Playback mode) or real-time audio (Record mode)

Both Playback and Record mode can use local wave files (to read from or write to) or a remote instance of iFaceStudio in the opposite mode. To specify a remore operation following format is necesary for file name:
mahine>port, where machine is IP address or computer name separated from port number by >.

For Playback the file (or remote address) must be Opened first. For Record, the Audio Recording File button must be used to set the target.

Creating a KeyFrame file starts and end by Define button in Keyframe group. Each key frame is then added after proper head movements are done in MPEG-4 tab.

Typical behaviours are defined as Keyframe files. Each personality has an entry file with extension .ptype in Behaviours folder and a corresponding sub-folder including two groups of keyframe files for impulse and emphasis. The current personality is set by Personality button in iFACE tab.

See iFaceStudio User's Guide for details on commands.

Configuration Files
The iFACE framework is configured through the following file types (specified with their extensions):
CHR (Character): Text file that contains the names of geometry and personality files to create a character
MSH (Mesh): Binary file that is generated by iFaceStudio and contains data for a HeadMesh object (this includes the name of Direct3D .X mesh file)
IMG (Image): Binary file generated by iFaceStudio for HeadImage object (when using 2D faces)
CMP (Component): Binary file for initializing the components
PTYPE (Personality Type): Text file that defines a personality through parameters and typical behaviours stored in keyframe files
X (DirectX Mesh): Standard text file that contains Direct3D mesh information. This file includes path to an image file that is used for texture.
KYF (Keyframe): Binary files that define key frames for a facial behaviour
DEF (Definition, optional): Binary or Text file for extra head definitions and new texture applied to the base mesh

The files do not need to be in any specific folder, but by default they are all in one main directory except PTYPE files which are in Behaviours subdirectory. Default files provided at installation assume these addresses. It is possible to edit the text files for new addresses or run the program and set new addresses for files. It is always possible to use . meaning the current directory (of the file that has the address in it) to write relative file paths, as done in default DirectX mesh file (Face.x). The address of texture file in .X file is written as .\\Face.bmp meaning the same directory as .X.

CHR file has following format:
*msh (or img or def or cmp or ptype)
{file path}

The file path can be written using .\ meaning the same directory as .CHR file. Here (unlike the .X file) one \ is enough.

 

iFACE Architecture Description

The iFACE framework consists of following namespaces:
FaceHelpers , for all layers
iFace.Geometry , for Data layer (see geometry hierarchy section )
iFace.Behavior , for Data layer
iFace.Stream , for Streaming layer
iFace.Wrap , for Wrapper layer
and Application-specific namespaces for Application layer

Each namespace provides classes for the related funcionality/layer in iFACE environment. 

Helpers
Main classes are Face2DForm (derived from ImageForm) and Face3DForm (from D3DForm). They use HeadImage (from AppImage) and HeadMesh (from AppMesh) classes to encapsulate actual head data. D3DForm (derived from GraphicsSample) uses AppMesh to manage mesh data with regions. ImageForm (from Form) uses AppImage for that purpose. These two classes have nothing to do specifically with Face.
None of the form classes in Helpers namespace are to be used directly. They do not have the actual graphic object or any other user interface controls. As seen in sample applications (e.g. iFaceStudio ), form classes are tobe derived from them, with proper controls such as image panel.
AppMesh (and HeadMesh) are serialized into/from .msh files. AppImage (and HeadImage) are serialized into/from .img files.

HeadMesh and HeadImage implement and expose IHeadData interface. HeadData objects (HeadMesh and HeadImage) include not only regions but also FaceParams object.
FaceParams has members for FDP and FAP. FDPs define the topology (feature pointa) and are serialized (i.e. saved). FAPs are used in runtime for animation, and are not serialized. FAPs in group-1 (viseme and expression) are combination of other FAPs. This combination is serialized. 

Geometry
The Geometry module controls the HeadData objects by accessing IHeadData interface (independent of 2D or 3D). Geometry itself is accessed using IGeometry, IHead and other interfaces. Head object is the main object in Geometry that implements and exposes these interfaces. Other objects are created for lower-level functionality, with their own interfaces, e.g. IMouth.

Head object also implements and exposes IAnimation for animation control. It creates default speech controller object that implements and exposes the ISpeech interface. Speech module is part of Gemetry namespace but is implemented as a separate DLL to allow defining other speech controllers. Default speech controller uses ETCodeX ActiveX , a non-managed code. Application can create Geometry/Head object with default animation and speech controllers or create new objects that implement IAnimation or ISpeech and pass them to Head

Behavior
The Behavior module includes three components each with its own interface: Knowledge, Personality and Mood. By default they are all implemented in one object called Actor, but external objects can be implemented to support and expose the same interfaces. 

IKnowledge interface controls the main actions performed during the animation. It is responsible for processing the input scripts such as Face Modeling Language documents and sending proper commands to Geometry or other Behavior components. IPersonality and IMood interfaces are related to long-term and short-term behavioral patterns. 

Geometry Hierrarchy

The actual head data is stored in an object called HeadData tht includes Points (vertex or pixel) and Regions (i.e. small groups of points that usually move together). Regions usually do not overlap but they can depending on design. Geometry object is a hierarchy of abstraction layers on top of HeadData. These layers are:
Head, i.e. whole head
Components, including functional groups i.e. Eyes, Ears, Brows, Nose, Mouth, and Base (the rest of the head)
Features, i.e. detail lines around main parts of Components
FeaturePoints, MPEG-4 FDPs
PhysicalPoints, actual HeadData points (Vertex or Pixel)

Each one of the objects in these layers have Regions of Influence (ROI), i.e. HeadData Regions affected by the object. In case of Components and Features, this is a group of HeadData Regions. Different Components can share some of their ROIs. For each FeaturePoint there is only one ROI, i.e. each FeaturePoint is associated with one HeadData Region. Components break a requested action into proper movements of FeaturePoints or move their ROIs directly. FeaturePoints move the points on their ROI proportional to the inverse of distance.

Head actions (moving and resizing) are defined for Components or Features which in turn affect the related FeaturePoints and their Regions. All action requests (params) are primarily sent to Head. Application can ask Head to create and use lower level objects if necessary, e.g. use a Component for actions with wide area of influence or send the action request directly to FeaturePoint.

iFACE Parameters

iFACE is a parameterized system. iFACE parameters have up to three numeric and one text data, plus following attributes:
Type
Group
Param Number
SubParam

iFACE parameter types can be:
FDP: MPEG-4 FDPs are from Group 2 to 10. Param Numbers are feature point numbers. SubParams control the direction:
    1 - X
    2 - Y
    3 - Z
FAP: MPEG-4 FAPs are from Group 1 to 10. Param Numbers are index within Group. Group 1 FAPs have two possible Param Number: 1 for Visems and 2 for Expressions, and use SubParam.
Component: Head Component Parameters are grouped as follows:
    1 - Whole Head
    2- Left Brow
    3 - Right Brow
    4 - Left Eye
    5 - Right Eye
    6 - Nose
    7 - Mouth
    8 - Left Ear
    9 - Right Ear
    10- Neck

Param Numbers 1 and 2 are Move and Scale. 3 to 5 are different for each Group. SubParams control the direction:
    1 - X
    2 - Y
    3 - Z
rFace: Legacy rFace Parameters.
Extra: Other Parameters:
    Group-1 (no subparam)
    1 - Texture (filename is the text value, material is subparam)
    2 - Open/Play (filename text value, value1 if nonzero is fps for play-to-file)
    3 - Record (no value)
    4 - Reset Geometry (no value, subparam 1 reset all, subparam 2 reset mood)
    5- Ambient Color (RGBA text value)
    6- Background Color (RGBA text value)
    7- Open personality type (filename text value)
    8- Open geometry (filename text value, .msh or .x for 3D, .img or .bmp or .jpg for 2D)
    9- RecordFile (filename text value)
    10- Background audio (filename text value)
    11- MakeMovie (filename text value)

    Group-2
    1- Camera ViewFrom Point (subparams 1-3 for xyz)
    2- Camera LookAt Point (subparams 1-3 for xyz)
    Group-3
    1- Light Source X-Direction (subparam is source index)
    2- Light Source Y-Direction (subparam is source index)   
    3- Light Source X-Direction (subparam is source index)    
    4- Light Source Red Color (subparam is source index)
    5- Light Source Green Color (subparam is source index)   
    6- Light Source Blue Color (subparam is source index)   

iFACE parameters can work in absolute and relative (delta) value modes. When creating a Keyframe file, all the parameters are assumed to have the same mode. When creating a delta KYF file, the design is done with normal (absolute) parameters (i.e. absolute values are used for all KYF params) but the Delta option is selected for params. This makes iFACE animation engine to convert the param values to delta before playing the KYF. This way more than one KYF can be safely animated together.

iFaceStudio User's Guide

iFaceStudio has three main windows:
Face3D
FaceImager
FaceControls

FaceControls has the detail UI objects to control the application. Face3D shows 3D head and FaceImager shows the texture. Application starts by locating a .CHR file as the main database. CHR files include address of other data files required for operation. Advanced Options in View/Tool tab of Face Controls allow selecting a different CHR file.

1- FACE-3D

This window shows the 3D results and allow moving the head around to see the other sides. X axis is horzontal (positive right), Y axis (positive up) is vertocal, and Z axis is in right angle to image plane (positive in). Defining regions, and selection of FDPs are also done in this window. Face3D has following local controls on top:

**Distance, to change the the Z coordinate of camera with a negative value.
**Vertex Select checkbox to go to vertex selection mode (you can select vertices and see their info)
**Wireframe checkbox to switch between full-render and wireframe modes
**Controls and Imager checkboxes to show/hide these windows
**Select Vertex button to select a vertex by entering its index
**Edit Vertices button to edit the vertices in a selected region

Selecting vertices (for viewing only or for editing regions) can be done in SelectVertex mode (checkbox) by SelectVertex button or by left-clicking. Left-click intersects the vector with the head and locates the poly. On that the closest vertex is selected. Top of the window shows the index of selected vertex. Righ-clicking will deselct the vertex. Some vertices share the same location but different polygons. Clicking on different polygons will select different vertices, in such cases.

2- FACE-IMAGER

This window shows the texture. The local controls are for zooming, enabling Pan mode (move the image by mouse-dragging) and showing Regions.

 

3- FACE-CONTROLS

This is the main window for controlling the iFaceStudio Application. It has following tabs:

3-1- iFACE

General controls can be found here.

Personality
**Set selects active personality by selecting a .ptype file (usually located in Behaviours subdirectory)
**An edit box shows the current personality

KeyFrame
There are two buttons in this group, each with two possible labels: Define/End and OneShot/AddFrame.

**Define starts/ends definition of KeyFrame file. Its label changes to End after this. The user changes the parameters to get the desired key frame and then uses the other button to add this frame to file. When done with frames, clicking on this button end the file by selecting a name.
**One-Shot creates a keyframe file containing only one frame based on current parameter setting

Animation
**Filename desplay currently selected file
**Open button opens a new file (or currently selected file) to play
**Play starts playback
**Stop ends the playback
**Record starts recording audio
**CaptureFile selects a file that audio is recorded to or animation is saved to if Save is selected
**Capture checkbox determines if data is saved to file
**Audio selects a background audio for the animation (must be selcted before opening the anim file)

To save the video frames to file, open the animations file, select a capture file (new BMP or JPG file), select Capture checkbox, change the frame rate if necessary, and start the playback. The frames will be saved by adding a 5-digit number to the file name you specifed.
Use MovieMaker.exe program (accessible in View/Tools tab) to create a video file from images and add optional audio. The program is a console application priovided with iFACE executables.


General
**Exit terminates the program
**FML Command Line brings up a window to enter FML commands. For example following command is used to apply an rFace parm 2 with value of 1 (limited support)
rfp type="2" value="1"

 

3-2- Geometry

This tab gives access to component-level parameters

Components
**Select determines the currently selected component from list
**New initializes the component settings
**Load opens a new component file
**Save and SaveAs store the changes to component file
**View/Edit Features Line give access to selected feture line of the selected component

Component Parameters
**Param
**SubParam
**Value
**Add defines a new component parameter using a KYF file
**Remove deletes an existing parameter

Definition
**Base resets the geometry to original .x file
**Load selects a definition file that modifies the base geometry
**Save and SaveAs store the changes to definition

To define a new geometry based on an existing .X file, use direct FDP movement or components and save the new shape in a new DEF file. Then add that to CHR file:
*def
.\face.def

 

3-3- MPEG-4

This tab is for MPEG-4 FDPs and FAPs . Each parameter has group, param, and subparam values. For example the expression Joy is a FAP in group 1 with param value of 2 (expressions) and subparam value of 1.

Select

To select a parameter and select regions if the param is an FDP.

Locate
**SetFAPs/FromMesh (the label depends on the type of param) sets the lower level FAPs based on current settings for a group-1 FAP or locate the vertex for an FDP (select the FDP, select the vertex, and then click on this)
**Load sets the param settings for current selection from file
**Save stores the param settings to file

The dynamic value of FAPs are not saved.

Group-1 FAPs are defined as a combination of others. To define a group-1 FAP (viseme or expression), set all other FAPs to achive desired look, then select the group-1 FAP and click on Set FAPs. You can simplify this by seting as many facial states using FAPs, saving them with Snapshot command in iFACE tab, bringing them up by Open/Play commands in iFACE tab, comparing them and clicking on Set FAPs for any one that looks better.

Move/Resize

Controls the value of params and can reset them all to initial values.

 

3-4- 3D Setup

This tab is for 3D settings. This includes the base mesh (.x file) and the regions. The info is saved in a .msh file.

Mesh
**New creates a new .msh file
**Load opens an existing .msh file
**Save and SaveAs store the .msh file
**Load Texture uses a new texture
**Save.X stores the current setting into .x file (no regions)
**SaveSnapshot captures the 3D screen and saves to a bitmap file.  

Regions
**Combo box selects the region
**Define starts definitionof a new region
**Delete removes currently selected region
**Edit modifies the currently selected region
**View shows the currently selected region
**ReadFromBitmap reads an image based on texture file with regions specified by color and determines the regions based on that
**Export saves the regions in an ASCII file
**Import reads regions from an ASCII file

3-5- Imager Setup

This tab is for 2D settings. This includes the base image and the regions. The info is saved in a .img file.

Image
**New creates a new .img file
**Load opens an existing .img file
**Save and SaveAs store the .img file
**Reload Bitmap resets the image
**SaveBitmap stores the current bitmap
**Symmetric copies selected region or all of them to the symmetrically located region. If we have a half-texture, we define, for instance, right-ear and copy it to left-ear

Regions
**Combo box selects the region
**Define starts definitionof a new region
**Delete removes currently selected region
**Edit modifies the currently selected region
**View shows the currently selected region
**SelectedRegionsTo3D exports selected region to 3D
**Export saves the regions in an ASCII file
**Import reads regions from an ASCII file

3-6- View/Tools

These are some general controls for viewing and special tools. Some important ones are explained here.

Advanced Options
**Select CHR open a new character file
**Lights control up to 6 directional light source