Hi !
Featured games (90)
games approved by the League of Dukes
Games in Showcase (748)
Games in Android Showcase (226)
games submitted by our members
Games in WIP (834)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  LipSync Demo  (Read 3512 times)
0 Members and 1 Guest are viewing this topic.
Offline zero

Junior Devvie

« Posted 2006-03-02 08:34:19 »

Hey guys,

I have put a lip synchronization demo online - not a game, but may be interesting to some of you:

  • Sometimes, syncronization, does not work the first time starting  (maybe has todo with the webstart cache)
  • Sorry Solaris users, I only put Linux, MacOS X and Windows jogl libraries online.
  • I noticed that there is an Xlib problem with Java 6 on linux (I'll post details into the jogl forum, after a get back home to my notebook)
  • The performance is very bad, because:
  • The complete Model (15k triangles) is rendered, including on-fly computation of all normals, although only the head is shown and animated.
  • I had to disable back-face culling and enable two-sided-lighting, since the face ordering from the maya model, which I have exported, is not consistent
  • Java Client VM is used, since -server flag seems not to be available on all runtimes
[li]Stay tuned, I hope to finish gestures (skeletal animation) at the week-end[/li]

Offline shawnkendall

Senior Devvie

« Reply #1 - Posted 2006-07-12 12:32:23 »

Ok, We want/need this tech.

We work only in Java3D though, but it uses vertex arrays the same as any (JOGL), so we would be willing to do porting to Java3D or whatever.
Thanks for any information.

Shawn Kendall
Cosmic Interactive, LLC
Offline zero

Junior Devvie

« Reply #2 - Posted 2006-07-12 16:41:07 »

hey shawn,

I'm glad that your are interested in the tech.  Smiley
Be sure not to miss the video showing the complete application.

I guess, everything should be possible in Java3D, although not as fast as in OpenGL: Even if Java3D's Geometry uses OpenGL 2.0's VertexBufferObjects, it is not possible to declare some components static and other dynamic, at least AFAIK. .The positions and normals have to be updated every frame, but the texture-coordinates don't vary and therefore one should send them to graphics-memory only once. - anyway, I guess this is unlikely to be bottleneck.

I plan to release the full source-code in September, because I'm busy now preparing stuff for a conference in late August. Unfortunately, the realese will be without the models, since I have no rights on them and my sensei from Tokyo won't agree. The Maya exporter is up-2-date, so you could use it, the 3dsmax exporter is behind the last version, which makes it unusable at the moment. In Septmeber, I'll wirte a converter for COLLADA 1.4.1 so everyone can export own models from most DCC-Tools.
If you're in a hurry, I could give you the fast hacked, undocumented source code now, but I guess it'll be a great effort to go through that!

As you are particular interested in the lip-synchronization, I will explain a few things:

1. We use a third-party tool: Annosoft's LipsyncTool (costs: ~500$). This outputs a time-schme for phonemes and their intensity from an audio file (wav,mp3) and the spoken text (optional) to a simple text-format.

2. A self-written tool converts such a file into our keyframe-based animation format. The file used int the demo is email.anim.xml.

3. Our characters have defined some facial features via Morph-Targets(3dsmax) / BlendShapes(Maya), e.g. LowerLipUp. Small offset vectors determine the offsets for each group of vertices, e.g. some around the lip. The animation parameter for one is simply an intensity, typically ranging from 0.0-1.0.

4. Since we wanted to be independent of the facial features defined for a character (some of our models use only about 20 ,others the full range defined by MPEG4's Facial Definition Parameters), we introduce a mapping between them and the phonems. More exactly a two way approach: from phonemes to visemes and from visemes to facial features. This is defined once per character in an expression file: Tsumabuki.expr.xml. In order to get good values, we simply moved the sliders in Maya/3DSMAX until the mouth looked like a particular viseme (the visual correspondence to a phoneme).

Note: We plan to remove step 1-2 by graping the phoneme-information directly from a TTS-system in real-time.

Please ask if you have an further questions.
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline shawnkendall

Senior Devvie

« Reply #3 - Posted 2006-07-12 17:05:34 »

My artists could make you some new example content (for Sept) that you could distrbute if you are interested.

Shawn Kendall
Cosmic Interactive, LLC
Offline zero

Junior Devvie

« Reply #4 - Posted 2006-07-12 18:42:54 »

That sounds great! In fact the absense of available characters is the main resean why I didn't put much effort in publishing a full demo with source code.
AFAIK, there is one free available for blender (with morph-targets and skeleton), but the blender COLLADA exporter isn't capable of it yet. For 3DSMAX and Maya I have my own exporters and the COLLADA exporters should also be able to handle all.

Btw. the parsers for my formats are graphics-api indenpendent, the same is true for the internal structures. Only the rendering classes contain OpenGL-code, so porting to Java3D shouldn't be that much work...

Oh, before I forget: I don't know whether this is interesting to you, because it is more an overview instead of an in-depth explaination, but there is an online article available about some work related to characters I did that time: Creating Three-Dimensional Animated Characters: An Experience Report and Recommendations of Good Practice.
Pages: [1]
  ignore  |  Print  

xxMrPHDxx (305 views)
2017-12-31 17:17:51

xxMrPHDxx (93 views)
2017-12-31 17:15:51

xxMrPHDxx (146 views)
2017-12-28 18:11:33

Ecumene (310 views)
2017-09-30 02:57:34

theagentd (492 views)
2017-09-26 18:23:31

cybrmynd (521 views)
2017-08-02 12:28:51

cybrmynd (463 views)
2017-08-02 12:19:43

cybrmynd (446 views)
2017-08-02 12:18:09

Sralse (405 views)
2017-07-25 17:13:48

Archive (1312 views)
2017-04-27 17:45:51
Java Gaming Resources
by philfrei
2017-12-05 19:38:37

Java Gaming Resources
by philfrei
2017-12-05 19:37:39

Java Gaming Resources
by philfrei
2017-12-05 19:36:10

Java Gaming Resources
by philfrei
2017-12-05 19:33:10

List of Learning Resources
by elect
2017-03-13 14:05:44

List of Learning Resources
by elect
2017-03-13 14:04:45

SF/X Libraries
by philfrei
2017-03-02 08:45:19

SF/X Libraries
by philfrei
2017-03-02 08:44:05 is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!