Proof of concept for lipsynced speech and enhanced w2scene encoder features

+
Proof of concept for lipsynced speech and enhanced w2scene encoder features

Hello everybody,

a small showcase vid of coming upgrades:


Scene uses (advanced) features from next version of the w2scene encoder. Still no encoder gui but a *supporting* storyboard ui w3 mod for creating a *rough* draft of scene/scene section will be available, too:



All lipsync is created directly from the audio (but timings were manually adjusted). Audio was converted to WEM with free version of wwise and packed with (yet unpublished) w3speech un/packer.


All tools will be published on nexusmods next week.


Thx erxv for hosting and Dahlialynn for (overly) nitpicking all sort of stuff ;)

End Music is a snippet from the track "Geralt of Rivia" by Marcin Przybyłowicz from the "The Witcher 3: Wild Hunt - Official Soundtrack".
 

erxv

Forum veteran
spencerolmos;n7006910 said:
Oh, my god this is so awesome! Could this make custom cutscenes? Holgar82 perhaps lift some iorveth dialogue if custom cutscenes are possible?

yes, you can make custom scenes, the main video is completely made with the encoder. obviously you use existing animations but thankfully there are lots to choose from.
 
that's amazing. I gave up any hope on quest mods without the redkit but you brought back the hope.
 
That's really awesome. :cheers:

spencerolmos;n7006910 said:
Oh, my god this is so awesome! Could this make custom cutscenes? Holgar82 perhaps lift some iorveth dialogue if custom cutscenes are possible?

Well i guess everyone will be able to make content about it, but ye i'd like to create something aswell when this will be released.
 
Looks awesome, cant wait to try it out my self.

Quick question, how many mods will I have to jugle to make a short video, like the one you showcased?
 
That is great work rmemr and I think I can speak for everyone if I say that your effort and commitment is hughly appreciated.
I am looking forward to the release.

and of course: keep up the good work
 
this work it really beyond my imagination . not even dream that some one can make this kind of work really want to try this one
 
Thank you all for the positive feedback!

soulfax444;n7010760 said:
Looks awesome, cant wait to try it out my self.

Quick question, how many mods will I have to jugle to make a short video, like the one you showcased?

The YT description did not contain much information and some news sites didn't find their way to this thread in the forums. Hence there is some confusing and wrong information about the involved tools in the wild. So to answer your question I'll expand a little on the tools and the creation process.

In addition to a normal texteditor the only W3 specific tools used to create this scene were
  • a small skip-all-videos-and-load-directly-the-last-savegame Mod for faster startups (that's optional but recommended)
  • a minimal witcher script to start the scene (as a separate mod)
  • the storyboard UI Mod (as seen in the 2nd and 3rd vid)
  • w2scene encoder (not a mod but an external CLI application)
  • w3speech tools (a phoneme extractor, a lipsync animation creator and encoder, a w3speech packer - all external CLI applications)
  • the (patched) modkit for packaging the scene as a DLC mod
  • wwise to convert the audio from wav to wem (that's easy but unfortunately there is no simple converter utility and you have to install the software package)
The audio was stitched from existing lines in the game but multiple words were mixed solely from phoneme snippets (with varying success). I used audacity for the stitching and Dahlialynn offered some help with the audio afterwards. She adjusted some parts with some other software (although we had a discussion about the necessity :))

As seen in the storyboard showcases (which are by the way best case scenarios for a conversion to a scene!) the result is a textfile with definitions for the scene. For the above scene almost all of the definitions needed to be adjusted by hand in multiple iterations as the storyboard UI does not support changing the weight, clipping, stretching and blend in/out of animations and mimics. But on the bright side: all of the camera definitions can be used mostly as defined in the storyboard (although it's only possible to setup moving/interpolating cams as static cams for start and end as two separate shots in the storyboard and manually link them in the resulting scene description).

However every settings change requires the repackaging of the dlc scenemod and restart of the game. There will be a batch file included which makes this package + restart process simple and "fast" - but adjusting and finetuning animations/mimics/lookats/timings will still be a time consuming process. Even if you are familiar with the settings and have a good grasp what the result will probably be.

So in summary: the number of mods is the smallest problem for creating scenes. It really depends what you want to create: I think these tools are good enough to easily create even complex dialogues with *some* nice animations and camera work (but emphasis must be on the *good* dialogue part) but totally inadequate for "cutscenes" where the visuals are the main focus.

Having said that, it's still mostly fun if you already have the "final" scene visualized in your head...

P.s. Sorry for the late answer I had some problems logging into the forums...
 
Last edited:
rmemr;n7006440 said:
nd Dahlialynn for (overly) nitpicking all sort of stuff
rmemr;n7057810 said:
Dahlialynn offered some help with the audio afterwards. She adjusted some parts with some other software (although we had a discussion about the necessity )

Ignorance is true bliss isn't it ;)

Nevertheless, you deserve all the kudos for outstanding (technical) work rmemr
 
rmemr;n7006440 said:
Hello everybody,

a small showcase vid of coming upgrades:


Scene uses (advanced) features from next version of the w2scene encoder. Still no encoder gui but a *supporting* storyboard ui w3 mod for creating a *rough* draft of scene/scene section will be available, too:



All lipsync is created directly from the audio (but timings were manually adjusted). Audio was converted to WEM with free version of wwise and packed with (yet unpublished) w3speech un/packer.


All tools will be published on nexusmods next week.


Thx erxv for hosting and Dahlialynn for (overly) nitpicking all sort of stuff ;)

End Music is a snippet from the track "Geralt of Rivia" by Marcin Przybyłowicz from the "The Witcher 3: Wild Hunt - Official Soundtrack".



Holy fucking shit!! Now we can create new scenes? Is it possible to create even quest or at least long scenes? Holgar82 can you (or anyone) do something with Iorveth and the scoia'tel?

This is fucking good news. I beg for some scoia'tel content please!!
 
Sam2305;n7079310 said:
Now we can create new scenes? Is it possible to create even quest or at least long scenes?

I guess there is no limit on how long scene can be, I'm not sure. Also there is still much left to do to make custom quests, and I believe it's not even possible. Also I'm looking forward to see any content with Saskia or Iorveth.
 
linaswas_001;n7079840 said:
I guess there is no limit on how long scene can be ...

even if it turns out that we can feed the game with very long cutscenes you have to consider that the limit is most likely the time you put into making those. It is basicly making a movie section or a series episode for a game. depending on what you want to show you probably have a few actors, all in need of specific animations, mimics and text lines for each scene element you shot. Adding camera positions and char placements. All this bound together in numerous scene elements. You have to go through those elements time after time. And the biggest problem of all is (as far as I know) you have to rearrange the entire scene once you left the game or if you made a new one but need some shots from a previous, spoken: there is no "safe the scene" and "load the scene" button for production. Everything is live and the prostproduction is really a postproduction without additional shots.

but the tool is awesome after all
 
Top Bottom