had some fun scripting recently, finally got a chance to polish my python skills. Im doing a small python script addon that imports kinect facial animation data into blender.
The facial data is captured from Brekel’s face tracking software named “Brekel Kinect Pro Face”, the software tracks/records facial points/movements and creates data that can be used for simulating facial animations.
The software exports float variables labeled as “Animation Units” these units are designed to be used for driving facial shapes/morphs created in a 3D software package. In our case, Blender Shapekeys.
each of the units has corresponding name and value, the names are pretty much self explanatory
blender users can use these to determine the type of shapekeys needed for their models as well as use it to drive the created shapes.
As of now the creation of facial shape drivers are done by hand, ill be writing a script soon that can apply these units as drivers to shapes effortlessly.
An additional button at the sidepanel perhaps.
Apart from the animation units I also managed to Import the “head frame data”, this contains the tracked data obtained from the head’s rotation and location movements .
The data was applied to an “Empty” and can be used to constrain the rotation / location of the head.
here on this example I used it on blender’s most beloved test monkey “Suzzane”
___________________________________________________________________________
Download:
finally here’s the import script addon>> brekel_addon.zip
the script is not done yet, there’s still a few more stuff to be done and I still have to import the tracked facial points.
you can already import the “Animation Units” and the head movements with the script.
For those who doesnt have access to kinect sensors, Ive included there the exported test file i used for the monkey. More test files can be found within Brekel’s site.
___________________________________________________________________________
Links:
for more info about the Brekel Face software please head on to this site
Blender can be found here
downloads for Brekel Face are available within Brekel’s website, just search for it at the right panel. There are Installers for 32 and 64 bit systems. As for me, I was lucky enough to aquire a Licence straight from Brekel for doing this script. Thanks man, really appreciate it.
full tutorial on Blender/Brekel Face workflow coming soon
Marwin'sThings 
Pingback: Blender support » Brekel
Do you have a blend file of a face animation? All I get is a wiggling box.
Hi Terry,
Ive included a blendfile link @ the BA forums, where you originally asked.
Thanks for reminding me, shouldve included that. Ill update my blog post.
Can’t wait for the full tutorial 🙂
Hi Pascal, yeah as of now the plugin is still experimental, still studying how ill be able to implement it with blender
anyways ill let you guys know @ BA if i make more progress, currently testing it out with some characters :).
Hi Marwin,
Downloaded your script, now I am trying to use the information to drive shape keys.
How should I go about connecting it up? It has me totally stumped.
Can’t find any solution on the net.
Thanks for your efforts!
Wow, found your Zombie file, figured out how to get it working.
Thanks again.
Hi Tim thanks for downloading the script. If you’ve found the “animation units”, those are actually blender custom properties you can use those to drive a shapekey from blender. here’s a quick tutorial on how to use custom property as drivers >>http://www.youtube.com/watch?v=hU8kGo8EZB4
the animation units are stored in the empty object named “parent” (the circular empty @ the top of the box). just select the empty go to the object tab and look for the drop down named “Custom Properties”.
The script is actually old, im currently making a new one that automatically applies the units as drivers to shapekeys, thus skipping the whole process of making the drivers for it. Keep an eye out, and things will go easy as pie.
Thanks again,
-Marwin
btw saw your site, nice works 🙂
Cheers for replying Marwin!
I’ll have to get working on better models in preparation ;P
Thanks for checking out my site!
For facial tracking, would it make a difference using the windows kinect vs the xbox kinect?
Hi Sami,
If you’re planning on developing a software windows kinect is a good choice, it has more features to offer than the xbox kinect.
Unfortunately if you’re planning on trying out hacks xbox kinect is still your best bet, most hacks were programmed using the xbox and some of them don’t support the windows kinect yet.
Not hacks, mostly softwares such as Brekel, Ipi Motion caputre studio and faceshift. You see I have the xbox kinect at the moment, but I plan on buying another kinect to capture more accurate full body tracking for animations, but im wondering if I should wait till get the windows kinect to use it for facial motion capture due to it’s “near” mode, if that would make a bigger difference with accuracy?
Hi Sami,
I don’t think it’ll make any difference other than to allow you to shoot your face closer (by the way we have kinect zoom for that).
The capture relies on an alogorithm that kinect SDKs provide and the output also relies on the artists hands and his understanding of retargeting so its strictly not on the hardware. I think every software will only differ depending on the libraries used, not unless windows kinect has a different and more advanced SDK than the xbox one and it specifically stated itself that its camera is better, I dont think its quite usefull for me other that taking advantage of its new features.
Hi man, your work is very interesting! In fact me and my friend we’re trying to figure out how to track a face (with his mesh modeled by SDK Toolkit) and connect it to a Blender’s avatar face that replicates some basic facial expression. Have you completed your script? If yes we’d like to test it! 🙂 Thanks!
Hi Reekoz thanks for the comment,
But dont give me too much credit, this is Brekel’s software and I just wrote a plugin for it.
If you’re interested you can contact Brekel on his site (www.brekel.com) and ask for an evaluation copy.
He has developed tons of kinect related mocap tools
there including a point cloud generator, its really cool check it out.
As for the plugin ill let you guys know when I release a new one,
anyways the old one works with current 2.5x versions of blender and also 2.6x i think, it imports .txt files brekel exports.
The captured data is imported as animated “blender custom properties”, that you can use for driving blend shapes.
The instructions are included on the .zip file if im not mistaken.
If you guys are not too much in a hurry you can also wait for the new one, I might perhaps make a tutorial along with it.
Someone also asked for this lately, so i just might include it to my to do lists.
-Marwin
“full tutorial on Blender/Brekel Face workflow coming soon”…???????
also, is there any further news on this project as I would love an update please 🙂
Hi Niel,
Thanks! yeah its been a while, the project that i worked this on just suddenly shutdown. Im really glad somebody still took interest, since you requested it ill go take another shot.
I also have a new version of this plugin at home, ill get it posted soon. 😀
-Marwin
That would be awesome Marwin, THANK YOU 😀
Yes, I am VERY interested indeed 😉
I look forward to hearing of an update (maybe a youtube vid tutorial of the work flow) as soon as you are able to provide a release 😀
You have no idea how happy this makes me and how loud I will be shouting your name from the 3D roof tops! 😛
I do a lot of Beta/Alpha testing for Otoy (Octane render plugins for Poser/Daz studip & Blender) and various other Devs (such as Look@myHair or Garibaldi) and so on so feel free to give me a yell if you want any testing done ok 😀
Thanks for your reply 😉
haha, thanks for the heart warming post.
cheers! 🙂
-Marwin
Great work! I am just getting started with a similar addon for Faceshift (realtime) so an update would be much welcome.
Hi Hacadela,
cool! I haven’t tried that software before, i’ll be happy to share any
import strategies I’ve come up with. It’s a bit busy @ the office for the moment, but I definitely have the brekel project on my list now. Just hang on for a few days, I promise i’ll get back to you guys.
I am looking for a solution for facial mocap that would integrate well with Blender 2.69 / 2.7x. My facial rig has no shapekeys, only bones. It’s a requirement of the game engine I am using.
Would Brekel work for me? Is there up-to-date add-on for Blender 2.69 / 2.7x ?
Thanks!
Hi motorsep,
Actually you can use the data to drive bone actions instead of shapekeys,
you’ll just have to setup a few action blocks as you would with shapekeys, although there’s just one problem here.
Driving actions require bone constraints that works only inside of blender, in order for you to export the bones with an actual action block you must use a script that can keyframe the the driven bones to an action. I can help you out with this if you’re interested.
-Marwin
Thanks for the reply Marwin. I am certainly interested in seamless workflow Brekel > Blender 🙂
I am just gathering resources / info currently, so I don’t have Brekel + Kinect yet.
Could you please e-mail me at alexander DOT zubov AT kot-in-action DOT com and we can talk about it ?
I was thinking if your add-on could have an option to parent emptys to vertices (the ones that are important for animation), and then parent Armatures to each empty (not bones inside one armature, actual Artmatures; if you have 100 emptys – that will be 100 Armatures). Then create one Armature, with number of bones equal to number of armatures parented to emptys + 1 root. Each bone would have Copy Rotation/Location constraints with target set to Armature (parented to the empty) and Bone (inside that armature).
This way vertex deformations will be passed on to final face rig, and I would just bake Animation and then Auto weight copy of the head mesh to that Armature with baked animation.
I know it works because I tried doing that manually for something else 🙂
The only thing is to figure out out how to automatize it, and I am not sure what needs to be done for jaw, neck, etc.
Hi motorsep,
Actually that’s quite possible really, brekel face provides data for the head matrices
although it’s not a complicated structure, we can make use of the captured data from the lips and eyebrows nevertheless.
I’ll consider including that in the next plugin release.
-Marwin
Do you have an approximate time frame for when new version will be released?
Any news on a new version for Blender 2.69+ ?
Okay, I’m really sorry for all of the delay but I promise I’ll get something done by “next week” for the new brekel plugin, as for the realtime capture feature though I guess that’s something that would have to wait.
I had problems communicating with the brekel app through blender last time but I promise I’ll get you guys this feature, I’ll try and get this to work one way or another..
-Marwin
Real-time output into Blender would be cool !
Currently I am just looking into getting animated head into Blender, with animation, so I could transfer animation from vertices into bones in my game-ready head model.
Hi Marwin.
I just discovered your addon and it looks cool. So i gave it a try.
I may be stupid but… i just had the cube. so the question is : must i create the vertex groups on my mesh, named with the bone rig ?
Are you still on it by the way ?
Thanks for your job, and good luck.
Dan
Hi Dan, just create the appropriate shapekeys for your mesh
then import the brekel file again while your mesh is selected,
the script will automatically apply the data to your mesh,
as well as the animation data that comes along with it.
here’s the list of shapekeys that brekel supports
important blender is case sensitive so you might wanna copy this
list straight to blender
Brekel Animation Units == Blender shapekeys:
Brows_Inner_Up
Brows_Inner_Down
Brows_Outer_Up
Brows_Outer_Down
Lip_Stretch
Lip_Kiss
Lip_Corners_Up
Lip_Corners_Down
Upper_Lip_Up
Upper_Lip_Down
Jaw_Open
Hope this helps 🙂
Hi, thanks for the quick answer.
So, i created the list you gave me in the shape keys.
Then i tried to import the txt file; but nothing move.
So, i tried again with each shape key made, and then, import again, but still standing with no moves.
I tried with, and without the base shape key each time.
I must do something wrong, right ? 🙂
If you have an idea of what is wrong… thank you very very much.
blender has bugs with drivers being driven by custom properties in the side panel
try playing back the file instead of scrubbing through the timeline (alt+a)
important things to remember:
make sure your head object is selected before
importing, this lets blender apply the data correctly to the desired object
lastly the shapekeys should have correct spelling
let me know if you still have problems
thanks for your query 🙂
-Marwin
Hi Marwin, thanks for your help again.
Well, i do not understand. Maybe it’s because i use a makehuman model, or Maybe it’s because the empty cube appears on the model’s feet, but, i’m all wrong.
i tried from scratch, and here is what i did :
-import an mhx model with its rig
-create the shapekey list, with Base shapekey, with exactly right spelling, without attributing any vertexes on it.
-imported txt file with space key -> Import brekel data
the result is an empty cube shaped on the model’s feet, wich is moving along the time line (head tracking movement)
As i describe it, i’m asking myself how can the script attribute the good vertexes to move if i do not identifie them first… so i attribute vertexes for one shape key in order to test, with selecting vertexes, moving it a little, just a little little.
tried one shapekey, with no results too.
In the end, just took an head, with the shapekey list, and nothin else, and place the head in the center, then import the txt file. the cube appeared, the head inside, but… No Results. Well, i’m bad.
Have you found a solution to this problem? – i have the same 😦
Hi Marwin, I was wondering if your script allows direct interface with Brekel (for real-time in Blender) just like NI Mate does or it is just for importing the end result after recording?
Thanks,
Eric
Hi Marvin,
is your plugin still relevant for Blender 2.75? Is there anywhere a full (beginner) tutorial how to use Brekel pro face 2 and Blender for face-animation? I tried a little bit… but it seems very difficult… for beginnners it is hard to know where to start.
Hi Mike, sorry for the late reply, I’ll try making some updates, currently I’m unable to access my kinect controllers, but I’ll give you a 2.75 working version.
Hi Marvin,
can you explain very short how i can convert the “Brekel”-face (i imported it as fbx-file into Blender, when i do it via bvh i only get as result some bones “in a row which move very strange) into empties?
I’m lost. How do I get the BVH into Blender? The test .txt file works, but that doesn’t really help me. And trying to import the BVH even with the script installed does nothing
Actually, importing the .txt doesn’t really have any data that I can find. I basically get the wiggling box thing the other guy mentioned. There is nothing in Object data at all.
Nevermind I figured it out. I didn’t realize I needed to look in the properties tab in the N menu. I was looking in all the wrong places. Although, I have run into a different problem. If I try to import a .txt file that I made using Pro Face using the plugin, I get an error and it doesn’t import. Not sure why.
The downloadable link is not working
Seems it won’t work for blender 2.8
Can’t download from the provided link