I’m starting off a new project. And this one is inspired by a recent UNIMA Youth Commission session about AI & Puppetry. The session prompted thinking about the faces of AI, and one day perhaps AI could be “person” with whom we will collaborate, not just a “fake” creator or tool for creators. While selfaware AGI (artificial general intelligence) is likely a long way off, I think we could try that collaboration right now. Or at least I’d like to know how far we can push it now and what sort of result might be possible. So here’s the rough plan…
Create a short story involving 2 characters, one created by me, the other by the AI.
Find a way for us both to manipulate elements of a scene. I’ll use Blender for this, having learned a lot during my last project.
Finally see if I get AI to choreograph puppet movements on screen. This could be the most interesting challenge, will take some thinking.
If others were interested or are experimenting themselves, leave a comment. How do feel about “tinkering with the garage door open”, letting us all know what you’re up to?
I’ll start by sharing a quote shared by Matt Smith from the UNIMA AI session.
My digital puppet collaborators are not machines, but they do learn about, and defer to, my human preferences while – at times – rebelling against my controlling nature. They experience glitches, and have accidents, and it is these that give them credibility. For me, a “successful collaboration between humans and machines” looks like artificial digital puppet life and intelligence
Lucy Childs (2022). The Quest for Life and Intelligence in Digital Puppets (Doctoral dissertation). p186-187
Choosing an AI (large language model)
I thought I’d read into this a bit, because you should know your tools, right? Well it got pretty technical and long-winded, so I’ve included what I found in a different blog post. In the end I felt pretty comfortable going with a later model of OpenAI’s GPT, specifically the gpt-4-0613 model. Oh, I also use the Android app ChatBoost because I can connect it to a paid OpenAI account and select a specific model to use for chat.
Part 1 – Creating a short story
Below I’ve included the discussion I had using gpt-4 to co-create a short story or scene. I think it went pretty well. Towards the end it stopped taking turns, perhaps decided to get a bit pushy with me 😉 but we got back on track.
You: Hi there. I'd like to try an experiment where we co-create the script for a scene together. The scene will involve two characters, one will be played by me, the other by you. The characters will be played by shadow puppets so they could be human, animals or fantasy creatures. I'll start by describing my character, then you can describe yours, then we can take turns saying what our character does next. Let me know if that makes sense to you.
Bot: Absolutely, that sounds like a fun and creative experiment! I'm ready to co-create this shadow puppet scene with you. Go ahead and describe your character.
You: My character is called Sandy. Sandy is a woodcarver who lives alone in a cabin by the woods. By day Sandy loves to walk through the woods, whistle tunes and work on carvings. At night Sandy loves to sit by the fire and eat fish caught from a stream. Your turn.
Bot: My character is named Whiskers. Whiskers is an adventurous squirrel who resides high in a tree just on the edge of the woods. During the day, he explores the woods, gathering nuts and playing with the other forest creatures. At night, he enjoys nestling in the branches of his tree, munching on acorns and listening to the sounds of the night. Now, let's start our scene.
You: Sandy is sitting on a log by a stream fishing. He is whistling a tune.
Bot: Whiskers, from high in his tree, notices Sandy's tuneful whistling. His curiosity piqued, he decides to descend from his tree and scampers towards the stream, rustling some leaves as he goes.
You: Sandy stops and say, "Well hello little squirrel. What brings you to this part of the woods on such a lovely Autumn day?"
Bot: Whiskers looks up at Sandy, eyes gleaming with curiosity. He makes a series of chittering noises and gestures towards the fishing line dancing on the surface of the stream. His way of asking what Sandy is doing.
You: "Ah, you mean my fishing line." Sandy holds up his line. "Well I'm trying to catch a fish for my supper. But I also don't mind sitting by the stream and enjoying the sun. Would you like to join me?"
Bot: Whiskers tilts his head, considering the invitation. After a moment, he scampers onto the log next to Sandy, settling down comfortably. He continues to watch the dancing fishing line with interest, every now and then glancing up at Sandy with a curious chirrup.
You: After a short while relaxing in the sun Sandy turns and says, "I have enjoyed your company little squirrel. And I would stay for longer, on any other day, but I'm afraid today I must get back to work. You see, the King has asked me to carve a statue as a present for his daughter. I have found the perfect piece of wood. But I still do not know what to carve."
Bot: Whiskers perks up at the mention of carving. He chitters excitedly, scampering off the log and running towards one of the nearby trees. He comes back moment later, holding a fallen acorn in his paws. Whiskers drops the acorn in Sandy's lap, looking at it and then Sandy expectantly. Was his suggestion of carving an acorn statue too ridiculous?
You: Sandy says, "My dear little friend, what a perfect idea! So simple and beautiful. Would you mind if I carved a statue of you holding the acorn?"
Bot: Whiskers looks at Sandy, surprise clear in his eyes. He chitters excitedly, scampering in a little circle before settling down again. He then poses with the acorn, as if giving Sandy his approval to proceed with the new project.
You: Sandy races off to get his tools and the wood to carve. In a moment he is back and working with joy and purpose on the block of wood. In what seems like no time at all, the wood is carved away, and what remains is a perfect statue of squirrel proudly holding a shiny acorn. Sandy sighs and steps back to look at his work.
Bot: Whiskers climbs onto the log, standing beside the statue. He chitters approvingly, his eyes wide with admiration. He runs his paws over the wooden version of himself, his tiny fingers tracing the intricate details that Sandy had carved.
"Well done, my friend," Sandy says to Whiskers, patting him gently. "This is a masterpiece, thanks to your inspiration."
The sun sets, painting the sky in hues of orange and purple. Happiness filled the air as Sandy packed up his tools and Whiskers scampered around the statue. It was a day well spent, and an unlikely friendship formed in the heart of the forest.
You: Hey, I was supposed to say what Sandy says. Couldn't hold back there, huh? It really does seem like a good place to finish the scene. But I'll add on more thing. Sandy says, "Would you like to come back to my cabin for supper? It's small but cozy with a warm fireplace."
Bot: With a swift nod, Whiskers accepts the invitation. He clings onto Sandy's shoulder as they make their way back to the cabin, their journey lit by the dwindling sunlight filtering through the autumn leaves. Their friendship promises many more shared adventures and warm evenings by the fireplace.
You: The end. Well done, you! It has been fun co-creating this story with you.
Bot: I enjoyed it too! Thank you for the creative scenario. I'm here if you want to create another story or need assistance with anything else.
This was a bit of an experiment inspired by 2 techniques. The first technique is use of digital projectors to throw rich background scenes onto a screen along with traditional shadow puppets casting shadows. The production Feathers of Fire is a great example of this (image below). The second technique is the use of electronic hand controllers, called “waldos”, originally used to control animatronic puppets, but now these are also used to control computer graphics (CG) puppets. Jim Henson’s Creature Shop as using this for shows such as Word Party (image below). In fact these waldos are in use together with motion capture of actors for the body of these CG character. So taking this a step further, could the hand controller be replaced by motion capture? And given a digital background scene could those hand gestures be used to puppeteer CG shadow puppets on top of that scene?
TL;DR – Well the answer is yes, to a limited degree, but I don’t think the results are good enough to persist with. Not without further investigation of more expensive equipment and software. And I guess that’s an extra question, is this accessible to everyone, can it be done cheaply then? Sadly no.
I used the free 3D modelling and animation software Blender, with the free BlendArMocap plugin, and sketches were done with pen and paper but eventually for digital drawings I used an XP-Pen tablet (not necessary but it makes Blender grease pencil drawing a lot easier). To summarise two of the key issues, the creator has discontinued development on BlendArMocap, and as-is the range and accuracy of motion control is limited.
If you want more details, or to try this for yourself, and I hope you’ll push this exploration even further — read on and I’ll go a bit deeper. I’d love to know what you discover too, so leave a comment. You can also watch my results on YouTube and decide for yourself. By the way, the characters occasionally jumping around was due to jerky mocap issue, it was not intended.
Is This Still Puppetry?
I felt I needed to answer this question. The World Encyclopedia of Puppetry Arts recognises virtual puppets. But isn’t what I’m doing just “animation” aided by motion capture? Well I hope you’ll see it as puppetry. The virtual puppets I created are very much like real world shadow puppets — essentially flat black figures placed against a screen with a few moving joints. The motion capture plugin was able to map my real world movements onto a virtual human skeleton (a “rig” inside Blender), which is like having a virtual twin. I decide to move this virtual puppeteer out of camera shot, but you still can see a screenshot of it below. Then in software I connected the puppets’ joints to the virtual puppeteer hand and finger movements, very much like connecting a marionette with strings to my fingers. I could watch the puppet move as I moved my real hand. The puppet did not emulate my motion capture, it’s more accurate to say that I was manipulating the puppet in real time. So I believe in essence this is puppetry, using manipulation to breathe life into an inanimate object. Brian Henson, while demonstrating a waldo device controlling a digital character on screen, compares this to muppets on a TV monitor.
“when we’re puppeteering we don’t think about our hand, we don’t think about our puppet. We look at a monitor and there’s a puppet on the monitor, and we make the puppet do stuff. So eventually you’re in the training, you’re forgetting that anything is happening up here (a raised puppet arm). All this has to happen automatically as you’re deciding what that puppet does. And it’s the same thing that’s happening here (points to the digital character on screen) but at a more sophisticated level.”
Brian Henson
Building all the Components
In thinking about putting this together I wanted it to be a real test. I wanted it to tell a story, to have multiple characters and to look decent. And perhaps because I started with a sketched story board, I decided for ease of transition that the main character, a magi, would essentially stay in place while the background would scroll from one scene to the next. Of course, some characters would be fixed to the scene, essentially scrolling with the background. So the final setup in Blender (screenshot below) included a stationary camera facing the scene, perfectly positioned to capture one background scene panel at a time, with stationary lighting that spotlit the current panel, a row of the background panels that moved from the far right to the left most panel as the story progressed, my virtual puppeteer twin just off camera to the right, the Magi puppet sitting just in front of the background panels (an opaque, black material, reflecting no light so appearing as a shadow), and a number of other moving puppets tied to their specific panels. You can see on the overview screenshot below the Magi just starting out kneeling in front of the fire on the first panel, the right most scene.
The concept puppets started out as sketches. Then in Blender I traced each of the shapes as a separate grease pencil drawing for each puppet. In the case of puppets with moving parts, I drew shapes for each of the parts/limbs then joined them. The Magi was the most complicated, requiring the parts to be joined/rigged with movable “bones” as part of their own armature or skeleton (screenshot below). The person in the sky is was another traced puppet but with only one moving joint, an arm that raises and points.
The sky person was done using a special effect. The puppet object was a thick outline that used Blender’s Geometry Nodes to replace the outline with a star field. It used a Driver that changed with a time transition, to disperse the stars randomly at the beginning and to coalesce them back to the outline at the end. A similar effect was used for creating the dove shaped comet in the sky. The tiger that transformed into a woman was simply two shadow puppets joined at right-angles, so a quick 90 degree rotation transforms from one shape into the other (screenshot below).
Tech Tips: for turning a Blender grease pencil object into a puppet.
Add a new object “Grease Pencil > Blank”. For drawing the puppet setup the main material, rename to say “BlackPaper”, to have Stroke settings of Line, Solid, Base Color of black, Self Overlap. Turn on Fill with settings of Solid, and a Base Color (something like dark blue initially, but change later to black to create shadow) ensuring Alpha is max 1.0. Add a new material call it “BlackPaperHoles” with Strokes settings Line, Solid, BaseColour black. Turn on Fill with settings Solid, Base Color black alpha 1.0 and set Holdout, which cuts/masks out anything else in the drawing. (You could create a new layer for holes and make it as a mask for the first layer, but there’s no difference, and Holdout can keep the holes on the same layer. You can also set Holdout for the Line settings if you want find cutouts.) You can then draw over this with a new material that has a Solid Fill but with the Holdout option selected, which will cut holes in your original material. Useful for eyes, mouth, and other detailing. Go into Draw Mode, toggle on Automerge (to easily continue lines, or connect ends, but perhaps turn off when doing fine work or holes), then adjust your pencil Radius (say 5px) and Strength to 1.0, and Advanced Active Smoothing to 0.35 (or .6 say). You can create other coloured materials to add some luminous parts of the puppet.
In the Data Properties I created different Layers for different moving parts, to ensure they overlap in the right order, and also for easily turning off layers to work on specific limbs. In this section you will also turn off Lights for each layer, so scene lighting will not affect the puppet as a shadow. You can change to Sculpt Mode to smooth or manipulate or clean up any lines. You can also change into Edit Mode to manipulate or delete individual points on the line curves. I used Edit Mode to select all vertices associated with a limb/part (hide layers or materials to make doing that easier), then under Data Properties I created a Vertex Group by Assigning the selected points, and created one for each limb, prefixed with “SHAPE_” and then it’s name. Later you can more easily reselect all these vertices to Assign them to an Armature Deform vertex group.
To “rig” the puppet, back in Object Mode Add an Armature Single Bone. I won’t go into this in too much detail, as there’s lot of tutorials on this. This first bone should represent the fixed bone controlling the whole body, so place, scale and rotate this into a good spot, then Extrude child bones from the end to create structure and joints, naming them all usefully. In Pose Mode, lock the transformations down to prevent movement in unexpected dimensions (eg. lock location if it’s not connected to a parent bone, lock scale so it will only scale if the parent does, lock rotation in all planes except the pane you expect movement). In Object Mode, select the Grease Pencil object then shift click to select the Armature last, press Ctrl-P to Parent > Armature Deform with Empty Groups. The Grease Pencil object should now have an “Armature” modifier added to it and each bone should have been added as a Vertex Group of the same name. In Edit Mode in the Data Properties > Vertex Groups properties, select a limb group you created already, press the Select button (all of the vertices should get selected), select the correct bone group to attach it to, press the Assign button. Do this for each bone and limb you need to attach.
The scene panels themselves were just flat planes in Blender that use Material Nodes to add a sky color ramp. Using Geometry Nodes a star field was added to the first panel. Using a sketch I traced a Grease Pencil drawing of each scene on top of each panel (screenshots of one scene sketch and final below). Some darker grease pencil objects were used between panels to help hide the transition from one scene to the next. The sun and the moon had low-power point-lights added to them, to give the effect of them glowing and shining light on the material/scene surrounding them. The campfire was similar but with a Noise modifier added to the light’s power so that it flickered.
Rigging the Motion Capture
By following the BlendArMocap plugin documentation and videos I was easily able to create a human Armature in Blender (a Human meta-rig), generate a poseable skeleton rig , then connect the rig to motion capture to the Mediapipe target for Hands. If you Start Detection your computer webcam then starts motion tracking, then you stop detection, and following a Transfer it translates the motions to positions of the skeleton rig along the animation timeline. These can be seen along Blender’s Animation view timeline (the Dope Sheet). So you can easy delete the movements and start again, or commence further capture from a specific position in the timeline. After your first Detection and Transfer, the plugin seems to be permanently connected and you can animate the rig in real-time during motion capture. But at this point, you can just see the human rig (the digital skeleton, virtual puppeteer twin) copying and recording your hand movements.
To connect those human movements to the puppets I used a feature called Drivers which I applied to the puppets’ Transforms. Almost any setting value in Blender can be setup to be controlled by a Driver. A Driver can be a value on a curve, it can use an Input Variable that’s copied from another object, it can be a simple formula, or it can call complex python functions. The position of the rig bones are input variables that can be used, for example selecting the object “cgt_ring_finger_pip.R” gives access to the values associated with the ring-finger proximal interphalangeal joint (anatomically name) on the right hand, and using the variable type “X Rotation” is a good representation of that ring-finger’s movement. The value tends to change from about 0.25 radians for an open hand to 1.5 for a clenched fist, so you’ll need to map this range onto an appropriate range for the setting you are driving/controlling. For example, say we set xrot as the name of that ring-finger input variable, then a simple formula to map values on the range of 0.25-to-1.5 onto a driver setting range of 0.0-to-1.0 would be “(xrot – 0.25)/(1.5 – 0.25)“. To make life interesting, Blender seems to default to reading input angles in Euler radians, but for objects the transform rotation settings seem to default to Quaternions… so you will have to experiment with ranges. (Perhaps I should have figured out how to input/set in the same units, or converted using trigonometry somehow.)
Tech Tips: BlendArMocap values representing all the hand movements.
In Blender the cgt_DRIVERS and cgt_HANDS collections are created by a “Transfer” operation in BlendArMocap. These contain all of the moving parts of the rig. Parts follow the naming convention “cgt_****_finger_###.%” or “cgt_thumb_###.%” or “cgt_wrist_.%” where **** = index, middle, ring, pinky (specific finger) and ### = dip, pip, mcp, tip, cmp, ip (joint of the finger) and % = L, R (left or right). Parts with an added “.D” suffix appear to be directly connect to those without the suffix, but the values are less sensitive, so I don’t recommend using them. The following are the values I recommend using and the range of values, which you’ll need to take into account when using them to drive other objects (such as puppet bones) in Blender.
Wrists:
right wrist use Y Euler Rotation meaningfully ranges from 0.3 (facing outwards) down to -2.4 radians normally, but can go down to as much as -3.7
eg. select variable: cgt_wrist.R > Y Rotation, Auto Euler, World Space
left wrist use Y Euler Rotation meaningfully ranges from -0.3 (facing outwards) to 2.6 normally, but can go up to as much as 3.5
Fingers:
use X Rotation of the PIP joint, which ranges for most fingers usually from 0.26 (open) to 1.49 radians (clenched), but index finger can go lower 0.1, the middle finger cannot open as much down to 0.6, and the pinky finger cannot clench as much up to 1.22. Different hands may vary and need tuning.
eg. var: cgt_ring_finger_pip.R > X Rotation, Auto Euler, World Space
Thumbs:
use X Rotation of the IP join for thumbs, which ranges usually from 0.06 (open) to 0.94 radians (clenched), but can go as high as 1.1. 0.45 seems a reliable mid-point to to detect if it’s closed.
eg. var: cgt_thumb_ip.R > X Rotation, Auto Euler, World Space
Next, I started to map out which of the available hand movements should controls what puppet or motion. And then I was able to write out the hand choreography needed to act out the show. (I didn’t have a lot of time for practice, so the puppet movements were not second nature for me.) The right hand was dedicated to the controlling the Magi.
thumb open – magi stands up
index finger close – magi is left of screen
middle fingers down – hand is out
little fingers up – head is looking up
little fingers relaxed – head is forward
little fingers down – head is down
hand clenched (all together) – the magi kneels down, is towards the left of screen, and the hand is out
hand relaxed open (all together) – the magi stands up, is towards the right of screen, and the hand is down, head is up slightly looking forward
The left hand was used to control everything else.
fingers closed – triggers the stars coalescing to form the sky person
thumb close – the sky person’s arm moves to point
fingers open – the first time this happens, it triggers the stars coalescing to form the dove/comet
twisting the wrist – this essentially moves the background panels along to the next scene. It’s a bit complicated, as while the wrist is twisted it actually progresses the magi location along the linear story panels, the sequence of background panels. So initially it progresses the magi towards the left of screen, but once it approaches the left edge of the panel, it moves all the panels right (and the magi with it).
open thumb & fingers – to start with the tiger is in the bushes
thumb close – the tiger rises to the right
fingers close – the tiger-woman puppet rotates, essentially transitioning to show the woman
twist the wrist – also turns the woman around (as the background moves)
Wrap Up Findings
In the end I was happy with the results, as I think it was the best I could achieve considering my inexperience and working within a time limit. (I was working toward a deadline to be part of the UNIMA Australia Silver Linings Film Festival.) I probably should have spent more time practicing controlling the puppets, as the Magi video up on youtube was only my 3rd take. Experienced puppeteers using waldos can get used to a new puppet setup in a few hours, I believe.
Important finding #1 on limited movement – Fingers alone have a relatively limited number of dimensions or degrees of motion. BlendArMocap only records finger movements and wrist rotation, barely 6 degrees (6D) of motion per hand, or at least I cannot move all my fingers independently, so perhaps it’s 5D. The motion capture did not seem to accurately recognise fine movements, but it is also possible that the resistance of a physical puppet and screen serves to steady hand movement. Taking a comparative guess about traditional shadow puppetry, you’re often moving smaller characters around a screen (2D), optionally rotating a character (1D), distance to the screen (zoom or fade, 1D), and say a movable head and arm (2D), and I’m leaving out any rag-doll movement of legs perhaps… so this is could be a total of 6D per puppet with 2 or more sticks. I’ve definitely seen Chinese shadow masters manipulating puppets with more than 2 sticks in one hand, and using both hands and feet. Keeping all this in mind, BlendArMocap seems like a limited tool.
Important finding #2 on pros and cons of BlendArMocap – The BlendArMocap plugin does however have some nice features. The capture function manipulates the skeletal rig in real time, so you can see the puppet movement almost as a live performance. It also records the location of the rig at corresponding times/frames for later playback as an animation, or rendering to a video. However there is one major drawback, live motion capture does not progress the Blender time cursor to keep in sync with other movement. So if objects in the environment also move on the timeline, you are not performing “live” with these moving objects. Another example of that drawback, you might record a first puppet moving successfully, but then wish to go back and record a second puppet acting together with the first. While the second puppet would record in real time, the Blender timeline does not progress during capture, so you would not see the first puppet move.
In the end this meant only one set of puppets could be recorded, so the limited number of dimensions in finger movements ended up limiting the total number and movements of the puppets. Also all animation in the environment needed to be the result of motion capture movement or it would not appear while performing live. Any animation that relied on transitions over a fixed period of time, such as a scene change, then needed to take into account what was the latest captured frame (the capture time) and how many frames had been skipped if any, as both real-time capture and real-time playback will drop frames if needed. The final render will not drop frames, but unless these discrepancies are accounted for, scene changes could look very different from performance, to playback, to final video render. In the end simple driver animations became calls to relatively complicated python code. And I still hadn’t worked out all the issues. Some things like the stars coalescing (first scene, triggered by hand movement) this happened quickly during rendering, but slowly during live capture, perhaps due to geometry nodes being used for the transition of a large number of individual stars.
Other technical difficulties also needed quite a bit of coding to attempt to reduce the issues. Smoothing jerky mocap movements. Utilities to cap movement range, or unevenly distributed ranges (displaced midpoint functions). Utilities to latch transitions in one direction based on a motion capture trigger (such as rotating a wrist to transition scenes). It’s quite a lot but if you’d like copies of the code just let me know.
The BlendArMocap plugin uses Google’s MediaPipe for the realtime capture of hand and finger movements, which was originally designed for the identification of hand gestures, so fine movement capture may not be possible. The location and angle of the hand would be the most intuitive way to move a shadow puppet and would provide greater control. While these seem to be recorded in the MediaPipe code, the plugin does not record these values. And at this point the developer has abandoned support for the plugin, adding to the evidence that it’s probably not a good choice going forward.
Important finding #3 regarding equipment – The frame skipping issues may be reduced by having a faster PC. At the very least rates of change might become similar. Smoothness of mocap might also be reduced by a faster PC, as there’s probably a lot going on, eg. video capture, capture processing, translation to rig movements, and then normal Blender animation. But depending on the complexity of the scenes, dropping frames might not be avoidable.
Rokoko Studio appeared to be a Blender compatible alternative. Rokoko does appear to capture very fine movements, as well as 2 more degrees of movement than BlendArMocap – angle of the wrist up and down and side to side – which could intuitively map to shadow puppet location on screen but this would still lack puppet rotation angle. However, hand accuracy in Rokoko Studio is not possible without purchasing hardware such as the Rokoko Smartgloves, and live streaming to Blender seems to require a paid software plan as well. (The total price is perhaps a bargain for professional artists, but it’s more than I’m willing to pay for experimenting.)
And in conclusion – I learned a lot, but I’m still left trying to think outside of the box for a better answer. How else could I be doing this? Using a proper and hand-built waldo? Animate the scenes based on standard animation timelines, and somehow persist with mocap puppeteering without the screen? Perhaps a green screen performance of real shadow puppets could be merged with Blender animation, using software such as OBS (Open Broadcaster Software)?
