So with my new creative friend and collaborator on this project, GPT4, we’ve come up with a pretty good script (in part 1). But my friend has a few challenges being a language model. They have no limbs to control a puppet. And no real vision nor ability to get visual feedback, so I’m not sure precision movement is possible. But hey, life is full of challenges, so I’ve come up with a plan to give the AI as much control as I can.
I had always intended for the scene to have subtitles, as practically I don’t have to do voice acting, and having seen this before in shadow theatre I like the aesthetic, text feels like a story book. “Once upon a time…” etc. But what I didn’t expect, the AI somehow decided their character (a squirrel) doesn’t speak, although it does make some noises, and my character the wood carver has some lines. (Obviously a squirrel can’t talk, but understands English?) Playing around in a Blender test scene, I wrote some python code to update a simple Text object with changing text. Then I upgraded the code to read from a CSV file (a simple spreadsheet) that contains time-stamps and lines of text. So that’s a big tick, subtitles are possible! I’m never going to perform subtitles in real time, so copying out my character’s lines and the squirrel’s “noises” makes a lot of sense for me. I’ll need to guess the timings, and then practice the scene a few times to test the pace, but I think I’d be doing this anyway.
What if the squirrel’s movement worked a similar way? The AI appears to have a good imagination, so could the AI describe what the squirrel was doing for each part of the scene? Maybe it could say where the squirrel was, how it would be moving, and if it moved to somewhere else. This seems quite doable for the AI. And if I’m already testing timings for subtitles, timings for actions would line up with these. And somehow I think I could perform my character’s movements alongside the timed squirrel actions. (Still need to figure out how I’ll eventually do that.) If something doesn’t work, I can try to tweak the timings, or potentially ask the AI for ideas to fix that part of the scene.
I created a dummy squirrel object in my Blender test scene, from a flat image, and wrote some more python code to attempt to move it. Starting simple, I just want to move an object from one place on screen to another. So I reused the code to read another CSV file with time-stamps, but instead of subtitles this file contains instructions for movement. Initially the code understands only 2 instructions, one that names a location on screen (eg. “tree”, “ground”, “top-of-log”), and another that moves a named object from one named location to another. After figuring out how to do some basic vector arithmetic in python and Blender… success!
OK, there’s a lot more work to do. Clearly a squirrel doesn’t fly in a straight line, so at the very least it needs to walk, or hop, or run, or jump. It will need to change direction, sit up, perhaps move subtly on the spot, or while “talking”, or focus on an object like an acorn. And maybe it’ll need to do these things in a “squirrel-like” way. But I’ll ask the AI what it thinks about this next.