Chaining for consistency: Prompt -> Structured Data -> Image -> Video

After yesterdays manual prompting for structured data, I added the option to provide a free text prompt to populate character data on the https://ki-storygen.com site.

Now with a prompt like:

An anthropomorphic country mouse, fond of bright red clothing and garish and gaudy jewelry, he's always playing the fool and clowning around. He is unexpectedly perceptive, and unbeknowst to his community he is always on the lookout for danger headed to his meadow and protects his friends through pure guile

A set of structured JSON is created and populates a page like this:

and an image:

and a video:

I'm getting pretty consistently generic backgrounds, see e.g. this frog video:

And the lack of action in the videos is a little disappointing, though the prompt does not specify any action explicitly (I likely need to prompt gemini to give a more specific video action for the character, as the generic prompt making them look heroic or villanious or interesting doesnt seem to be doing much -- the image also tends to a pose at rest, which is also hampering things as the video looks to always start from the image pose, when prompted with an image).

In any event, I think this is starting to be a promising base for stable, reproducible results, but it looks like I'll now have to get more creative to make the output much more interesting, engaging & feeling like it has more life.

I think there are some really interesting things that could be done with video generation on the fly for a child's story being told (hooking up a speech to text service for prompting would be trivial). Cost is definitely a little bit prohibitive to do that too frequently, but I suspect I can get some really good character vignettes from generating an action scene involving two of these generated characters in a location.

The next step is probably putting more of a focus on generating interesting environments for the characters to be in.