Ballad of the Botanical Bomb Bug
I made a rather fun character for a Pathfinder 2E campaign: A mad-science botanist, who uses grown constructs for everything imaginable. Cracks jokes about finding vanadium pentoxide for more sulfuric acid, or silicon life being her least favourite half-valence-full-based life. Mechanically, she uses the actions of the Alchemist, throwing alchemical bombs with various effects. Narratively, she grows adorable little plants which can get up and independently move, leap, and explode.
I had a very vivid mental image of how this should look and work. But how should I get there? I’d done plenty of pixel art in my days hacking mods for Commander Keen, but habits built with the low resolution EGA palette don’t lend themselves well to the kind of smooth, colourful, and organic shapes I wanted.
Eliding society scale concerns about AI for the sake of a personal project, is it viable here? My partner had recently had a lot of fun playing around with DALL-E, and I’d previously spent an evening jamming with a friend to explore ComfyUI. To date, my experience was AI did best to get to the 80% of the 80/20 split, when you’re not worried about the details, willing to let it have significant creative freedom, and willing to look at it from a long distance to ignore features melting into one another. Still, human artists also chafe under the unpleasant experience of a neurotic client constantly reprompting to elicit exactly what they had in mind, without creative freedom of their own.
I knew there was much I hadn’t explored yet: controlnets, iterative inpainting, regional masks, and anything other than basic euler A. How far could I get with a modestly powerful laptop in an Airbnb?
The end result took a week of downtime research, and a packed day of iterating between 61-75 generations, picking the best, performing a lot of manual painting, then repeating with the new version as a base.
On your own
First, a quick sanity check. How well does the model do on its own?
…Not great. There were two freeform results worth saving:
- The first had a cute and only slightly too cartoonish style, and wasn’t oriented to show off the interesting features.
- The other was too realistic, but still did a sort of cute, anthropomorphic wave with an arm that melted into the nose. It did have a pretty nice shell and, unusually, actually generated with a plausible launching-tail.
Doodle doodle, do me a favour!
Text to image alone wasn’t going to cut it. I’d already had a lot of fun using a doodle controlnet to generate “Loss” in otherwise innocuous images. Maybe it could be used to get started here?
This starts to get somewhere! From here, I could mask out areas that I didn’t like, and let a specialized inpainting model regenerate the masked region with a more specific prompt.
At this point, we’re mostly structurally correct. The features exist in the right places, in mostly the right colours. The sacs exist and are protected and exposed in the right places. There is no visible launch-tail, but it is covered by the wooden shell in a reasonably plausible fashion similar to some click beetles. The antennae/fruit aren’t quite right yet, but they’re spatially separated, and should be a relatively easy fix.
I have one gripe: I really don’t like the legs.
I warned you about stairs
hands legs!
I want the bomb bugs to scuttle and perform small leaps! Just like frogs, and without needing to engage their high-energy launch-tail, while maintaining an insectoid appearance. (See above for where to send corrections about amateur fantasy biology!) This… isn’t really going to exist, and prompting pushes the model to generate one or the other.
The model is going to need some help. Worse, it’s going to need a lot specified at once to remain cohesive. Inpainting elements one at a time ignores the context of the broader image, and the smaller the pieces, the weirder the results. Even with mask growth and feathering, there’s not enough context to preserve the sparkly chemical sac through to the other side of a leg.
Combining with unmasked conditioning helps the model understand the complete picture. Combining too many conditioning regions at once leaves the model to forget about the legs entirely, or to ignore detail for the wooden shell. Karras scheduling seems to do better at making interesting details than Euler A, but also likes to introduce detail where there shouldn’t be, which is annoying when you’re going for a semi-cartoonish style.
I finally get the results I want sampling in two phases: Pipe in the base image for non-SDE, karras 70% denoise isolated to the leg region, but with conditioning combining an overall prompt, leg regional prompt, and sac regional prompt. This is enough to not overload the model and generate a good intermediate result that understands the overall image. Then, pipe the latent to upscale and remove the noise mask, then combine the previous conditioning with additional regional conditioning for the head, shell, and negative regional conditioning at the rear to stop turning the rear into a head. It’s already told where the head should be, but it really likes turning the tail into another head, for some reason. The latent alone is pretty lossy, so all of this conditioning is finally piped into another controlnet which takes lineart generated from the decoded latent to preserve the overall structure and detail, again at Karras 70% denoise.
Final conditioning weighting:
- Legs: 0.80
- Chemical membrane: 0.60
- Head: 0.95
- Wooden shell: 0.60
- The rear is not another head: 0.95
I’m fairly happy with this result. The right rear leg still causes some weirdness, and it’s clear that continuing something on the other side is something that the model struggles with. But it’s good enough to illustrate the fantastic creature I had in mind.
SDXL is supposed to be better all around (though with more resources than I have locally right now), but this problem seems inherent to the flat denoising process. I wonder if it’s possible to create a layered image process which generates partly occluded elements in full before layering occluding elements on top. You’d still need a final denoise for consistent lighting and environmental effects, though.