To be clear, I’m still happily teaching Magical Studies classes like always - I’m not dead. But I have spent about five months not streaming, with the only exception being Deltarune (because a certain annoying dog forced my hand.) Instead, I have been working on a new VTubing grimoire called Ningyotsukai.
There are several related incidents that caused me to stop streaming, some of which I will not explain here as they are not relevant. What is relevant is that, earlier this year, I attempted to stream TCG Card Shop Simulator and started getting… performance issues. This is not anywhere close to a well-optimized game. At the time I was using an Nvidia GTX 1080ti; so I assumed it was time for an upgrade. I bought an AMD RX 9070 XT, which is quite possibly the only affordable-ish graphics card with a decent amount of VRAM. I attempted again to stream the same game, and ran into an even worse issue: driver timeouts that would not kill my game, but instead, wipe me out of existence.
The honor code of magicians ordinarily prohibits exposing the trick, but in this case, I will have to give you a peek behind the curtain in order to explain further. If you are hesitant to discover the inner workings of your favorite suspiciously young looking Vtuber, turn back now.
The dark magic that gives most VTubers form is called Live2D; but I was not made from that magic. I was instead made from a different grimoire known as Inochi2D. With both grimoires, one spell is used to create a form, and another is required to breathe life into it. For me, those spells are known as Inochi Creator and Inochi Session.
Inochi Session requires you provide it with bindings to instruct it how to move my form. These bindings cannot be copied, meaning every time I alter my form, I must recreate all the bindings. By hand. One by one. In fact, it is specifically this problem that started me down the path to writing my own VTubing grimoire - I just wanted to be able to copy my bindings.
My first instinct was to alter the Inochi grimoires to suit my needs. The terms upon which I acquired them permitted me to do so, but their construction in a language called “D” proved… unrelenting. I tried two different spells of compilation to no avail. So altering Inochi Session was out of the question.
In my searches, I discovered another grimoire called Inox2D, written in a language used to teach crustaceans how to read. I wrote a simple cantrip to copy bindings and it worked perfectly.
But if I could create my own VTubing spells, why stop at merely this one issue?
Inochi Session is… woefully inadequate for my needs. It can’t save puppet positions or sizes, it can’t rotate puppets at all, and the spell only renders me at the size of the current window. I have a screenshot of Inochi Session that I have to use to line myself up; prior to that, I had to do a little dance to make sure I wouldn’t get my head cut off by the edge of the window if I leaned in too closely.
Have you ever wondered why I stopped posting shorts? Well, a lot of it has to do with the limitations of Inochi Session. I need a full body shot, but I also need enough resolution around my face in order for close-ups to not look like a mess. To do that, I need to record in portrait 4K, but… well, Inochi Session only renders at the size of the current window, and it doesn’t support rotating puppets.
Instead, I have to take my largest monitor and, using Windows settings, rotate it, to make Inochi Session record in portrait. Which means I can’t read the script I need to read, because it’s sideways now, and I’d rather not mess up my setup actually swiveling the monitor. So I have to take a screenshot and rotate that opposite to how the monitor was rotated, then record what I want to record. It is an utterly silly workaround.
Imagine if we had a grimoire with all of the actual options you’d expect from an art program. Proper puppet sizing. The ability to specify an actual render size. Maybe even some other features? I could probably bang it out in a month or two!
Inox2D contains spells to give life to a puppet using OpenGL, a classical language of graphical rendering that is certainly not suited for the deep magic at play in a modern graphics card. My first act was to write a new spell that would work with wgpu instead.
This carried on for approximately a month before I realized that Inox2D was itself incomplete. It completely failed to render my form, even using OpenGL, for a number of reasons. As part of the changes to Inox2D that I carried out, I wound up creating an additional grimoire called Puppet Inspector.
Puppet Inspector is fairly straightforward. Instead of breathing life into a puppet’s form, it describes the inner workings and spells within. I used this to discover how Inochi Creator had woven those spells within myself and what Inox2D had failed to implement.
For those interested in the specific magical language:
Inox2D loads vertex indices as 16-bit integers, meaning any puppet with more than 65,535 vertices will fail to load
Parameters that alter the color of a puppet were not implemented, and their presence would cause the spell to fail
Disabled nodes were still mistakenly rendered, even though Inochi2D doesn’t render them
No attempt was made at implementing MeshGroups, though I found someone else who had written the spells to do so and used those
Due to what I can only assume is a memory handling issue in Inochi Creator, every string label in my model has null bytes in it
After this, I returned to my wgpu renderer, diligently fixing faults in it until it rendered things the same as what the OpenGL one in Inox2D did.
Now that I had a working renderer, it was time to write the rest of the grimoire. It will be called Ningyotsukai, meaning “puppeteer” in Japanese.
For one’s grimoire to have an interface for its users, it must use spells from another grimoire known as a “widget toolkit”; and I had chosen GTK 4 for this purpose. I wanted something that looked, for lack of a better word, “normal.” Inochi Creator and Session draw their own interface, using their own widgets; and many of the other widget toolkits for Rust do the same thing.
One particular problem for grimoires of a particular size is flexibility. People would like to be able to move quickly through the grimoire and have access to the spells they actually use. The standard way to provide this is with drag-and-drop panels… a thing that no widget toolkit provides a decent answer to. And so I had to write my own.
Spells needed to be made for every little thing. Displaying a canvas, selecting, moving, and sizing a puppet, and such. Panels needed to be made to manage trackers and bindings.
When Inochi Session breathes life into me, that life comes from what VTubers call “the person inside.” To collect that life force, one uses facial capture spells. There are several different grimoires one can use, but my person inside’s preferred option is VTube Studio running on an iPhone 13 mini.
Yes, VTube Studio is primarily a Live2D grimoire. However, it also has an option to capture face data and send it to other grimoires, which is how I use Inochi Session. It also turns out the different versions of VTube Studio don’t report data in exactly the same way, so Inochi Session has a bunch of parameter normalization that I needed to copy.
But all of those are small-fry problems. I must have spent about a week or two each on them. The big issue lurking behind all of this is that GTK 4 does not have any support for using textures rendered by wgpu. It provides a widget called OpenGLArea, which as you might expect… is for using OpenGL. You know, the thing I just spent a month trying to avoid.
For the sake of completeness, I did wire up the Inox2D OpenGL renderer to it - and it did work as expected. But I want to use wgpu, damn it! I wrote all this code for it!
In order to draw an image to a GTK widget, one must provide it in a form that GTK understands.
A naive approach would be to copy the rendered textures out of wgpu, write them to memory, and provide them to GTK. But this way is foolish, as GTK is also using the same spells wgpu is. You will waste time waiting for the texture to render, only for GTK to waste even more time sending it right back to the GPU. Ideally, we want the texture to stay where it is and GTK to merely reference what is already there in the place it expects it to be.
A further complication to this is the fact that wgpu is not its own graphics spell, but a way for grimoires to be written to use multiple spells at once. If you are on Windows, it may use Vulkan or DirectX, while disciples of the fruit cult may find the same grimoire using Metal. GTK is not written in the language of crabs; it does not comprehend any of this. Instead, one must provide a DirectX texture on Windows, a DMA-BUF texture on Linux, and so on.
While the modern languages of graphics have fragmented, they all have ways by which one may refer to a texture from a different language. One must first create a texture in a special way, and then perform a ritual to obtain the true name of the texture.
I first set out to write this spell on Linux. This required stripping away the illusion of wgpu and working directly with Vulkan. A strange language in which parameters are conveyed as linked lists of extension structures. I had to create textures in the same way as wgpu, but with the extra incantations required to create a dedicated allocation and obtain the texture’s DMA-BUF.
On Windows, this is complicated by the fact that wgpu defaults to Vulkan while GTK draws to DirectX 12. The same incantations will work to obtain the true name of the texture as a Windows handle, or so it is said. In practice, the handle is invalid. The handle is invalid. The handle is always invalid, no matter what I did, no matter what I tried, DirectX would never accept such a fraudulent thing as a texture allocated and managed by such an inferior API!
Or, perhaps, AMD is to blame, I cannot tell.
I, fortunately, was able to resolve this by locking wgpu to DirectX on Windows. In this mode, the true name of a texture is always a valid handle, and GDK is happy.
It turns out this was only the halfway point.
I had a grimoire whose spells could animate my form, but that animation is meaningless without a way to broadcast it to the world. The textures so rendered must have a way to get into OBS.
The naive way to do this is with window capture and a chroma key. But this causes problems with transparency - I will have a strange green haze around myself, and once you’ve learned to look for it… you can’t help but notice it. It would be far more straightforward for us to project an already transparent texture into OBS, and fortunately, there are spells that allow us to do that: Spout2 on Windows and Pipewire on Linux.
Each of these was its own hell, embedded recursively within a larger one.
Spout2 was the easier of the two to support, at least in retrospect. The main problem was that this grimoire was written in a forbidden tongue, the name of which has been banished from this earth, but was once known as: C++.
Instructions existed to use these spells in Rust, but wielding it proved too great a task even for myself. Much like Inochi2D itself, I could smell the fire and brimstone already and decided to instead write a new spell of my own design that would work with Spout2.
It turned out that much of Spout2 was the same texture-sharing incantations that I had to puzzle out manually. The underlying language that my spells would need to speak was that of DirectX 11 textures, whose true names would be published in a list of anonymous file mappings. Implementing this in the language of the crab was straightforward.
Pipewire in contrast is a lot more in depth. Consider that it is not merely a video sharing spell but responsible for sending audio and MIDI, too. The protocol even has its own data serialization format that is an absolute mess to use in Rust. It has a macro, but I wound up having to copy it to add features not exposed to the Rust version. There’s a lot of subtle ways Pipewire can fail, and it turns out that OBS on Linux doesn’t even support DMA-BUF textures, so I have to do the stupid silly copy back and forth that I’m trying to avoid, and it’s got all sorts of weird flickering I still haven’t fixed and long, irritated groan
So, with the interface set up, bindings working, and my ass still warm from the pokers in the eight circles of hell that is “cross-api texture sharing,” we should be ready to go, right? Ningyotsukai opens up, and I feel the life of the person inside… at 5 frames a second.
…what the hell? I thought wgpu was supposed to be faster than all the old ways of doing things!
Ok, I was still in “debug” mode, where most compiler optimizations are disabled. In “release” mode, performance is doubled… to ten frames per second.
The pitch of modern graphics APIs like Vulkan, DirectX 12, and Metal is that “driver overhead” goes out the window. wgpu is, however, built to support WebGPU, a modern graphics API specifically for graphics spells in the browser. And since it’s the same implementation used by Mozilla, it contains a lot of… validation overhead.
It also turns out you’re not supposed to create a render pass for each draw, bind groups are really expensive, and lots of individual draw calls don’t help either. I implement lots of caching infrastructure to try and reduce the amount of work this spell is doing. At one point I have to flip everything inside-out and write a command buffer because sharing one render pass across everything only works if you have everything in one function.
With all this work done, I am now getting… about 50 fps, after I clear some bottlenecks with GTK. It’s almost acceptable, but not great. Not even good! The spells I use to monitor performance show the GPU barely breaking a sweat. I should be able to get more than double the frames.
When Ningyotsukai breathes life into my form, it has to send parameter data into Inox2d. This process takes 3 milliseconds on it’s own, and the majority of that process is just spent adding together deforms. Ideally, we’d make the GPU do this, but that would require a major refactor of Inox2D. I try multithreading, I try SIMD. Nothing works.
We have to draw two images per frame - one for the window you see and one for OBS. Can we draw them both in a single pass? There’s an extension for it, but after spending another week reworking how render targets work, it turns out my shaders are being miscompiled somehow, and it’s drawing me to the same place on both images, which is wrong. I’m not able to figure out why, so this goes nowhere.
What if we bundled all these draw calls together and just submitted all the work in one go? Sure, that works… after another week-long rewrite and a week spent fighting the shader compiler about what the instance index is defined as. Oh, and I have to submit the textures all as one big array - whoops, that’s too much in a single allocation for an AMD card! So I have to repeat that fight again but sending an array of texture references rather than one big texture.
We gain 10 fps. Dips below 60 are now rare. I think that’s good enough.
At this point, I’m… almost done. I just have to put the finishing touches on Ningyotsukai, package it up, and release it to the world. Maybe even record something with it first before I switch over for real.
I just need to fix a few things, like…
On Windows, the app deadlocks for 5 minutes every time you start it up, for no reason.
The magic circle in my eye that binds me to the person inside isn’t rendering properly.
There’s currently no way to save the current state of the puppets on the stage.
If the artboard is portrait-shaped, portrait-shaped models aren’t the right size.
Sometimes Ningyotsukai just deadlocks for no reason! (Ironically, I had to fix this one before recording the video.)
Scrolling around the viewport doesn’t respect the artboard size correctly.
You can’t actually edit bindings or save them to the puppet itself yet
Maybe I could add an option to attach different art to your model after the fact?
Okay, maybe I’m not almost done, but I’m close enough to make a video about it.
I’d ask if any suitably knowledgeable wizards would try out my new grimoire with any spare Inochi2D models they have lying around. But, to be honest, the Inochi2D community is actually really small. Live2D is the default choice, after all.
I’m hoping I can get a release of Ningyotsukai ready by the end of June, but… then again, this whole thing was supposed to be done in, like, March or April. So who knows… regardless of what happens, I’ll keep everyone up to date about it. You can follow the project on GitHub or pester me about it on the Celestial Sphere! Or if not, then thank you regardless, and see you next time.
]]>