VNyan is a VTubing software used for animating a 3D model for streaming and content creation. I’ve been using it for the last couple of months because of some of its features such as the nodes system.
Nodes are a way to visually script events based on Twitch, WebSockets or whatever is supported in the app. Lately I’ve been thinking about doing a bit more and giving the possibility to just go wild and queue events such as model swap.
Version 1.2.1b is required, the update introduces a wizard that can be skipped if you already have VNyan setup.
What is a queue?
We want to push items to the end of the array and take items from the top of it.
This is what we call first in first out. It’s the absolute basic idea behind the queueing of events.
I want to setup commands to do stuff and have them be processed in order with a short delay between each commands. The commands will be normal Twitch chat commands to make things easier.
We will need two things:
An interval timer that will be triggered to run when the queue is not empty;
The event handling that will pop the first element on the top of the queue and run it;
The logic needed to tell when the loop must run and what it is stopped.
In my example I will have two commands:
!throwApple which will throw an apple at me;
!throwDuck which will throw a rubber duck at me.
This can work for doing model swaps but will require a bit more thought and care since you might want to animate that (through effects or transformations) and you will want to take into account the loading delay.
Queuing in VNyan
First of all we need to create the logic that will handle the events, the entry point will be a trigger and it will start a timer that will call that trigger again until the queue is empty.
The events will be handled by conditions that will be checked, if true you’ll apply you action and you’ll delay another call of the event loop. If false you’ll just flag the event queue as finished and not call it again (until another command is called).
Each command will push an element into the queue and it if the event queue isn’t running it will call the event trigger and setup a flag.
It’s important to note that we use the Enqueue Text Array and the Dequeue Text Array.
Prior to this version dequeueing an item was done by using an Ordered Execution node that would get an item from the queue and remove the item by index.
I have a passion an addiction called Gunpla, it’s about building models from the Mobile Suit Gundam franchise which is basically a giant robot franchise. Giant robots are great.
While thinking about how to change a bit my streaming content I thought about the huge 13 boxes of Gunpla I have not built yet and that has been staying unopened for a couple of months. Basically I thought about an excuse to force myself to take time to build.
I first fell in love with Gunpla back in December 2021 and I have been building until October 2022. I stopped since I had planned to travel to Japan between November and December.
I have currently built over 30 models, mostly from the Mobile Suit Gundam franchise, and while I’m starting to have some pretty crowded shelves I have not intention of stopping.
Call it an addiction if you want.
Mobile Suit Gundam
That one big mecha anime franchise that appears to be really hard to get into until you realize it’s easy. Pick wherever you wanna start and start.
Most series have their own timeline and are standalone such as:
And while this isn’t a complete a list of course you should feel free to pick it up anywhere you like. I started with Hathaway’s Flash and Unicorn, you will probably miss some plot points but whatever gets you started is valid.
For real just pick wherever you wanna start, don’t care about people judging you, care only about getting into it.
Not just models
Gunpla comes from Gundam and Plamo, and Plamo comes from Plastic and Models. The truth is that they are much more than just mere models you look at: they are fun to build and pose.
Unlike Games Workshop’s Warhammer (40K) franchise you get to built you model without needing glue nor paint and the figures can move thanks to over engineering part that just move in the most satisfying way.
I never thought I’d be blown away by how a plastic model can move a legs and feet but I still am to this day. Some models have some very precise movement that actually does improve the movement range beyond what you would expect.
Am I fanboying over plastic models? Yeah I am.
I’d personally recommend to built the RG Nu Gundam to whoever got into Gunpla and is looking to try the best model there is to build out there at the time of writing.
Setting up a stream camera
Let’s get into the main subject after presenting the hobby and my addiction.
I’m the happy owner of an Avermedia GC551 capture card, a 7 meter long HDMI cable, a Canon 90D and a RØDE VideoMic GO II. This is all you need to get started with showing an image if you know what you are doing.
The basic setup I was going for at the beginning was pretty terrible since I was using a tripod on the side but I’ve since then upgraded to Elgato’s Master Mount L and Flex Arm L to hold my camera in an overhead view… And promptly sent it back because the Flex Arm can’t hold a DSLR when it weighs just 1Kg…
I should have thought about why they don’t specify how much load it can take and some things are surprising…
So if the Elgato Flex Arm isn’t an option, what else can I try? Well there’s this Tarion Camera Destop Arm (spelt like that). Am I throwing money to the problem to check what sticks? Yes…
If we take a closer look it’s shorter, with the vertical pole being only 52cm long compared to the Elgato one going up to 128cm. Is it going to be an issue?
If I’m using my Canon EF 50mm f/1.8 STM Lens I need at least 1 meter of distance between the lens and the table to properly record the surface. Buying a 35mm lens would help but it would increase the cost too much for it to be worth right now.
Adding iPhone facetracking, with webcam body tracking ?
Oh yeah, because I’m not streaming in front of my desktop BEEF PC I need to connect a camera to track my body… My laptop is a MacBook Pro M1 Pro, I’m lucking that I can connect my GoPro as a webcam (as it need to be centered) and I can run OpenSeeFace, the facetracking backend for VSeeFace.
OpenSeeFace requires Python 3, I tried using Python 3.10 but one dependency doesn’t seem to exist for my MacBook, so I’ve opted for Python 3.9 and used the following commands to install and run:
# Clone or download the zip
git clone email@example.com:emilianavt/OpenSeeFace.git
# Install the latest Python version & pew for easy venv
brew install firstname.lastname@example.org
pip install pew
# Create an environment for the appropriate Python version
pew new openseeface -p3.9
# Or run you venv
pew workon openseeface
# Install dependencies
pip install onnxruntime opencv-python pillow numpy
# Run facetracker.py
python facetracker.py -c 2 -F 30 -W 1920 -H 1080 \
--model 4 --gaze-tracking 1 --discard-after 0 \
--scan-every 0 --no-3d-adapt 1 --max-feature-updates 900 \
--log-output output.log --max-feature-updates 900 \
--ip 192.<the_computer_with_vseeface> -p 11573
Now to explain how to select the camera:
Parameter -c is for selecting the number associated with the camera;
Parameter -F is for the framerate of the camera;
Parameters -W and -H are for the camera resolution;
Parameter --model improves accuracy, I don’t know why I was able to go with 4 but this is the command line I got from VSeeFace running on Windows and it ran on Mac OS.
Sadly, without running the OpenSeeFace scripts on PC you will have to guess the camera number, frame rate and resolution… Yeah that’s how those things work when there’s no support for Linux and Mac.
My time to shine with a pull request, I should look into that.
For the iPhone I’m running iFacialMocap and this is all you need for your facial expressions, but do setup blendshapes for your model and face, this post isn’t about this.
I only use iFacialMocap and skip the webcam for body tracking, this makes the setup easier but I still wanted to write up all that in case I change my mind.
I didn’t cover the VTubing aspect of it because it’s out the scope of the video but I’ll have to write a post about some of that and explain my setup since my desktop computer is used for streaming but I’m not building anything on my desk.
I need to improve the lighting and maybe do some color calibration for streaming a better image quality.
Keep in my you can change the bitrate and increase it by tweaking the value following -b:v.
This small trick is something I spent a long time looking for and I have had mixed results with it. But lately I’ve been getting into preparing custom alerts on StreamElements for my own streams and I was hoping to add something nice.
In the input field for Google Script API-KEY paste the one from your Apps Script. If you didn’t copy the key in the previous step you can retrieve if by going into the editor, in the deploy menu you need to click on manage deployments and you’ll find it under deployment ID:
Change the languages you want to use lower on the page, feel free to use Google Translate to find out what is what:
On the bottom of the page you will find the URL that you can copy and paste in your web browser, Google Chrome might be the only supported browser at the time of writting.
Step 3: Setting up OBS
Now comes the less fun part. Since the web page needs to listen to your voice you will have to run the page in your own Google Chrome instance, I’m unsure if it works on other browser since I wasn’t able to make the web app perform every single time.
To set it up all you need is a window capture and some filters. If the content of the web browser doesn’t appear you might need to disable hardware acceleration.
For the filters I usually just crop the top and a bit of the bottom, then I apply a chroma key filter to remove the screen background:
Congratulations, you are know streaming while subtitling at the same time. Subs are the way to go and this is what it looks like:
Refreshing the tab might help making the web app work, but if your speech isn’t appearing on screen you should instead try closing and reopening your browser.
I kept the tab loaded before closing Google Chrome and when I opened it again it was working.
Once again this isn’t flawless and some words might appear wrong or be heard wrong, these are the limits of speech recognition. Unless we slap some machine learning over your accent.
It might also strain your performances and require relaunching Google Chrome once in a while.
Turns out streaming gunpla instead of video games helped so much more in getting over 3 average viewers at the same time. People also feeling more engaged and tend to participate much more in chat too.
That was really unexpected but I’m fine with that because I liked streaming my build of the HG Gundam Aerial:
I will definitely do more of that and improve my setup since I have no plans of stopping. I’ll write up a post about the setup once it’s more set in stone.
As I’ve announced previously on YouTube and also on the blog I’m trying to get a bit more into streaming. The grind isn’t easy because the path to affiliate has some specific requirements that are actually a bit hard to achieve.
Path to Affiliate
This is what it looks like:
The harder part of it is getting an average of 3 viewers because if I stream towards an audience of no one then I’m just digging down.
I also tend to get much more views on YouTube but more on that later down the post.
Streaming on multiple platforms is forbidden
Once affiliated with Twitch I won’t be able to stream to Twitch and YouTube at the same time. Streaming towards a Restream Docker container was really convenient to automatically upload my VODs to YouTube but it increases the stream latency so much that the interactions are slow.
Once I gain my affiliate status I won’t be streaming to two platforms at the same time.
Streaming on YouTube?
To enable monetisation on YouTube there are different rules:
Getting at least 1k subs is easier than keeping viewers engaged for 4k hours.
I’m thinking of trying something with the public shorts views but that would mean that I need to line up some content and be as widespread as possible.
I could edit some of my VODs on YouTube to extract some highlights but there could be much more work than I’m expecting.
I’m also not very fond of YouTube shorts as I classify that under “infinite scrolling brain dead content for zombie”.