Streaming – SenpaiSilver

Making an overhead camera streaming setup for gunpla

I have ~~a passion~~ an addiction called Gunpla, it’s about building models from the Mobile Suit Gundam franchise which is basically a giant robot franchise. Giant robots are great.

While thinking about how to change a bit my streaming content I thought about the huge 13 boxes of Gunpla I have not built yet and that has been staying unopened for a couple of months. Basically I thought about an excuse to force myself to take time to build.

A passion

I first fell in love with Gunpla back in December 2021 and I have been building until October 2022. I stopped since I had planned to travel to Japan between November and December.

I have currently built over 30 models, mostly from the Mobile Suit Gundam franchise, and while I’m starting to have some pretty crowded shelves I have not intention of stopping.

Call it an addiction if you want.

Mobile Suit Gundam

That one big mecha anime franchise that appears to be really hard to get into until you realize it’s easy. Pick wherever you wanna start and start.

Most series have their own timeline and are standalone such as:

Iron Blooded Orphans;
Mobile Suit Gundam 00;
Mobile Suit Gundam SEED (& SEED Destiny);
Mobile Suit Gundam Wing;
The Witch from Mercury.

The list does go on.
And then some are considered as the main timeline called Universal Century (UC):

Mobile Suit Gundam;
Mobile Suit Gundam Z and ZZ;
Char’s Counterattack;
War in the pocket;
Stardust Memories;
Mobile Suit Gundam Unicorn;
Hathaway’s Flash.

And while this isn’t a complete a list of course you should feel free to pick it up anywhere you like. I started with Hathaway’s Flash and Unicorn, you will probably miss some plot points but whatever gets you started is valid.

For real just pick wherever you wanna start, don’t care about people judging you, care only about getting into it.

Not just models

Gunpla comes from Gundam and Plamo, and Plamo comes from Plastic and Models. The truth is that they are much more than just mere models you look at: they are fun to build and pose.

Unlike Games Workshop’s Warhammer (40K) franchise you get to built you model without needing glue nor paint and the figures can move thanks to over engineering part that just move in the most satisfying way.
I never thought I’d be blown away by how a plastic model can move a legs and feet but I still am to this day. Some models have some very precise movement that actually does improve the movement range beyond what you would expect.

Am I fanboying over plastic models? Yeah I am.

I’d personally recommend to built the RG Nu Gundam to whoever got into Gunpla and is looking to try the best model there is to build out there at the time of writing.

Setting up a stream camera

Let’s get into the main subject after presenting the hobby and my addiction.

I’m the happy owner of an Avermedia GC551 capture card, a 7 meter long HDMI cable, a Canon 90D and a RØDE VideoMic GO II. This is all you need to get started with showing an image if you know what you are doing.

The basic setup I was going for at the beginning was pretty terrible since I was using a tripod on the side but I’ve since then upgraded to Elgato’s Master Mount L and Flex Arm L to hold my camera in an overhead view… And promptly sent it back because the Flex Arm can’t hold a DSLR when it weighs just 1Kg…

Elgato Flex Arm L used with a camera, as seen on Elgato.com

I should have thought about why they don’t specify how much load it can take and some things are surprising…

So if the Elgato Flex Arm isn’t an option, what else can I try? Well there’s this Tarion Camera Destop Arm (spelt like that). Am I throwing money to the problem to check what sticks? Yes…

If we take a closer look it’s shorter, with the vertical pole being only 52cm long compared to the Elgato one going up to 128cm. Is it going to be an issue?

If I’m using my Canon EF 50mm f/1.8 STM Lens I need at least 1 meter of distance between the lens and the table to properly record the surface. Buying a 35mm lens would help but it would increase the cost too much for it to be worth right now.

After fiddling a bit with it I’m able to clear the distance and find out that I was wrong: I need more than 1 meter of clearance. So now I’m looking at the SmallRig RA-S280A Air-cushioned Light Stand with Arm 3737, it’s called a light stand but it will be perfect for the camera.

I can also use a counterweight on the handle thanks to a very handy screw in hook and what it looks like from the point of view of the stream ?

It’s perfect for what I need.

What’s next?

Next up is setting up a microphone and an iPhone 11 to capture my voice and facial expressions. Let’s do a combo.

I’m mounting the phone on a Joby GorillaPod 5K with a K&F Concept CA02 Quick Release Plate clip and on top of that I’m putting the RØDE VideoMic GO II on that clip. The mic is connected to the camera and the sound is then fed into the the capture card over HDMI.

The GoPro Hero 11 with the 3-Way Tripod is used as a webcam to complete the body tracking and needs to be facing me.

Adding iPhone facetracking, with webcam body tracking ?

Oh yeah, because I’m not streaming in front of my desktop BEEF PC I need to connect a camera to track my body… My laptop is a MacBook Pro M1 Pro, I’m lucking that I can connect my GoPro as a webcam (as it need to be centered) and I can run OpenSeeFace, the facetracking backend for VSeeFace.

OpenSeeFace requires Python 3, I tried using Python 3.10 but one dependency doesn’t seem to exist for my MacBook, so I’ve opted for Python 3.9 and used the following commands to install and run:

# Clone or download the zip
git clone git@github.com:emilianavt/OpenSeeFace.git
cd OpenSeeFace

# Install the latest Python version & pew for easy venv
brew install python@3.9
pip install pew

# Create an environment for the appropriate Python version
pew new openseeface -p3.9
# Or run you venv
pew workon openseeface

# Install dependencies
pip install onnxruntime opencv-python pillow numpy

# Run facetracker.py
python facetracker.py -c 2 -F 30 -W 1920 -H 1080 \
    --model 4 --gaze-tracking 1 --discard-after 0 \
    --scan-every 0 --no-3d-adapt 1 --max-feature-updates 900 \
    --log-output output.log --max-feature-updates 900 \
    --ip 192.<the_computer_with_vseeface> -p 11573

Now to explain how to select the camera:

Parameter -c is for selecting the number associated with the camera;
Parameter -F is for the framerate of the camera;
Parameters -W and -H are for the camera resolution;
Parameter --model improves accuracy, I don’t know why I was able to go with 4 but this is the command line I got from VSeeFace running on Windows and it ran on Mac OS.

Sadly, without running the OpenSeeFace scripts on PC you will have to guess the camera number, frame rate and resolution… Yeah that’s how those things work when there’s no support for Linux and Mac.
My time to shine with a pull request, I should look into that.

For the iPhone I’m running iFacialMocap and this is all you need for your facial expressions, but do setup blendshapes for your model and face, this post isn’t about this.

I only use iFacialMocap and skip the webcam for body tracking, this makes the setup easier but I still wanted to write up all that in case I change my mind.

Conclusion

I didn’t cover the VTubing aspect of it because it’s out the scope of the video but I’ll have to write a post about some of that and explain my setup since my desktop computer is used for streaming but I’m not building anything on my desk.

I need to improve the lighting and maybe do some color calibration for streaming a better image quality.

The grind of streaming on Twitch

Update 2023/03/26:

Turns out streaming gunpla instead of video games helped so much more in getting over 3 average viewers at the same time. People also feeling more engaged and tend to participate much more in chat too.

That was really unexpected but I’m fine with that because I liked streaming my build of the HG Gundam Aerial:

Built my HG Gundam Aerial on stream! #gundam #gunpla #witchfrommercury pic.twitter.com/Np0Xpf1GXu

— SenpaiSilver ? FR/EN VTuber // Otaku&CodeMonkey (@SenpaiSilver) March 25, 2023

I will definitely do more of that and improve my setup since I have no plans of stopping. I’ll write up a post about the setup once it’s more set in stone.

Original post:

As I’ve announced previously on YouTube and also on the blog I’m trying to get a bit more into streaming. The grind isn’t easy because the path to affiliate has some specific requirements that are actually a bit hard to achieve.

Path to Affiliate

This is what it looks like:

The harder part of it is getting an average of 3 viewers because if I stream towards an audience of no one then I’m just digging down.

I also tend to get much more views on YouTube but more on that later down the post.

Streaming on multiple platforms is forbidden

Once affiliated with Twitch I won’t be able to stream to Twitch and YouTube at the same time. Streaming towards a Restream Docker container was really convenient to automatically upload my VODs to YouTube but it increases the stream latency so much that the interactions are slow.

Once I gain my affiliate status I won’t be streaming to two platforms at the same time.

Streaming on YouTube?

To enable monetisation on YouTube there are different rules:

Getting at least 1k subs is easier than keeping viewers engaged for 4k hours.

I’m thinking of trying something with the public shorts views but that would mean that I need to line up some content and be as widespread as possible.
I could edit some of my VODs on YouTube to extract some highlights but there could be much more work than I’m expecting.

I’m also not very fond of YouTube shorts as I classify that under “infinite scrolling brain dead content for zombie”.

Conclusion

Streaming is a grind and it ain’t easy.

‘Nought said.

Discord animated voice PNGs

Update 2023/05/03

Use Reactive.Fugi.Tech instead, it’s easier. You can setup your profile pictures and share a link to be integrated into OBS. It just works.

Original post

When building your streaming overlay you might want to add the other users that are in your current vocal chat but you want to customize a bit the way it’s displayed.

Discord doesn’t offer any of those options natively but with OBS Studio it’s possible to use some tricks.

CSS & web knowledge required to understand how to use CSS and how to link images.

Setting up the widget

Nothing really hard here, got to the Stream Kit page for Discord in the overlay section.

Setting up the server and the voice channel to be used will alter the URL so make sure you are using the right channel, but it’s possible to use a personal and private channel to use for PNG VTubing.

In OBS within your scene you need to create a browser source and use the provided link as such (no preview until it’s saved):

This is the basic setup. Next we will need a way to get the user ID on Discord.

Setting up Discord

To be able to get the user IDs we need to enable the developer options within Discord. Go to your settings within Discord and in the advanced tab you’ll find the developer toggle:

Getting the user’s ID is as simple as right clicking a name in the chat and clicking on the Copy ID button:

Customizing the widget

Since the widget is a web page within a web view in OBS Studio it’s easy to customize through CSS. Here’s the code to copied & pasted into the custom CSS text area in the browser source:

/*
To be used with the Voice Widget from the Discord Stream Kit
https://streamkit.discord.com/overlay

Do not hotlink images, please rehost them to avoid stealing
bandwidth and keep control over your images, Imgur is a fine
option for this.

Credits:
- Generator by kukushie: https://www.reddit.com/r/VirtualYoutubers/comments/k4q8jw/css_generator_for_discord_reactionary_images_that/
- Edited & improved by SenpaiSilver: https://twitter.com/SenpaiSilver

Please keep the credit chain when making & publishing your changes.
*/

@keyframes speak-now {
    0%  { bottom:0px; }
    15% { bottom:15px; }
    30% { bottom:0px; }
}

body {
    background-color: rgba(0, 0, 0, 0);
    margin: 0px auto;
    overflow: hidden;
}

/* Aligning everything on the same line */
.voice-state {
    display: inline-block;
}

/* Images must be square, change height for it to be scaled
back to 128px wide, or else deal with bad alignments */
.avatar {
    height: 128px !important;
    width: auto !important;
    border-radius: 0% !important;
    filter: brightness(50%);
}

.speaking {
    border-color: rgba(0,0,0,0) !important;
    position: relative;
    animation-name: speak-now;
    animation-duration: 1s;
    animation-fill-mode: forwards;
    filter: brightness(100%);
}

/* Hiding names because the discord parameter on the streamkit
doesn't seem to be taken into account correctly */
.name { display: none; }

/* Default avatar */
.avatar {
    content: url("https://link-to-image.png");
}

/* Users go here: */

.avatar[data-reactid*="new"] {
    content: url("https://link-to-image.png");
}
.speaking[data-reactid*="new"] {
    content: url("https://link-to-image.png");
}

The result is an overlay, where the images change went talking in some way:

Brightness is increased;
The image is bumped 15 pixels higher;
The image changes depending if an image is set for talking only.

Explaining some parts of the code further

It’s important to understand some parts of the properly before running this code, even though it’s just CSS.

First of all you cannot trust users, I recommend using a default image for users that are not part of the CSS to avoid displaying potentially unwanted profile pictures:

/* Default avatar */
.avatar {
    content:url("https://link-to-image.png");
}

This code will replace the img tag’s content with the provided image for everyone that connects to the voice channel. After that it will be possible to set below the specific users with their IDs:

.avatar[data-reactid*="user discord id"] {
    content:url("https://link-to-image.png");
}
.speaking[data-reactid*="user discord id"] {
    content:url("https://link-to-image.png");
}

When we override .avatar we are already setting the default image for the user, by specifying that the attribute data-reactid contains a certain string we can refine our selector and make it target a specific user ID (since they are unique).

In this case the same applies when .speaking is added to the user .avatar.

If you wish to retain round avatar you will need to tweak the border-radius within the first definition of .avatar.

Possible improvements

Since we are dealing with images, animated GIFs are an option and could be used to display an animation with transparency.

If the GIF format doesn’t provide enough colors then it’s possible to fallback to something like the animated WebP or APNG format.

Credits, warranties & notes

This code is based on a generator found on reddit, I’ve changed it to make it work better and fit my own settings. If you modify the code please keep the credit list and add yourself to it.

The code is provided as is and might break one day if the Stream Kit is updated.

This code can be used for son PNG VTubing too.

Separate your audio streams for streaming on Windows

Streaming and recording video games is something I’ve been into, casually, for years (since 2006). Recording gameplay with audio usually includes all the system audio and this can be a problem.

Recording all system audio issue

Windows, your web browser and the game’s audio will play through the default device. That default device is usually your headphones or your speakers.

OBS and ShadowPlay usually record that default device, this is something we’ll have to change.

Virtual Audio Cable

I’m using Virtual Audio Cable (VAC). You can use the trial version, buy it or use an alternative virtual cable driver, this is all up to you.

Open the Virtual Audio Control Panel as an admin (always) and setup the desired number of lines. Each line will be setup to receive audio from certain apps depending on the usage you want.

I have the main line for gameplay, my Discord line to record people I’m talking to. I also have a third line for miscellaneous things and a fourth that I’m not using.
Ideally I would’ve have set one line specifically for gaming, but Windows 10 being what it is (and standards being what they are) it ain’t going to work like that.

To complete the audio setup I also need to output those lines to something and that something is my headphones. Each output line automatically inputs the sound into a virtual input too, that way you can listen to it.

Basically this is how things go:

Outputs:
1. Headphones;
2. Line 1: Main output (default output device);
3. Line 2: Discord output;
4. Line 3: Misc output.
Inputs:
1. USB Microphone (default input device);
2. Line 1: Main output mirrored as input into my headphone;
3. Line 2: Discord output mirrored as input into my headphone;
4. Line 3: Misc output mirrored as input into my headphone.

Windows 10 doesn’t remember

Like I said previously one problem that forces me to set the line 1 as the default output device is because Windows 10 can’t exactly remember correctly which app is outputting on which device.

This issue could also be the other way around where an app will have a different output selected but still output to the default device, I’m not sure why but answers would be appreciated.

OBS recording

OBS is a great piece of software that permits me to merge lines and split some too! This is perfect for recording gameplay and streaming (at the same time).

First we need to specify that we will be recording several tracks into our ouput in OBS, for that we need to go into the settings: File, then Settings.
From there we go into the Output section and select in Streaming the first audio track as the one we will be using for streaming like so:

As a bonus you can also use a different audio track for the VOD so you can have avoid having Spotify playing in the VOD (and avoid the DMCA).

In recording we need to tick all audio track to enable all audio tracksfor later:

Then of course we need to name them and change the audio bitrate if you desire:

In my case I have everything playing on track 1 since it is the default track.
Track 2 will contain the game audio or the main subject of the video/stream.
Track 3 is my voice, followed by track 4 with voices from Discord.

Having different tracks enables me to control what is output when I record and stream, this way I’m able to make audio commentary when preparing a video and after that replace that commentary with a carefully recorded one, while keeping the gameplay audio without altering it.

To select where what track plays it’s necessary to go into the Edit menu, then Advanced audio properties. Set the different inputs and outputs as you wish in the new window:

Keep the first track checked for all lines, check the other numbers as you need.

Troubleshooting: static/glitch audio

After a while if the computer is not rebooted for days it might happen that the audio fizzles, glitches or becomes static. This isn’t a good experience.

The fix consists in restarting Virtual Audio Cable or the audio service from the control panel (as an admin).