Hand-tracked VFX with MediaPipe, OpenCV, and Pygame

In the anime Jujutsu Kaisen there is a technique called Hollow Purple: the character brings a blue force and a red force together between his hands and they collapse into a single violet sphere that tears through everything in front of it. I wanted to see how close I could get to that feeling with a webcam and some Python.

The result is a small Python app. It finds your hands in the camera feed, hangs a glowing orb with a particle stream and rotating holographic rings off each index fingertip, and when you bring the two fingertips together it fires a burst of particles and a white flash. It is a toy, not a product, but it is a good excuse to look at what a real-time vision loop actually has to do every frame.

The stack, taken from the project’s virtual environment: Python 3.14, OpenCV 4.13 for capture, MediaPipe 0.10.32 (the Tasks API hand landmarker) for tracking, pygame-ce 2.5.7 for drawing, NumPy 2.4, and PyInstaller to ship it as a Windows executable. There is a short demo recording, made with the app’s own recorder; I’ll add it here once I have somewhere to host video.

In short:

One loop does everything: read a frame, find up to two hands, take the index fingertip of each, decide if a gesture happened, draw the effects, and flip the display.
The only “gesture” is two fingertips crossing under 100 pixels apart, and it is edge-triggered so it fires once, not every frame.
Particles are plain Python objects with a spiral, a little gravity, and a lifetime that doubles as their alpha.
The cheap parts are vectorised NumPy and normalised landmarks. The expensive parts are allocating thousands of tiny surfaces per frame.
In a one-file PyInstaller build, a relative path to the model file silently misses the copy you bundled.

The loop

There is no framework and no threads. main.py is a single while running: loop, and every frame walks the same chain:

fig. 1 / one frame of the loop. if the model fails to load, fake fingertips keep the effects testable.

Capture is ordinary OpenCV. On startup the app probes camera indices 0 to 2 and keeps the first one that actually returns a frame, rather than assuming index 0 is the right camera. Each frame is mirrored (so moving your right hand moves the effect to the right), resized to the 1280x720 window, and darkened to 60% brightness with a single NumPy multiply so the glow has something to stand out against.

OpenCV hands you BGR arrays in row-major order; Pygame wants RGB surfaces indexed by x first. The bridge is two calls: cv2.cvtColor for the channel order and swapaxes(0, 1) before pygame.surfarray.make_surface. Getting that wrong gives you a sideways, blue-tinted you, which is a useful early sign that the plumbing is connected.

If MediaPipe fails to initialise, the app does not exit. It switches to a demo mode where one fake fingertip circles the screen and a second one appears and disappears, so every effect can still be tested without a camera-facing hand. The status line says “Demo” instead of “Real”.

From landmarks to a gesture

The MediaPipe hand landmarker returns 21 landmarks per hand, each with x and y normalised to 0..1 across the image. I only use one of them: landmark 8, the tip of the index finger.

rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb_frame)
detection_result = detector.detect(mp_image)

if detection_result.hand_landmarks:
    for hand_landmarks in detection_result.hand_landmarks:
        index_finger_tip = hand_landmarks[8]   # index fingertip
        fingertip_positions.append((index_finger_tip.x, index_finger_tip.y))

Because the coordinates are normalised, mapping to screen space is just multiplying by the window size. That also means the detector could run on a smaller image and the effects would still land in the right place, which matters later.

The “Hollow Purple” moment is the two fingertips meeting. The rule is deliberately simple: if there are two hands and their fingertips are less than 100 pixels apart, fire. The detail that makes it feel right is that it only fires on the frame the distance crosses the threshold:

if len(screen_positions) >= 2:
    distance = calculate_distance(screen_positions[0], screen_positions[1])
    if distance < 100 and previous_hand_distance >= 100:
        burst_x = (screen_positions[0][0] + screen_positions[1][0]) // 2
        burst_y = (screen_positions[0][1] + screen_positions[1][1]) // 2
        particle_system.spawn(burst_x, burst_y, count=100)
        flash_active = True
    previous_hand_distance = distance
else:
    previous_hand_distance = float("inf")

Without previous_hand_distance, holding your hands together would spawn a hundred particles and a full-screen flash every frame. With it, the burst fires once, and the trigger re-arms only after the hands move apart again or one of them leaves the frame.

fig. 2 / the burst fires on the frame the fingertips cross 100 px, not on every frame they stay close.

The flash itself is a white overlay drawn for three frames with alpha 255, 170, and 85. Short enough to read as an impact, long enough to notice.

The particle system

Every frame, each tracked fingertip spawns 10 particles. The burst adds 100 at the midpoint between the hands. Each particle is a small Python object:

Two populations. 20% are big and slow (size 8 to 12, speed 0.5 to 2), 80% are small and fast (size 4 to 7, speed 2 to 5). The mix gives the stream depth without any real 3D.
A spiral. Each particle gets a random spin between -0.1 and 0.1 radians per frame, and its velocity vector is rotated by that amount every update.
A bit of gravity, 0.02 pixels per frame squared, so the stream droops.
Lifetime as alpha. Lifetime starts at 255 and drops by 4 per frame, and the same number is used as the draw alpha. Size shrinks by 3% per frame.

curr_speed = math.sqrt(self.vx**2 + self.vy**2)
curr_angle = math.atan2(self.vy, self.vx) + self.spiral_speed
self.vx = math.cos(curr_angle) * curr_speed
self.vy = math.sin(curr_angle) * curr_speed

self.x += self.vx
self.y += self.vy
self.vy += 0.02          # gravity
self.lifetime -= 4       # also the alpha
self.size *= 0.97
return self.lifetime > 0 and self.size > 0.5

fig. 3 / a particle's life. a merge adds a burst of 100 at once. lifetime runs out after about 64 updates, before size does.

The update returns whether the particle is still alive, and the system rebuilds its list from the survivors. With a 255 lifetime and a step of 4, a particle lives about 64 frames, so one tracked hand settles at roughly 640 live particles and two hands at roughly 1,280, plus any bursts.

Each particle is drawn as a short trail through its last three positions and three stacked circles: a faint bloom at twice its size, a medium glow, and a small core that is white for the big particles. The orb on the fingertip uses the same layering idea at a larger scale: six translucent bloom layers out to a radius of about 120 pixels that pulses by 20 pixels on a sine wave, a vivid mid layer, a white-hot core, and a thin halo ring. Around it, three segmented rings rotate at different speeds and directions, with a small block of flickering readout text to the side, for an Iron Man style HUD.

There are also four colour modes on the keyboard (purple, blue, red, white), an “air writing” layer that draws a persistent line behind your fingertip, and a recorder that writes the composited screen to an MP4.

Where the frame time goes

I want to be honest about this section: the app has an FPS counter, but I never profiled it properly, so there are no numbers here. What I can do is point at what the code does.

The cheap parts are cheap because they stay in C:

Darkening the frame is one vectorised NumPy expression over the whole image, not a per-pixel loop.
Colour conversion, flipping, and resizing are single OpenCV calls.
Landmarks are normalised, so mapping to screen space is two multiplications per hand.
clock.tick(60) caps the loop so it never spins faster than the display needs.

The expensive parts are all on the Pygame side, and they share one cause: allocating new surfaces every frame. Each particle circle is drawn by creating a fresh alpha surface, drawing one circle on it, and blitting it:

def _draw_circle(self, surface, x, y, radius, color, alpha):
    s = pygame.Surface((radius * 2, radius * 2), pygame.SRCALPHA)
    pygame.draw.circle(s, (*color[:3], alpha), (radius, radius), radius)
    surface.blit(s, (x - radius, y - radius))

Three of those per particle, at roughly 640 particles per hand, is close to 2,000 short-lived surfaces per frame for one hand, before counting the eight layers of each glow orb. The vignette is rebuilt from 150 rectangle outlines every frame too, even though it never changes.

On the vision side, detect() runs in MediaPipe’s default image mode on the full 1280x720 frame. Image mode treats every frame as a fresh photo, so the tracking confidence setting does nothing, and the model works at a resolution far above what it needs to find a fingertip.

If I picked this up again, in order:

Build the vignette once at startup.
Pre-render particle sprites per colour and size bucket, and blit those instead of allocating.
Switch the landmarker to video mode with detect_for_video and a timestamp, so it can track between frames instead of re-detecting.
Run detection on a downscaled copy of the frame. The landmarks are normalised, so nothing downstream changes.
Move particle state into NumPy arrays if the counts ever need to go higher.

Packaging: where did my model go?

The project ships as a single Windows executable. A small build.py runs PyInstaller with --onefile and --windowed, and adds the 7.8 MB hand_landmarker.task model as data. The output is an executable of about 72 MB.

The bug is in how the model is found. The code opens it with a bare relative path, and if that path does not exist, it downloads the model:

model_path = 'hand_landmarker.task'
if not os.path.exists(model_path):
    urllib.request.urlretrieve(MODEL_URL, model_path)   # simplified

A relative path resolves against the working directory: wherever the user launched the program from. A one-file PyInstaller build does not put bundled data there. It unpacks everything into a temporary folder at startup and exposes that folder as sys._MEIPASS.

fig. 4 / in a one-file build the model is unpacked to a temp folder, but the code looks for it in the working directory.

So the bundled copy is never used. Started from a folder without the model, the executable quietly downloads a fresh 7.8 MB copy into that folder. Started offline, MediaPipe fails to initialise and the app drops into demo mode. And because it was built with --windowed, the console messages explaining all this go nowhere. The only clue is the small “DEMO MODE” line on screen.

The fix is a few lines that check for the PyInstaller folder first:

# Written for this post: resolve bundled files in both dev and frozen builds.
def resource_path(name):
    base = getattr(sys, "_MEIPASS", os.path.dirname(os.path.abspath(__file__)))
    return os.path.join(base, name)

model_path = resource_path("hand_landmarker.task")

Two smaller packaging notes. The build script passes --add-data with a ; separator, which only works on Windows (the comment in the script says so, but it is easy to miss). And it adds the effects package as data, which is unnecessary: PyInstaller already follows the imports and bundles those modules.

One more trap from the same project: an earlier HandTracker class still in the repo uses the old mp.solutions.hands API. The MediaPipe version installed in this environment, 0.10.32, does not ship mp.solutions at all. main.py moved to the Tasks API, which is why it works, but the old module would fail on import. If you are following an older MediaPipe tutorial, check which API it uses before you copy it.

What I’d do differently

Resolve resource paths once, in one helper, from the first line of code, not when packaging. Every desktop Python app that ships data files hits this.
Never allocate in the hot loop. Surfaces, arrays, fonts: create them once and reuse them. It is the single biggest cost in this app.
Use the tracking mode the library offers. Video mode exists because consecutive frames are related; image mode throws that away.
Record at the real frame rate. The recorder writes at a fixed 30 FPS whatever the loop actually achieves, so playback speed depends on how fast the machine was. Measure the rate and pass it in.
Give each hand its own state. The air-writing layer keeps one “last position” for both fingertips, so with two hands up it also draws a line from one hand to the other.
Make failure visible in windowed builds. If the app is going to fall back to a demo, say why on screen, not in a console nobody can see.

None of this needed more than a webcam, a free model, and a few hundred lines of Python, which is the part I still find remarkable. If you are building something interactive with computer vision and want to compare notes, you can reach me through my site.