Ivan Popleyshev has been working hard upgrading the libraries he has authored to PixiJS 6. The next one up is PixiJS Picture Kit – a shader-based implementation for blending modes that WebGL doesn’t actively support. Apart from blending, the “backdrop” texture it exposes can be used for other kinds of filtering.

Blend Modes

This section goes over blend modes and how they work in WebGL.

The blend mode defines how colors output by the fragment shader are combined with the framebuffer’s colors. In more simple terms, a blend mode is used to mix the colors of two objects where they overlap. Below, blend modes supported in PixiJS are shown by rendering a semi-transparent blue square over a red circle:

The normal blend mode makes the source color, blue, appear in the foreground over the destination color, red, in the background.

Porter Duff operations

The blend modes in the 2nd and 3rd columns have suffixes OVER, IN, OUT, and ATOP describing Porter Duff operations. These represent image compositing operations.

• OVER – colors are mixed for the whole object being rendered
• IN – only overlapping areas are rendered
• OUT – colors are outputted only in non-overlapping areas
• ATOP – colors are outputted only over existing content

In PixiJS, the blend modes only apply to pixels in the object being rendered, so the compositing operations look a bit different. For example, SRC_IN and SRC_ATOP look the same when. An actual IN operation would erase non-overlapping areas in the red circle. But since PixiJS only applies the blend mode in the blue square’s area, this is not possible with blending.

The blend modes with prefix DST switch which color is in the foreground. Even though the blue squares are rendered after the red circles, they are behind with DST blend modes. The DST_OVER blend mode will make a scene appear as if z-indices were reversed.

Arithmetic

The blend modes in the 1st column change the arithmetic used to mix the source and destination color.

• ADD – Sums the source and destination color with equal weighting instead of alpha-weighting
• SUBTRACT – Subtracts the source color from the destination. Negative values are clipped to zero.
• MULTIPLY – The colors are multiplied, which always results in darker colors.

Blend equation

The blend equation is a linear function on the source color and destination color that calculates the output color. This equation can be set separately for the RGB and alpha components of colors.

blendFunc

blendFunc is used to set the weights for the source and destination colors. Instead of passing predefined values for these weights, a WebGL constant representing these weights needs to be passed. For example, gl.DST_ALPHA will set the weight to the destination color’s alpha.

For the normal blend mode, you’d use:

gl.blendFunc(gl.ONE, gl.ONE_MINUS_SRC_ALPHA)

blendFuncSeparate can be used to separately set these weighting for the RGB and alpha channel of colors.

blendEquation

blendEquation sets which equation is used to mix the colors after they’ve been multiplied by weights. You can add, subtract, reverse subtract, and even min / max. Most blend modes use the add equation.

blendEquationSeparate can be used to separately set which equation is used to mix the RGB and alpha channels of colors.

StateSystem

The StateSystem manages the blend mode for the PixiJS renderer. It works by mapping BLEND_MODES to the parameters for blendFunc and blendEquation described above. If you want to add more blend modes of your own, you can modify the undocumented blendModes map in the state system

blendModes basically maps each blend-mode to a list of parameters to blendFuncSeparate and blendEquationSeparate. These lists can have up to 8 parameters but only the first two are required. The ADD equation is used by default.

import { BLEND_MODES } from 'pixi.js';

const stateSystem = renderer.state;

// [gl.ONE, gl.ONE_MINUS_SRC_ALPHA]
console.log(stateSystem.blendModes[BLEND_MODES.ADD]);

// Add OVERWRITE blend mode and set it to a unique value!
BLEND_MODES.OVERWRITE = 100;

// This blend mode will basically overwrite the destination
// color with the source color. The destination has zero
// weight in the output.
renderer.state.blendModes[BLEND_MODES.OVERWRITE] = [
  renderer.gl.ONE,
  renderer.gl.ZERO,
];

In the above snippet, I create a OVERWRITE blend mode that will make the background disappear wherever an object is rendered and only keep its pixels in the framebuffer.

Custom blending with shaders

The “fixed function” blend modes shown so far are perhaps limited. The normal blend mode is by far used for alpha compositing. The other Porter Duff variants can be used for masking. To make more complicated and artistic blend modes, a shader that samples the “source” and “destination” colors from textures is used.

PixiJS Picture implements this kind of shader as a filter. The source object is rendered into a filter texture. The destination pixels from the framebuffer or canvas are copied into a backdrop texture. The dimensions of these textures must be the same.

// Fragment shader

// Texture coordinates for pixels to be blended.
varying vec2 vTextureCoord;

// Filter texture with source colors
uniform sampler2D uSampler;

// Backdrop texture with destination colors
uniform sampler2D uBackdrop;

This type of filter is called a BlendFilter. The fragment shader in BlendFilter is a template:

void main(void)
{
   vec2 backdropCoord = vec2(vTextureCoord.x, uBackdrop_flipY.x + uBackdrop_flipY.y * vTextureCoord.y);
   vec4 b_src = texture2D(uSampler, vTextureCoord);
   vec4 b_dest = texture2D(uBackdrop, backdropCoord);
   vec4 b_res = b_dest;

   %blendCode%

   gl_FragColor = b_res;
}

• b_src is the source color sampled from the filter texture
• b_dest is the destination color sampled from the backdrop texture
• b_res is the output color calculated from the source and destination colors by the blending code. It’s set to the destination color by default.

When a BlendFilter is constructed, the %blendCode% token is replaced by the blending code, which calculates the output color. This way multiple blend modes can be implemented by just writing the blending code for each one. You can find examples of these shader parts in the source code. To emulate the normal blend mode, the blending code would look something like this:

// Note: b_src and b_dest are premultiplied by alpha like
// all other colors in PixiJS.

b_res = b_src.a + (1 - b_src.a) * b_dest;

To use this code as a blend filter, you can construct a BlendFilter and apply it to a display object.

import { BlendFilter } from '@pixi/picture';

// Blending code for normal blend mode
const NORMAL_SHADER_FULL = `
    b_res = b_src.a + (1 - b_src.a) * b_dest;
`;

// Create globally shared instance of blend filter. This is a
// good optimization if you're going to use the filter on multiple
// objects.
const normalBlendFilter = new BlendFilter({
    blendCode: NORMAL_SHADER_FULL,
});

// Apply the filter on the source object
sourceObject.filters = [normalBlendFilter];

Built-in blend filters

PixiJS Picture implements filters for these blend modes:

• MULTIPLY
• OVERLAY
• SOFTLIGHT
• HARDLIGHT

getBlendFilter maps each blend mode to instances of their blend filters, which can be applied on a display object to emulate the blend mode.

import { BLEND_MODES } from 'pixi.js';

sourceObject.filters = [getBlendFilter(BLEND_MODES.OVERLAY)];

PixiJS Picture also exports special versions of Sprite and TilingSprite where you can set the blendMode directly and a blend filter is implicitly applied:

// Note: Use the Sprite exported from @pixi/picture and
// not the default one from pixi.js!
import { Sprite } from '@pixi/picture';

// Set the blendMode on the sprite directly. When the
// sprite renders, it will use the blend filter from
// getBlendFilter() automatically.
Sprite.blendMode = BLEND_MODES.OVERLAY;

Dissolve

The dissolve blend mode randomly chooses pixels from the source or destination texture to output. The likelihood of choosing the source color is equal to its alpha, i.e. a 0.5 alpha means half of the output pixels will come from the top layer and the rest will be from the bottom layer. In this mode, colors aren’t truly “mixed”.

The blending code for this is really simple:

// Noise function that generates a random number between 0 and 1
float rand = fract(sin(dot(
    vTextureCoord.xy ,vec2(12.9898,78.233))) * 43758.5453);

if (rand < b_src.a) {
    b_res = b_src;
}

The famous one-liner rand is used to generate a random number between 0 and 1. If this random variable is less than the alpha of the source color, then the resulting color is set equal to the source color. Otherwise, the resulting color is set to the destination color (by default).

A BlendFilter is needed to use it:

import { BlendFilter } from '@pixi/picture';

// Copy blendingCode
const DISSOLVE_FULL = `
    // Noise function that generates a random number between 0 and 1
    float rand = fract(sin(dot(
        vTextureCoord.xy ,vec2(12.9898,78.233))) * 43758.5453);

    if (rand < b_src.a) {
        b_res = b_src;
    }
`;


// Create blend filter
const dissolveBlendFilter = new BlendFilter({
    blendCode: DISSOLVE_FULL,
});

// Apply it!
sourceObject.filters = [dissolveBlendFilter];

You can also augment BLEND_MODES and create a DISSOLVE blend mode. The blendFullArray exported from @pixi/picture contains the blending code for each mode – the dissolve code needs to be added as well.

import { BLEND_MODES } from 'pixi.js';
import { Sprite, blendFullArray } from '@pixi/picture';

// Any non-conflicting number high enough works here!
BLEND_MODES.DISSOLVE = 100;

// Register the blending code with @pixi/picture
blendFullArray[BLEND_MODES.DISSOLVE] = DISSOLVE_FULL;

// Set it on a PixiJS Picture sprite
new Sprite().blendMode = BLEND_MODES.DISSOLVE;

Now, you can set the blendMode to dissolve directly on a PixiJS Picture sprite.

Backdrop filters

Blend filters use the backdrop texture to read the destination color. If you have imported @pixi/picture, you can use the backdrop in other filters as well!

PixiJS Picture augments the filter system so that it copies pixels from framebuffer / canvas into the backdrop texture before a filter stack is applied. This backdrop texture is then available to filters as a uniform. The name of the uniform is configured by the backdropUniformName property. For BlendFilter, this is set to uBackdrop.

import { BackdropFilter } from '@pixi/picture';

const fragmentSrc = `
    // The filter texture containing the object being rendered
    uniform sampler uSampler;

    // The backdrop texture
    uniform sampler2D uBackdrop;

    // TODO: Your shader code
`;

class CustomBackdropFilter extends BackdropFilter {
    constructor() {
        super(/* vertexSrc, fragmentSrc */);
        
        // Set the backdropUniformName so the backdrop
        // texture is available to shader code.
        this.backdropUniformName = 'uBackdrop';
    }
}

Magnifying glasses

Ivan shows how you can use the backdrop texture with his “magnifying glasses” example:

The grass background is rendered first. The two “lens” sprites are then rendered with a “displacement” filter. The lens texture is a displacement map – each texel encodes how much each pixel must be displaced.

The R channel holds the displacement in the x-direction and the G channels holds it for the y-direction. The (R, G) values are centered at (0.5, 0.5) and then scaled by a certain magnitude.

x = (r - 0.5) * scale;
y = (g - 0.5) * scale;

The centering is done because color values must be between 0 and 1, and displacements can have negative values.

The displacement filter samples the “lens” texture and calculates the displacement vector for the current pixel. It then samples the backdrop texture by adding this displacement to the passed texture coordinates:

// Read the displacement data from the lens texture
vec4 map =  texture2D(uSampler, vTextureCoord);

// Map it into the displacement vector
map -= 0.5;
map.xy *= scale * inputSize.zw;

// Add the displacement vector to the texture coordinates,
// and then clamp it in case it goes outside the
// the backdrop texture.
vec2 dis = clamp(vec2(vTextureCoord.x + map.x, vTextureCoord.y + map.y), inputClamp.xy, inputClamp.zw);

// Handle y-flipping
dis.y = dis.y * backdropSampler_flipY.y + backdropSampler_flipY.x;

// Sample backdrop and output color
gl_FragColor = texture2D(backdropSampler, dis);

Mask filter

The MaskFilter allows you to apply a filter on the backdrop wherever it overlaps with a mask. A common use case is the backdrop blur effect, which can be implemented by simply passing a blur filter to MaskFilter:

// Gaussian blur filter
import { BlurFilter, Graphics } from 'pixi.js';

// Masking filter
import { MaskFilter } from '@pixi/picture';

const mask = new PIXI.Graphics()
  .beginFill(0xffffff, 1)
  .drawRect(0, 0, 100, 100);

mask.filters = [
  new MaskFilter(new BlurFilter()),
];

The above have the effect of blurring the background in the rectangle (0, 0, 100, 100). The white rectangular mask itself won’t be visible. Instead, another translucent white rectangle must be added so it appears visible.

If you’ve been reading up until here, I’m glad this article was informative. As the software industry goes remote, we all need a new office. Check out Teamflow, a virtual office built for the future.

Ivan Popelyshev recently migrated the PixiJS Layers Kit to be compatible with PixiJS 6. I helped him also document the @pixi/layers API here. This package introduces an interesting concept – making the order in which items in your scene tree render separate from the tree’s hierarchy. It allows you to change the display order of display objects by assigning layers.

Let’s start with a scenario in which this might be helpful – notes that can be dragged using handles that are on top. Each note and its handle is kept in the same container – so they move together; however, they need to be in separate “layers” – one below for the items and one above for the handles. In a conventional scene tree, it would not be possible to have this layering without splitting the notes and their handles into separate containers and setting their positions individually.

But @pixi/layers makes this possible! You can group items in your scene tree and render those groups as layers. These items will only render when their parent layer becomes active during rendering.

// The stage to be rendered. You need to use a PIXI.display.Stage
// so that it correctly resolves each DisplayObject to its layer.
const stage = new PIXI.display.Stage();

// Set the parentGroup on items in your scene tree.
const GROUPS = {
  BG: new PIXI.display.Group(),
  FG: new PIXI.display.Group(),
};

// These groups are rendered by layers.
stage.addChild(new PIXI.display.Layer(GROUPS.BG));
stage.addChild(new PIXI.display.Layer(GROUPS.FG));

// How to make an item so that the handle is above all 
// other content
function makeItem() {
  const item = new PIXI.Container();
  const handle = item.addChild(new PIXI.Graphics());// Do drawing
  const body = item.addChild(new PIXI.Graphics());// Do drawing

  // Set the group of the handle to foreground
  handle.parentGroup = GROUPS.FG:

  // Set the group of the body to background
  body.parentGroup = GROUPS.BG;

  return item;
}

This changes the display order of items in the scene tree.

How it works

When you import @pixi/layers, it applies a mixin on PixiJS’ DisplayObject and Renderer. It adds these new behaviors:

• If the scene root is a Stage, the renderer would now call updateStage – which traverses the scene tree and resolves each item to its group and layer.
• A DisplayObject will only render when its active parent layer is being rendered (which is resolved in the update-stage step).
• It also patches the Interaction API’s TreeSearch to correctly hit-test a layer-enhanced scene tree.

In the updating phase, every Group with sorting enabled will sort its display objects by their zOrder. This z-order is different from the built-in z-index that PixiJS provides. Both z-index and z-order are used for sorting objects into the order you want them to be rendered. z-order is the implementation provided by @pixi/tilemap. You can use both of them in conjunction – objects will be sorted by z-index first then by z-order.

When a layer renders, it will set the active layer on the renderer – which then indicates that objects in that layer can now render.

Note that Layer extends Container, so you can add children directly to it like Item 3 in the diagram. You don’t have to set parentGroup or parentLayer on these children as it is implicit.

Z-grouping to reduce sorting overhead

In containers with many children, if only a few z-indices are being used they can be replaced with a fixed number of layers. For example, when users edit a group of items they expect them to come on top. Instead of setting a higher z-index on these items, this can be implemented by promoting items to an “editing” layer. This is much easier than shifting items into another container, which interferes with interaction.

This technique replaces sorting with a linear-time traversal of your scene tree. It should especially be used when your scene tree is large.

Using zOrder

A Group will sort its items by their z-order if the sort option is passed:

import { Group } from '@pixi/layers';

// Group that sorts its items by z-order before rendering
const zSortedGroup = new Group(0, true);

If the z-order of a scene is relatively static, it’s more efficient to disable automatic z-order sorting and invoke it manually:

// Don't sort each frame
group.enableSort = false;

// Instead, call this when z-orders change
group.doSort();

Another neat feature is that Group emits a sort event for each item before sorting them. This can be used to dynamically calculate the z-order for each item, which is particularly useful when you want to order items based on their y-coordinate:

// You have to enable sort manually if you don't pass
// "true" or a callback to the constructor.
group.enableSort = true;

// Sort items so that objects below (along y-axis) are on rendered over others
group.on('sort', (item) => {
  item.zOrder = item.y;
});

Check out this example – here the bunnies are moving back and forth but the bottommost bunnies are rendered over others:

Layers as textures

Layers can be rendered into intermediate render-textures out of the box. The useRenderTexture option enables this:

layer.useRenderTexture = true;
layer.getRenderTexture();// Use this texture to do stuff!

When a layer renders into a texture, you won’t see it on the canvas directly. Instead, its texture must be drawn manually onto the screen. This can be done using a sprite:

// Create a Sprite that will render the layer's texture onto the screen.
const layerRenderingSprite = app.stage.addChild(new PIXI.Sprite(layer.getRenderTexture());

The layer’s texture is resized to the renderer’s screen – so too many layers with textures should not be used. This technique can be used to apply filters on a layer indirectly through the sprite rendering its texture:

// Apply a Gaussian blur on the sprite that indirectly renders the layer
layerRenderingSprite.filters = [new PIXI.filters.BlurFilter()];

By rendering into an intermediate texture, layers can be optimized to re-render when their content changes. You can split your scene tree so that layers are rendered separately – and then use the layer textures in the main scene.

// Main application with its own stage
const app = new PIXI.Application();

// The relatively static scene that is rendered separately. This is not
// directly shown on the canvas - later, a sprite is added
// to render its texture onto the canvas.
const staticStage = new PIXI.display.Stage();
const staticLayer = new PIXI.Layer();

staticLayer.addChild(new ExpensiveMesh());
staticLayer.useRenderTexture = true;

// Add a sprite that renders the static scene's snapshot to the main
// scene.
app.stage.addChild(new PIXI.Sprite(staticLayer.getRenderTexture());

// Rerenders the static scene in the next frame before
// the main scene is rendered onto the canvas. This should be invoked
// whenever the scene needs to be updated.
function rerenderExpensiveMeshNextFrame() {
  app.ticker.addOnce(() => {
    renderer.render(staticStage);
  });
}

Double buffering

A beautiful use of layer textures is for showing trails of moving objects in a scene. The trick is to render the last frame’s texture into the current frame with a lower alpha. By applying an alpha, previous frames quickly decay which ensures only a few frames are seen trailing.

Since WebGL does not allow rendering a texture into itself, double buffering is needed. The layer needs to flip-flops between two textures, one for rendering into and one for rendering from. This can be enabled by useDoubleBuffer:

// Ensure the layer renders into a texture instead of the canvas
layer.useRenderTexture = true;

// Enable double buffering for this layer
layer.useDoubleBuffer = true;

Note that useRenderTexture must be enabled for double buffering – not enabling it will result in the layer rendering directly to the canvas.

Now, since the layer flip-flops between rendering into the two textures, the texture used to render the last frame back into the layer needs to flop-flip. The layer kit does this internally by hot swapping the framebuffer and WebGL texture of getRenderTexture() each frame.

// Create a sprite to render the last frame of the layer
const lastFrameSprite = new PIXI.Sprite(layer.getRenderTexture());

// Apply an alpha so the last frame decays a bit
lastFrameSprite.alpha = 0.6;

// Render the last frame back into the layer
layer.addChild(new PIXI.Sprite(layer.getRenderTexture()));

// Render the layer into the main stage
stage.addChild(new PIXI.Sprite(layer.getRenderTexture()));

In the above snippet, sprites are created using the layer’s texture. When it runs, the sprites are actually flip-flopping between two different textures each frame. See it in action by Ivan’s example:

Thanks to Mat Grove’s work, PixiJS 6.1.0 will ship with support for Uniform Buffer Objects, a WebGL 2 optimization to make uniform uploads faster.

Context

UBOs are handles to GPU buffers that store uniform data. They can be attached to a shader in a single step, without needing to upload each uniform field individually. If you share uniforms between multiple shaders, this can be used to reduce uploads of relatively static data.

Theoretically, you can optimize filters with UBOs. The common uniforms passed to Filter don’t change between them: inputSize, inputPixel, inputClamp, outputFrame.

Usage

To use UBOs in a shader, you’ll need to use GLSL 3 ES.

#version 300 es
#define SHADER_NAME Example-Shader

precision highp float;

To migrate an existing GLSL 1 shader (the default), you need to use the in keyword instead of attribute, out instead of varying in the vertex shader, in instead of varying in the fragment shader, and then create an out variable in the fragment shader instead of using gl_FragColor.

You can then move some of your uniforms into a UBO block:

uniform location_data {
  mat3 locationMatrix;
};

In this example, you can reference the locationMatrix uniform directly.

void main(void) {
  mat3 matrix = locationMatrix;
}

To upload the UBO uniforms, you need to add an UniformBufferGroup in PixiJS in your shader’s uniform.

import { Shader, UniformBufferGroup } from 'pixi.js';

Shader.from(vertexSrc, fragmentSrc, {
    location_data: UniformBufferGroup.uboFrom({
        locationMatrix: new Matrix().toArray(),
    }),
})

UniformBufferGroup.uboFrom creates a “static” UBO. If you ever change a value in it, you’ll need to update it.

Here’s an example that applies a gradient color to a texture using a UBO:

When should I use UBOs?

UBOs are useful if you have multiple shaders that share static uniform data. If your uniforms are dynamic and change very often, UBOs will not be much of an optimization.