Blog about computer vision, AR and all in between

How To Create Face Filters Like Snapchat in Banuba Studio. Lesson 2_Effect Construction

Posted by Tolya Kozhemiako on Mar 5, 2019 5:37:28 PM

This tutorial will guide you through creating your own face filters like Snapchat in Banuba Studio. As an example, I use a selfie filter of HooliganGirl. I designed its geometry and textures in Maya and Substance Painter. You can design your camera filters and effects or motion stickers using any editor that you like.

 

 

NOTE: Banuba Studio is our proprietary engine. It allows designers to create, adjust and test in real time their AR camera effects before uploading them to their apps via FaceAR SDK. To access the Studio, you need to use FaceAR SDK in your filter app.

See FaceAR SDK

ALSO READ: Lesson 1_ Banuba Studio Overview

banuba studio _ camera effect construction

Face filter construction process

 

Step 1. Create root folder + SRC folder

Create the root folder. Name it the way you name your effect. It’s important that you used only latin letters and numbers. No spaces either – use underscores.

Create the SRC folder in your root folder. Here where we’ll store all the source files like 3D models in FBX format and textures in PNG.

Step 2. Apply materials to geometry

To ensure a good-looking effect, we need to set its geometry materials. I use 3 materials:

  • mat_Hair for hair.
  • mat_Main for hat, glasses, headsets and animated waves.
  • mat_Morph to modify our basic face model by slimming down the chin and enlarging the eyes.

We don’t need the back of the model — it’s either not detailed, or missing. It saves the number of polygons and optimize resources but doesn’t influence filter experience as the user always sees the front part.

Step 3. Export geometry

Now, let’s export geometry from our 3D editor to the SRC folder. Use default export settings and set scene scale in centimeters.

  1. Select the morphing geometry, click “Export selection” and save them to morph.fbx file.
  2. Select the geometry of glasses, headsets, animation and the hat. Click “Export selection”, tick the animation option and name the file HooliganGirl.fbx.
  3. Export the hairs and name the file Hair.fbx.

Once all files are exported we can construct the effect.

Step 4. Face filter construction

Start the Constructor and open your root folder. You see 4 tabs here.

  • TEXTURES. It’s empty now as we don’t have textures in our SRC folder.
  • FBX MODELS shows our geometry files.
  • MEDIA. You can add audio here.
  • RENDERING is the main tab where we set our effect.

Studio rendering tab to create camera filters and effects

Here we define the draw order putting transparent surfaces at the bottom for they would not overlap visible objects.

We set cutting off geometry using morph_mat, FACE_Z and FACE_MOUTH_Z layer. I’ll tell you more about each layer in the following lessons.

Now, we enable the material of the morph geometry and place it above the geometry that we want to cut off.

In the material settings, we select Template Cut and use the GLFX_USE_UVMORPH option.

Now you can press the button ‘Construct’ and see how your effect looks like in a digital camera in real time.

Step 5. Effect geometry and animation preview

Jump to your root folder. You’ll see one more sub-folder added named HooliganGirl which contains your constructed effect.

Drag and drop this folder to the ViewPort. You’ll see your current effect’s geometry and animation.

Step 6. Morphing setup

Now we enable morphing. Find the warp_mesh option in the Rendering tab and specify the morph geometry file.

To make the effect deform only our face, and keep the 3d geometry unchanged as in the 3D editor, we turn on the WARP_IMAGE layer and move it to the top.

Step 7. Add textures

We have 4 textures for each material that ensure a good-looking effect:

  • Base Color
  • Metallic
  • Open GL tangent space Normal
  • Roughness 

Once you finished working with textures you’d better convert them to .KTX format. It’s a native format for graphical processor and this way your effect will load faster.

So we copy and save PNG textures to the SRC folder. Press Reload and here they appear in the tab ‘Textures’.

To ensure a good-looking effect we add one more texture in HDR (high dynamic range).

This texture surrounds the entire scene, and we use it to reflect surfaces in the scene.

To enable it, tip the HDR option in the main material and hair material.

Reload the Constructor and see our texture in the list.

Now we assign textures for each material by uploading texture files in KTX format.

Reload the constructor to see textures added.

Step 8. Add beautification

Now we set beautification. Jump to the field Draw order, move the Retouch layer atop and enable it.

To adjust the color of the eyes and teeth we use LUT textures. It’s important that these files were named with the lut3d_ prefix.

Copy these files to the build folder. For makeup we have a PNG file with eyelashes, eyebrows and hair shadows. We also have softlight texture for tone correction. And finally, we add the effect of sparkling eyes.

To set the beautification layer we assign the needed textures in Rendering tab and press Construct. Now we see makeup, teeth whitening and skin tone correction.

Step 9. Add audio

Finally, we add music in the MEDIA tab.

Press 'Construct' and you’ll hear the background audio in your effect.

 

These are the basic settings for an effect construction. Next lessons we’ll explore the possibilities of each function in detail. See you there.

 

 

 

 

 

Topics: Face filters, Studio, AR SDK