Inside Face Filters: Vision, Segmentation, and Shaders in Action

Usually, an iOS developer implements UI and works with APIs, but what lies beyond such tasks?

There is the world of computer vision and graphics, where mathematics ceases to be abstract: linear algebra and trigonometry bring facial expressions to life, while optimization transforms into GPU shaders.

In the talk, we will break down how face filters work under the hood: from face tracking and segmentation through Core ML to realistic filters.

We will cover two approaches:

• Based on face landmarks with point-wise processing of different facial zones.
• Based on segmentation models with color processing of different segments.

Special attention will be paid to the technical implementation: how Vision works, how segmentation Core ML models are structured, and how to write custom shaders for CIKernel to achieve maximum performance.