Martin Guay

I am a research scientist in visual computing working on markerless motion capture using deep learning. I am excited about combining graphics with vision: on the one hand, to create large datasets for machine learning, and on the other, for unlocking new capabilities in Augmented Reality (AR). For example, putting watertight clothes onto people in AR, or having virutal characters interact with the real world in a realistic way.

Intresting note: I spent last year working on a smart 3D virtual assistant (that you can talk with and interact with).

My original background is in graphics and games. After my bachelor's, I spent two years working on different video games, which led me to doing a master's on real time fluid simulation and a ph.d. on character animation.


Projects and Publications

Pose and shape estimation from images and video

Marker-less motion capture for animation and virtual production.

Putting together state-of-the-art body tracking from a single camera using deep learning,
leveraging 3D motion capture and lot's of 3d models.

Capturing Animal Motion from Video
There is really no available dataset of animal images labelled with 3D position information.
This is learned from synthetic data, using computer generated imagery.

Augmented Reality


AR Costumes: Automatically Augmenting Watertight Costumes from a single RGB Image
ACM SIGGRAPH Conference on Virtual Reality and Applications in Industry (VRCAI), 2018.
Christoph Maurhofer, Gokcen Cimen, Mattia Ryffel, Robert W. Sumner, Martin Guay


AR Poser: Automatically Augmenting Mobile Pictures with Digital Avatars Imitating Poses
Computer Graphics, Visualization, Computer Vision and Image Processing (CGVCVIP), 2018.
Gokcen Cimen, Christoph Maurhofer, Robert W. Sumner, Martin Guay


Interacting with a Fully Simulated, Self-Balancing Bipedal Character in Augmented and Virtual Reality
Encyclopedia of Computer Graphics and Games, 2018.
Dominic Borer, Simone Giguarri, Robert W. Sumner, Martin Guay


Modeling Physical Interactions with Real World Objects:
Interacting with Intelligent Characters in AR

Workshop on Artificial Intelligence in Virtual and Augmented Reality, 2017.
Gokcen Cimen, Ye Yuan, Stelian Coros, Robert W. Sumner, Martin Guay

Control of Digital Avatars, Parametric Character Motion


Animating an Autonomous 3D Conversation Avatar

Computer Graphics, Visualization, Computer Vision and Image Processing (CGVCVIP), 2018.
Dominik Borer, Dominic Lutz, Robert W. Sumner, Martin Guay

Automatic Animation of a Digital Band from Midi Signals
Live Demo @ Symposium on Computer Animation (SCA), 2016.
Renzo Roth & CGL Ph.D. Students, Martin Guay

Variational Autoencoder Applied to Human Walking Motion: Automatically Learning
an Intuitive Animation Interface from a Collection of Human Motion Clips

Computer Graphics, Visualization, Computer Vision and Image Processing (CGVCVIP), 2017.
Marcel Ludi, Martin Guay, Brian McWilliams, Robert W. Sumner


Keys-to-Sim: Transferring Hand-Crafted Key-framed Animations to Simulated Figures using
Wide Band Stochastic Trajectory Optimization

Computer Graphics and Visual Computing (CGVC), 2018.
Dominic Borer, Robert W. Sumner, Martin Guay
PaperVideoOnline Policy Learning PDFOnline Policy Video

An Intuitive Interface for Human Performance Tracking with Simulated Characters
Computer Graphics, Visualization, Computer Vision and Image Processing (CGVCVIP), 2017.
Gokcen Cimen, Martin Guay, Stelian Coros, Robert W. Sumner

Authoring Character Motion

The Line of Action: an Intuitive Interface for Expressive Character Posing
SIGGRAPH Asia 2013.
Martin Guay, Marie-Paule Cani, Remi Ronfard

Space-time sketching of character animation
Martin Guay, Remi Ronfard, Michael Gleicher, Marie-Paule Cani

Adding dynamics to sketch-based character animation
Sketch-based modeling and interfaces (SBIM) 2015.
Martin Guay, Remi Ronfard, Michael Gleicher, Marie-Paule Cani

Authoring Motion Cycles
Symposium on Computer Animation (SCA), 2017.
Loïc Ciccone, Martin Guay, Mauri Nitti, Robert W. Sumner

Flow Curves: an Intuitive Interface for Coherent Scene Deformation
Pacific Graphics, 2016.
Loïc Ciccone, Martin Guay, Robert W. Sumner

Programmable Animation Texturing using Motion Stamps
Pacific Graphics, 2016.
Antoine Milliez, Martin Guay, Marie-Paule Cani, Markus Gross, Robert W. Sumner

Physics Simulations on the GPU

Simple and Fast Fluids
GPU Pro 2, 2011.
Martin Guay, Fabrice Colin, Richard Egli

Simple rasterization-based liquids
GPU PRo 5, 2013.
Martin Guay


Screen space animation of fire
SIGGRAPH Asia, 2011.
Martin Guay, Fabrice Colin, Richard Egli

Multi-dimensional Unconditionally Stable Alternate Direction Explict (ADE) Scheme for the Convection-Diffusion Equation
Conference on Nonlinear Evolution Equations and Wave Phenomena: Computation and Theory, 2010.
Martin Guay, Fabrice Colin, Richard Egli

Video Games I worked on

A Game of Thrones: Genesis (PC)
Cyanide Studio Montreal, 2011.

Blood Bowl
Cyanide Studio Montreal, 2009.

Students and Collaborators

Dominik Borer, Ph.D. Student on Motion Models for 3D Computer Animation
Jakob Buhmann, Post-Doc on Motion Capture
Prof. Bob Sumner, Supervisor

Christoph Maurhofer, Master's on Poser and Costumes in AR.
Gökcen Cimen, Ph.D. on Character Motion Models for Interaction in AR and VR.
Victor Cornillere, Autonomous 3D Avatar Chat Bots.
Dominic Lutz, 3D animator on chat bot avatars.
Stelian Coros, Prof. on Synthetising Quadruped Motion.
Ye Yuan, Synthetising Quadruped Motion.
Simone Guggiari, Bachelor's on Interacting with a self-Balancing Character in AR and VR.
Dominik Borer, Master's on Trajectory Optimization for Simulated Character Control
Loïc Ciccone, Ph.D. on Character Animation Authoring.
Renzo Roth, Master's on Animated Music Band.
Marcel Lüdi, Master's on Deep Learning for Character Animation.

Remi Ronfard, Ph.D. co-advisor at INRIA.
Marie-Paule Cani, Ph.D. co-advisor at INRIA.
Richard Egli, Master's advisor at Sherbrooke U.