Publication

Scene Designer: a Unified Model for Scene Search and Synthesis from Sketch

Leo Ribeiro, Tu Bui, John Collomosse, Moacir Ponti

A novel method for searching and generating images using free-hand sketches of scene compositions; i.e. drawings that describe both the appearance and relative positions of objects. A single unified model to learn cross-modal search embedding for matching sketched compositions to images, and an object embedding for layout synthesis. A graph neural network (GNN) followed by Transformer under our novel contrastive learning setting allow learning correlations between object type, appearance and arrangement, synthesizing coherent scene layouts, whilst also delivering state of the art sketch based visual search of scenes. 

• Code here: https://github.com/leosampaio/scene-designer

SC-GlowTTS - an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model

Edresson Casanova, Christopher Shulby, Eren Gölge, Nicolas Müller, Frederico Oliveira, Arnaldo Candido Junior, Anderson Soares, Sandra Aluisio, Moacir A. Ponti

SC-GlowTTS is an efficient zero-shot multi-speaker text-to-speech model that improves similarity for speakers unseen during training. We propose a speaker-conditional architecture that explores a flow-based decoder that works in a zero-shot scenario. As text encoders, we explore a dilated residual convolutional-based encoder, gated convolutional-based encoder, and transformer-based encoder. Additionally, we have shown that adjusting a GAN-based vocoder for the spectrograms predicted by the TTS model on the training dataset can significantly improve the similarity and speech quality for new speakers. Our model converges using only 11 speakers, reaching state-of-the-art results for similarity with new speakers, as well as high speech quality. 

• Demo here: https://edresson.github.io/SC-GlowTTS/

Foreseeing future falls with accelerometer features in active community-dwelling older persons with no recent history of falls

Patricia Bet . P. C. Castro .  Moacir A. Ponti

We investigate the risk of fall in active and healthy community-dwelling living older persons with no recent history of falls, using a single accelerometer and variants of the Timed Up and Go (TUG) test. A prospective study was conducted with 74 healthy non-fallers older persons. After collecting acceleration data from the participants at the baseline, the occurrence of falls (outcome) was monitored quarterly during one year. A set of frequency features were extracted from the signal and their ability to predict falls was evaluated. The results confirms previous evidence that accelerometer features can better estimate fall risk, and support potential applications that try to infer falls risk in less restricted scenarios, even in a sample stratified by age and gender composed of active and healthy community-dwelling living older persons.

• this research result was recipient of a Google Latin America Research Award

Sketchformer: Transformer-based Representation for Sketched Structure

Leo Sampaio Ferraz Ribeiro • Tu Bui • John Collomosse • Moacir Ponti

Sketchformer is a novel transformer-based representation for encoding free-hand sketches input in a vector form, i.e. as a sequence of strokes. Sketchformer effectively addresses multiple tasks: sketch classification, sketch based image retrieval (SBIR), and the reconstruction and interpolation of sketches.

https://www.paperswithcode.com/paper/sketchformer-transformer-based-representation

Learning image features with fewer labels using a semi-supervised deep convolutional network

Fernando dos Santos . Cemre Zor . Josef Kittler.  Moacir A Ponti

A novel semi-supervised deep network training strategy that comprises a convolutional network and an autoencoder using a joint classification and reconstruction loss function. We show our network improves the learned feature embedding when including the unlabelled data in the training process. The results using the feature embedding obtained by our network achieve better classification accuracy when compared with competing methods, as well as offering good generalisation in the context of transfer learning. 

Analyzing college student dropout risk prediction in real data using walk-forward validation

Rodolfo Santos, Moacir A. Ponti, Kamila Rios Rodrigues

College dropout is a concern for educational institutions since it directly impacts educational management and academic results, as well as being directly related to social problems. Although machine learning techniques were shown to have potential for this task, there are many steps involved when it comes to the use of real data. We used data from 32.892 students enrolled between 2008 and 2020 from all courses offered by a public high-education institution. A protocol for data preparation is proposed and found to be more important than designing complex classifiers. We present guidelines when modelling a college dropout classification task using a public university data and experiments using Walk-Forward Validation that showed the predictive capacity for the first years.

https://drive.google.com/file/d/1AKD6cU8_PldhGNtuqBZAaZhgOCZ83MhI/view?usp=drive_link