Weakly-Supervised Action Segmentation and Alignment via Transcript-Aware Union-of-Subspaces Learning

• We address learning to segment actions from weakly-annotated videos (videos accompanied by ordered list of actions) by learning a deep union of subspaces network.


• We provide the Python/Pytorch implementation of the method. When using the code in your research work, you should cite the following papers:
  
  Z. Lu and E. Elhamifar, Interaction Compass: Multi-Label Zero-Shot Learning of Human-Object Interactions via Spatial Relations,
  International Conference on Computer Vision (ICCV), 2021.
  
  

Project Page

Interaction Compass: Multi-Label Zero-Shot Learning of Human-Object Interactions

• We developed a compositional learning model to recognize seen and unseen human-object interactions by learning and leveraging their spatial relations.


• We provide the Python/Pytorch implementation of the method. When using the code in your research work, you should cite the following papers:
  
  D. Huynh and E. Elhamifar, Interaction Compass: Multi-Label Zero-Shot Learning of Human-Object Interactions via Spatial Relations,
  International Conference on Computer Vision (ICCV), 2021.
  
  

Project Page

Learning to Segment Actions from Visual and Language Instructions

• We address the problem of unsupervised action segmentation and feature learning in procedural task videos using both visual and language instructions. Our key contributions include proposing i) a Soft Ordered Prototype Learning (SOPL) method for learning visual and linguistic prototype sequences corresponding to subtasks (key-steps) in videos, and ii) a new class of alignment methods, referred to as Differentiable Weak Sequence Alignment (DWSA), for weak alignment of visual and linguistic prototype sequences while allowing self-supervised feature learning.


• We provide the Python/Pytorch implementation of the method. When using the code in your research work, you should cite the following papers:
  
  Y. Shen, L. Wang, E. Elhamifar, Learning to Segment Actions from Visual and Language Instructions via Differentiable Weak Sequence Alignment,
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  
  

Project Page

Introvert: Human Trajectory Prediction via conditional 3D attention

• We developed Introvert, a deep model that predicts human trajectory and path based on their observed trajectory and the dynamic scene context, captured via a conditional 3D visual attention mechanism.


• We provide the Python/Pytorch implementation of the method. When using the code in your research work, you should cite the following papers:
  
  N. Shafiee, T. Padir and E. Elhamifar, Introvert: Human Trajectory Prediction via Conditional 3D Attention,
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  
  

Project Page

Compositional Zero-Shot Learning via Fine-Grained Dense Feature Composition

• We address the problem of zero-shot learning using a generative fine-grained feature composition method.


• We provide the Python/Pytorch implementation of the method. When using the code in your research work, you should cite the following papers:
  
  D. Huynh, E. Elhamifar, Compositional Zero-Shot Learning via Fine-Grained Dense Feature Composition,
  Neural Information Processing Systems (NeurIPS), 2020.
  
  

Project Page

Unsupervised Procedure Learning from Instructional Videos

• We address unsupervised learning of complex procedural tasks from instructional videos.


• We provide the Python/Pytorch implementation of self-supervised multi-task procedure learning. The code can learn from videos of one task or multiple tasks. When using the code in your research work, you should cite the following papers:
  
  E. Elhamifar, D. Huynh, Self-Supervised Multi-Task Procedure Learning from Instructional Videos,
  European Conference on Computer Vision (ECCV), 2020.
  
  E. Elhamifar, Z. Naing, Unsupervised Procedure Learning via Joint Dynamic Summarization,
  International Conference on Computer Vision (ICCV), 2020.

Project Page

Shared Attention for Multi-label Zero-shot Learning

• We address zero-shot multi-label learning for recognition of a large number of seen and unseen labels using a shared multi-attention method with a novel training mechanism.


• We provide the Python/Tensorflow implementation of LESA. When using the code in your research work, you should cite the following paper:
  
  D. Huynh, E. Elhamifar, A Shared Multi-Attention Framework for Multi-Label Zero-Shot Learning,
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

Project Page

Fine-Grained Generalized Zero-Shot Learning via Dense Attribute-Based Attention

• We develop a zero-shot fine-grained recognition method with the ability to localize attributes using a dense attribute-based attention and embedding mechanism.


• We provide the Python/Pytorch implementation of DAZLE. When using the code in your research work, you should cite the following paper:
  
  D. Huynh, E. Elhamifar, Fine-Grained Generalized Zero-Shot Learning via Dense Attribute-Based Attention,
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

Project Page

Interactive Multi-Label CNN Learning with Partial Labels

• We address efficient end-to-end learning of multi-label CNN classifiers with partial labels using an interactive learning framework.


• We provide the Python/Tensorflow implementation of IMCL. When using the code in your research work, you should cite the following paper:
  
  D. Huynh, E. Elhamifar, Interactive Multi-Label CNN Learning with Partial Labels,
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

Project Page

Deep Summarization via Supervised Facility Location (SupFL)

• Supervised Facility Location (SupFL) is a deep representation learning algorithm for learning to find representative data from ground-truth summaries.


• We provide a Python implementation of SupFL. When using the code in your research work, you should cite the following paper:
  
  C. Xu, E. Elhamifar, Deep Supervised Summarization: Algorithm and Applications to Learning Instructions,
  Neural Information Processing Systems (NeurIPS), 2019.

Download Code

Dissimilarity-based Sparse Subset Selection (DS3)

• Dissimilarity-based Sparse Subset Selection (DS3) is an algorithm based on simultaneous sparse recovery for finding data/model representatives from a large collection of data/models.


• We provide a MATLAB implementation of DS3. When using the code in your research work, you should cite the following paper:
  
  E. Elhamifar, G. Sapiro and S. Sastry, Dissimilarity-based Sparse Subset Selection,
  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2016.

Download Code

Sparse Subspace Clustering (SSC)

• Sparse Subspace Clustering (SSC) is an algorithm based on sparse representation theory for segmentation of data lying in a union of subspaces. For more information please visit the SSC research page.


• We provide a MATLAB implementation of SSC. When using the code in your research work, you should cite the following paper:
  
  E. Elhamifar and R. Vidal, Sparse Subspace Clustering: Algorithm, Theory, and Applications,
  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2013.

Download Code (ADMM)
Download Code (CVX)

Sparse Manifold Clustering and Embedding (SMCE)

• Sparse Manifold Clustering and Embedding (SMCE) is an algorithm based on sparse representation theory for clustering and dimensionality reduction of data lying in a union of nonlinear manifolds.


• We provide a MATLAB implementation of SMCE algorithm. When using the code in your research work, you should cite the following paper:
  
  E. Elhamifar and R. Vidal, Sparse Manifold Clustering and Embedding,
  Neural Information Processing Systems (NIPS), 2011.

Download Code

Sparse Modeling Representative Selection (SMRS)

• Sparse Modeling Representative Selection (SMRS) is an algorithm based on sparse multiple-measurement-vector recovery theory for selecting a subset of data points as the representatives.


• We provide a MATLAB implementation of SMRS algorithm. When using the code in your research work, you should cite the following paper:
  
  E. Elhamifar, G. Sapiro, and R. Vidal, See All by Looking at A Few: Sparse Modeling for Finding Representative Objects,
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.

Download Code

Block-Sparse Subspace Classification (BSSC)

• Structured-Sparse Subspace Classification is an algorithm based on block-sparse representation techniques for classifying multi-subspace data, where the training data in each class lie in a union of subspaces.


• We provide a MATLAB implementation of the algorithm. When using the code in your research work, you should cite the following paper:
  
  E. Elhamifar and R. Vidal, Robust Classification using Structured Sparse Representation,
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.

Download Code