TCD research team

Abdullah Bulbul
Dr. Abdullah Bulbul
Marie Curie Research Fellow
FP7-PEOPLE-2013-IAPP (612334)
recruited in TCD
20 months: 09/11/2014-08/06/2016
Cyril Bourges
Cyril Bourges
Marie Curie fellow - TCD Research Assistant
FP7-PEOPLE-2013-IAPP (612334)
seconded from Tapastreet to TCD
20 months: 01/03/2014-30/06/2015 and 01/09/2015-31/12/2015
Thomas Vulin
Marie Curie fellow - TCD Research Assistant
FP7-PEOPLE-2013-IAPP (612334)
seconded from Tapastreet to TCD
4 months: 13/02/2017-12/06/2017

Project description and workpackages in TCD

WP1: Video Summarisation

This package aims at developing video summarization algorithms (vsa) for amateur social media video to display local event highlights as they occur anywhere in the world. The first brute force strategy to create a gif summary by down-sampling videos, will look at the different resolutions that could be proposed to the user with the requirement that the information displayed is of good quality for a quick understanding of the content of the input video. software 1.0 will be used to process a database of videos and create short summaries at different resolutions. The analysis of the low level content (e.g. motion, colour) of the video can be performed with metrics defined in information theory. We propose to create a smart software prototype for creating summaries that will use these metrics for measuring automatically the content of videos. These metrics will be used to automatically select what are the images in the video that should be retained to be part of the summary, and also this will help in selecting the best spatial resolution automatically. Some artifacts (hand shake, blur, and occlusion) that often occur in amateur videos will be dealt with to improve the quality of the summaries.

MCFs: Zbigniew Zdziarski, Cyril Bourges


WP2: 3D Scene Rendering

WP2 aims at developing automatic local 3d scene rendering algorithms (sra) leveraging public geo-located social media photos of a particular location. merging several images or videos to create an augmented image can be a step further towards creating a good quality summary. this work package led by tcd will look at merging several images and/or videos recorded at the same place at the same time for creating a 3d rendering of a scene. using location information embedded with the input images and videos, and potentially using additional 3d content available (e.g. google maps in 3d), this work package will look at computing local descriptors in the images, suitable for image stitching and 3d reconstruction, but also for image classification (wp5). we propose to use a modeling based on the generalised relaxed radon transform (gr2t) to estimate a probability density function of the 3d location and colour. an animated gif will then be created by moving a virtual camera into the scene and a perceptual testing using questionnaires and eye tracking technique will be used to assess the quality of the rendering. the path of the virtual camera will be automatically chosen such that the summary is both informative, and visually pleasing. metrics from information theory will be used to assess the information content of the summary.

MCF: Abdullah Bulbul


WP6: Implementation

WP6 is to develop strategies and methods for implementation of video summarisation and automated 3d scene rendering into tapastreetís social media search engine platform. Research effort in wp6 is led by tapastreet and will be looking at what strategy (e.g. cloud computing, parallel processing, gpu processing, etc.) can be used for a fast reliable implementation of wp1 & wp2 into the tapastreet platform. video summarization will need to be processed rapidly, and therefore some solutions may be better adapted than others. a hierarchical approach can be considered where about simple summaries are proposed and then replaced when the smarter ones becomes available. beside the strategy for processing the information in a timely fashion, this workpackage will look at storage of the summaries, and easy access via mobile platform.

MCFs: Zbigniew Zdziarski, Cyril Bourges

TCD publications

Journal papers
  1. Social Media based 3D Visual Popularity
    A. Bulbul & R. Dahyot, Computer & Graphics, 2017
  2. Populating virtual cities using social media, A. Bulbul & R. Dahyot, computer animation and virtual worlds journal 2016, doi:10.1002/cav.1742.
    Also presented at computer animation and social agents (casa) conference, geneva switzerland, may 2016. (presenter: a. bulbul)
Conference proceedings
  1. Deep shape from a low number of silhouettes, X. Di, R. Dahyot, M. Prasad, eccv workshop geometry meets deep learning, Amsterdam, 9th october 2016 (presenter: x. di)
  2. [PDF] Social Media based Up-to-Date 3D Modeling and Visualization
    A. Bulbul and R. Dahyot, Conference on Visual Media Production, London, November 2015. DOI:10.1145/2824840.2824860, (Presenter: A. Bulbul)
  3. [PDF] 3D Reconstruction of Reflective Spherical Surfaces from Multiple Images
    A. Bulbul, M. Grogan and R. Dahyot, Irish Machine Vision and Image Processing conference, pages 19-26, (Permanent link to full book: ISBN 978-0-9934207-0-2, August 2015. (Presenter: A. Bulbul)
  4. [PDF] L2 Registration for Colour Transfer
    M. Grogan, M. Prasad and R. Dahyot, European Signal Processing Conference (Eusipco), ISBN 978-0-9928626-4-0, Nice France, September 2015. DOI:10.1109/EUSIPCO.2015.7362799 (Presenter: R. Dahyot)
  5. [PDF] L2 registration for Colour Transfer in Videos
    M. Grogan and R. Dahyot, short paper in Conference on Visual Media Production, London, November 2015. DOI:10.1145/2824840.2824862 (Presenter: M. Grogan)
  6. [PDF] Information visualisation for social media analytics
    R. Dahyot, C. Brady, C. Bourges and A. Bulbul, International Workshop on Computational Intelligence for Multimedia Understanding, Prague, Czech Republic, 29-30 Oct. 2015. DOI:10.1109/IWCIM.2015.7347082 and some Code on GitHub (Presenter: R. Dahyot)
  7. [PDF] On summarising the 'here and now' of social videos for smart mobile browsing
    Z. Zdziarski, C. Bourges, J. Mitchell, P. Houdyer, D. Johnson, and R. Dahyot, International Workshop on Computational Intelligence for Multimedia Understanding, Paris, 1-2 Nov. 2014. DOI:10.1109/IWCIM.2014.7008797 (Presenter: Z. Zdziarski)
  8. [PDF] An Architecture for Social Media Summarisation
    Z. Zdziarski, J. Mitchell, P. Houdyer, D. Johnson, C. Bourges and R. Dahyot, Irish Machine Vision and Image Processing Conference, Derry-Londonderry, Northern Ireland, 27-29 August 2014. (Presenter: C. Bourges)
  9. [PDF] Mesh from Depth Images Using GR2T
    M. Grogan and R. Dahyot, Irish Machine Vision and Image Processing Conference, Derry-Londonderry, Northern Ireland, pp. 15-20, 27-29 August 2014. (Presenter: M. Grogan)
  10. [PDF] GR2T Vs L2E with nuisance scale
    R. Dahyot, International Conference on Pattern Recognition (ICPR), Sweden, August 2014. DOI:10.1109/ICPR.2014.662 (Presenter: R. Dahyot)
Other related publications
  1. On Using Deep Learning for Sentiment Analysis,
    Conor Brady, final year project Computer Science (supervisor Rozenn Dahyot), Trinity College Dublin, 2014-15. Some Code on GitHub