D-Lib Magazine

January/February 2012
Volume 18, Number 1/2

A Report from the 2011 ICSTI Workshop on Multimedia and Visualization Innovations for Science

Tomasz Neugebauer
Concordia University Libraries
tomasz.neugebauer@concordia.ca

doi:10.1045/january2012-neugebauer

Abstract

Innovations in science oriented web multimedia, large-scale data exploration and visualization, speech recognition, and video and image indexing and analysis, offer opportunities for accelerating scientific discovery. The 2011 ICSTI Workshop on Multimedia and Visualization Innovations for Science was held on February 8, 2011, hosted jointly by the International Council for Scientific and Technical Information and Microsoft Corporation. It offered a unique opportunity to discuss the current leading-edge projects in scientific multimedia and visualization in government, business, industry and academia.

Introduction

The February 2011 one-day workshop on Multimedia and Visualization Innovations for Science, hosted jointly by the International Council for Scientific and Technical Information (ICSTI) and Microsoft Corporation, was nothing short of spectacular. Although the videos of the presentations are available to ICSTI members only, some of the PowerPoint presentations have been posted at http://www.icsti.org/spip.php?rubrique42. This report from the workshop is a descriptive review of its contents with hyperlinks to the various examples of leading-edge multimedia and visualization projects.

The presenters came from government, business and industry, publishers and academia. The workshop was divided into four sessions:

• An emphasis on interactive multimedia and visualization
• Tools and technologies for communication and problem solving
• Accelerating discovery through multimedia and visualization
• Combining and refining data to reach new discovery

Interactive multimedia and visualization

The workshop opened with a presentation by Wilmot Li from Adobe Systems, who presented impressive software prototypes for different types of visualization techniques. The origin of this research included a thorough cataloguing of existing techniques and conventions, resulting in the identification of the following categories:

• Cutaways—shapes such as a box, window and wedge are removed from a drawing to reveal underlying structure
• Exploded Views—revealing layers that may otherwise be invisible due to compacted organization of parts
• "How things work"—with the use of motion arrows, frame sequences, animations, etc.

The challenge is to visualize both functional and spatial information and deal with the problem of occlusions, visual clutter and simultaneous motions. Wilmot Li proceeded to demonstrate proprietary visualization software prototypes that use these three types of techniques. The most impressive aspect of the presentation was the demonstration of the fact that multiple techniques can be used simultaneously in the same visualization.

Robert M. Hanson demonstrated Jmol, an "open-source Java viewer for chemical structures in 3D with features for chemicals, crystals, materials and biomolecules." It is capable of animation, the display of vibrations, surfaces, orbitals, measurements of distance and angle, and more. The use of Jmol for the publication of interactive figures in online-accessible journal articles allows the reader to view a protein or crystal structure, for example, from different perspectives and to save the structure data to their own computer or forward it to colleagues.

Open source advocates can confidently cite Jmol in their examples of success stories. The feature set is rich and mature. Jmol runs on cross-platform Java; it is interoperable with many data file formats (such as CIF/mmCIF, CML, PDB, and many more) and exportable to many formats (such as jpg, gif, pdf, png) and applications (such as Maya, POV-Ray). Jmol has been used on websites and for the production of figures in academic journals such as Nature.

One of the features that Robert Hanson demonstrated was the relatively recent (starting with Jmol 12.1.6) interoperability with the SMILES-to-3D and database service at the NIH Cactus server. Jmol can load 3D models based simply on SMILES strings or chemical identifiers such as chemical names, CAS registry numbers, and InChI keys. For example, load $CCC loads the structure of propane, load $C1CCCCC1 loads a chair cyclohexane model, and load $tylenol loads a model of acetaminophen (see the load command documentation).

Michael Ackerman, from the National Library of Medicine (NLM), reported on the data visualization application developed by The Optical Society (OSA) in cooperation with Kitware Inc., and the NLM. The OSA Interactive Science Publishing (ISP) software allows for interactive viewing of a wide range of 2D and 3D image data formats.

The viewer software is available for download from Optics InfoBase. The Optics InfoBase advanced search makes it possible to list all of the articles that contain the ISP multimedia component, a total of 39 articles in January 2012. The architecture is such that the datasets exist as independently catalogued entities in MIDAS, "a collection of server, client, and stand-alone tools for data archiving, analysis, and access", but the plan is to eventually host the datasets at NLM. The datasets need to be independently discoverable and citeable.

The NLM is working in partnership with OSA, beginning with the pilot project (including special journal issues such as Optics Express) on biomedical research topics that include datasets visualized with the ISP software. The project involves collection of user feedback from readers, article authors and the peer reviewers. The results include complaints from the peer reviewers about being overwhelmed by the data, as well as complaints about the difficulties encountered with having to install separate software, and in using the interface.

Tools and technologies for communication and problem solving

Curtis Wong, from Microsoft Research, presented WorldWide Telescope. This visualization environment allows users to navigate trillion-pixel images of galaxies, planets, etc. It is possible to load datasets into the environment, allowing for the visualization of new datasets. Curtis Wong demonstrated this by loading a dataset of earthquake location/time information and was able to create an animated visualization of the earthquakes using the 3D rendering of planet Earth.

WorldWide Telescope also allows experts and educators to prepare virtual tours. One of the features that makes these virtual tours more interesting than videos is that the user can stop the tour at any moment, and they are paused within context; they can navigate away to an alternate path from the tour.

Tim Smith, from CERN, described many examples of how multimedia is used by CERN. One of the primary uses of multimedia has been to engage and communicate with the public. There is an artistic installation, the Colliderscope, of 96 LED lights distributed over the facade of the Niels Bohr Institute that reproduce the actual signals coming from particle collisions at the Transition Radiation Detector (TRT) in ATLAS. In addition to visualization, real and simulated scientific data can also be turned into sound and music. A member of the ATLAS collaboration was one of the initiators of the Large Hedron Collider (LHC) sonification project. The Science & Technology Facilities Council funded project LHCSound resulted in the sonification of Higgs Boson decay in ATLAS and proton-proton collision in LHC.

For a younger audience, CERN developed CERNland, an award-winning site featuring educational games and videos. A million viewers tuned in to watch a live webcast from CERN of the launch of the Large Hadron Collider (LHC). The "multimedia & outreach" section of CERN Document Server contains over a thousand videos, including movies such as "Animation of one of the first heavy-ion collisions producing muons in the CMS experiment". In addition to engaging the public, multimedia is increasingly used to enhance CERN publications with supporting data, post-publication reviews and comments, as well as presentations and videos. Tim Smith also reported that they are experimenting with GIFT (GNU Image Finding Tools) on the Ivenio software that powers the CERN Document Server so as to extract figures, images and plots from the article contents and place them alongside the abstract information and in additional tabs on the browse screens.

Peter Tu, from GE Global Research, presented "Video Analytics and its Applications to Science and Technology". Although ultra-sound and microscopic imagery were mentioned as applications, the presentation was focused on applications in pathology, forensic analysis, facial recognition and reconstruction, and behaviour recognition from video. Much of the technology presented by Peter Tu is as sophisticated as it is ethically problematic. Much of this presentation was about surveillance technology, and this raises difficult ethical questions about privacy. For example, will hospitals accept an automated video surveillance system that would signal an alarm of some sort if a health provider failed to wash their hands between patients?

Accelerating discovery through multimedia and visualization

The perspective of an academic publisher was presented by Rafael Sidi, from Elsevier. He emphasized usability concerns; information visualization as an applied technology should be effective for researchers. Similarly to the CERN Documentation Server, Elsevier is creating separate tabs on the article display interface for various parts of the research article, allowing for independent views of the figures and captions and in some cases, such as the journal Cell (for example, see doi:10.1016/j.cell.2010.09.002 available through ScienceDirect), supplementary multimedia. It is also possible to limit search results to images in ScienceDirect; the results are presented as a grid of figures with contextual links to corresponding articles. Supplementary dataset visualization is also available in some cases. For an example of georeferenced data, see doi:10.1016/j.palaeo.2007.03.038. This supplementary data appearing in ScienceDirect is visualized using Google Maps, and actually resides at PANGEA (Publishing Network for Geoscientific & Environmental Data), an "Open Access library aimed at archiving, publishing and distributing georeferenced data from earth system research". The supplementary data in ScienceDirect includes many formats, such as MOL files for chemicals, and a growing number of supplementary data visualization tools such as Jmol, Protein Viewer (dynamically fetches data to visualize from Protein Data Bank based on identifiers in the article) and BrainNavigator, always within the context of a corresponding research article. From a recent article in D-Lib Magazine we learn that Elsevier has also established a dataset linking cooperation with Cambridge Crystallographic Data Center (CCDC).

Elsevier's Engineering Village uses facets on its search results interface, allowing searchers to navigate the result set by author and controlled vocabulary. Elsevier also sells specialized search products such as GeoFacets, a web based tool that combines georeferenced data and peer-reviewed content, and Illumin8, "Semantic search technology that "reads" content from a variety of sources, pulling relevant content and data, and highlighting relationships between topics, technologies, organizations and people."

Rafael Sidi also mentioned the work of information designer Moritz Stefaner. Elsevier collaborated with Moritz Stefaner, using data analysis based on SciVerse Scopus to produce a multi-touch installation that reveals how Max Planck Institutes collaborate.

Lorrie Apple Johnson, from the U.S. Department of Energy Office of Scientific and Technical Information, on the occasion of this workshop, officially launched ScienceCinema. SciencCinema uses MAVIS audio indexing technology from Microsoft Research to index the audio portion of videos, allowing the user to easily access the exact point in the video where specific words are spoken. Currently, the videos indexed include those by the Department of Energy as well as videos produced by CERN. A search for "solar energy" on ScienceCinema produces links to video results that display in the context of the surrounding words and allow the user to skip to the exact time when the keywords are uttered in each video.

Sebastian Stüker, from the Karlsruhe Institute of Technology, spoke about Quaero, a French research program for innovation on technologies for automatic analysis and classification of multimedia information such as spoken language, images, video and music. Sebastian Stüker emphasized the importance of research on automatic speech recognition, machine and speech-to-speech translation. The growth of research and academic content in video format on the Internet includes academic courses through sites such as: iTunes U Service, OpenCourseWare (OCW) by MIT, Carnegie Mellon Open Learning Initiative, as well as scientific dissemination and training through sites such as www.ted.com and www.videolectures.net. The video content is in many languages, so the international audience on the Web encounters a language barrier; the common assumption that English as a lingua franca solves this problem is incorrect. Sebastian Stüker cites the research by Lera Boroditsky at UCLA on how language shapes thought in making the point that the way our language is structured influences the way we think, and thus the preservation of language diversity is also the preservation of diversity in thinking, reasoning and knowledge. Efficient and effective translation technologies are necessary tools for this purpose. Quaero research projects, usually in collaboration with industry, include Web lecture translators, speech to text for the transcription of voice into text from audio or video sources (see the demo), speech-to-speech (between English and Spanish) translation on the iPhone that does not require a network connection, and more.

Combining and refining data to reach new discovery

Peter Fox, from the Tetherless World Constellation at Rensselaer Polytechnic Institute (RPI), presented a comprehensive overview of the challenges in scientific data visualization. He referred to an article that he recently published with James Hendler in Science magazine, "Changing the Equation on Scientific Data Visualization", arguing for the need to address what he identifies as a visualization bottleneck in the research life cycle. The bottleneck is the result of the fact that the costs of visualization technologies are not decreasing as quickly as the costs of generating data. Fox and Hendler argue that since the scientific process is becoming increasingly data-intensive, the use of visualization should be a routine method of exploration and analysis, universally accessible, as opposed to an expensive end product. The challenge stems from the fact that data is created in a form that facilitates generation, not use; data is often missing metadata and can be inconsistent, incomplete, evolving, and obtained by diverse instruments and protocols. The generated data is increasingly large-scale, complex and semantically heterogeneous. Fox argues that in order to make progress towards interoperability in e-Science, the publisher and provider of scientific data will need to put more work into describing its structure and content as well as providing and supporting Web services for the data. Fox and Hendler point out in their article that new data approaches such as NoSQL, "big data", and scalable linked data that are currently used by web companies such as Google and Facebook should be adopted more in scientific efforts.

Peter Fox also showed the 2010 Semantic Web Challenge 2nd prize winning TWC LOGD Portal for publishing Linked Data versions of open government data and the sharing of tools, services and expertise. The open data is in RDF triples, allowing for sophisticated querying, but the actual graphical rendering and visualization remains a challenge. Peter Fox described a new collaborative effort, enlisting the help of digital artists at the Experimental Media and Performing Arts Center at RPI with this step. He ends the presentation with speculation as to the role of visualization in new information systems that are particularly well suited for abductive reasoning and the exploration of 'hunches'.

Jennifer Lin presented sample use of PivotViewer. One of the samples showed visualized genetic data from ENSEMBL BioMart. The other sample visualized hundreds of images of wildlife captured by automated cameras with the purpose of learning about the effects of climate change. The latter sample shows the strengths of PivotViewer; the ability to interact with massive amounts of data on the web through an interface that is optimized for the web. The use of Deep Zoom (DZI/DZC) format for the visual data ensures short load times on the web and natural transitions through animations, while the use of CXML for metadata provides for many possibilities of interaction with the data. Interacting with the PivotViewer samples requires that the free Silverlight plug-in be installed.

The final presentation of the day, "Visualization and Indexing of Ecological & Biological Data", was by Jeff Falgout from the U.S. Geological Survey Center for Biological Informatics. He presented an overview of the National Biological Information Infrastructure (NBII), a program that provided increased access to data and information on American biological resources. In June 2011, four months after this ICSTI workshop, an announcement of termination was posted on the various NBII web services and databases: "In the 2012 President's Budget Request, the National Biological Information Infrastructure (NBII) is terminated. As a result, all resources, databases, tools, and applications within this web site will be removed on January 15, 2012."

Conclusion

The ICSTI Workshop on Multimedia and Visualization Innovations for Science showcased a variety of innovative visualization projects in scientific and technical information. The importance of visualization and multimedia search, indexing and presentation technologies will continue to grow as more and more scientific data and non-textual material is published on the Web. One design principle that was echoed in a number of the presentations at the workshop was the need for visualization tools that are truly useful to the variety of audiences that they are created for. Another discussion theme that surfaced from the workshop is the multidisciplinary nature of research in this area.

During the discussion, Robert M. Hanson recommended the Gordon Conference on Visualization in Science and Education which has helped support multidisciplinary research of effective visualization since 1994 through eight conferences, workshop tutorials and peer-reviewed mini-grants.

Looking ahead, ICSTI's 2012 Annual Members' Meeting will be followed by a One-Day Workshop titled "Delivering Data in Science". 'Delivering' scientific data implies presentation to end users, so visualization is likely to return as an important theme during this meeting, to be held on March 5, 2012. The equally relevant and increasingly important theme of 'multimedia' will be discussed further at the Workshop on "Non-Textual Information" following the ICSTI's 2013 Annual Members' Meeting in March 2013.

About the Author

Tomasz Neugebauer is the Digital Projects & Systems Development Librarian at Concordia University Libraries and editor of PhotographyMedia.com. He holds a Bachelor's degree in Philosophy and Computer Science and a Masters in Library and Information Studies from McGill University.