00

Project Objective

Notes on Perspective is an application that visually reconstructs the thoughts and feelings which occurred during a classical musical critique at the Mannes School of Music in New York City. It experiments with making visually tangible and beautiful the thoughts and feelings that occured during this musical performance among the ensemble members. It visualizes a color score by clustering the musical score being performed by musical notation and sequential progression. Notes on Perspective acknowledges the shortcomings of digital artifacts in replicating lived experience, attempting to playfully humanize temporal data, bringing information to life. This approach to embodied data visualizations for temporal data is referred to as designing in Kairos.

01

Data Architecture

Notes on Perspective is built as a single page application (SPA) with three states. Each state visually experiments with binding data to an abstraction of embodied experience: 1) situatedness, 2) activity and 3) perception.

02

Activity View

The activity view display borrows from popular social media conventions to offer easy access to the raw data behind the data visualization. As the audio of the performance plays, the timestamp is continually checked against the reaction data to see if there are any reactions that occur in that moment. If there is, a function to get the reactions is executed. Notes on Perspective follows Khrouf et.al’s perspective in developing the application EventMedia, which mines events’ connections on the social web. The visualization of “distributed data fragments provides a key advantage not only to deliver enriched views, but also to gain insight into interesting sociological aspects. [The] goal is to build a web-based environment that allows users to discover meaningful, surprising or entertaining connections between events, media and people (Khrouf, Milicic, and Troncy 2014, 3-10).” To achieve this in context of the musical performance under visualization, Notes on Perspective pops up reaction data as comments in time to the performance, allowing interactors to focus on the connection between musical moments and particular thoughts and feelings experienced by the professional musicians in the study.

03

Situated (Theater) View

In the situated view, participants are invited to meet the Ensemble class to personalize the humans driving the activity that are abstracted and visualized in all other aspects of the data visualization. To invoke a sense of personal identification and understanding of ensemble members as personalities, tooltips are offered to provide a view into personal details regarding their familiarity with the piece, the instruments they play, as well as random thoughts that refresh upon each new “encounter” (hover) over that ensemble member.

04

Perception View

Developing a design theory for building data visualizations sensitive to the ways that humans experience events in time and embodying that digitally is the original underpinning of this project. One of the fundamental properties of the visual world is that “it is filled with things that have meaning” and “we perceive a world whose fundamental variables are spatial and temporal - a world which extends and which endures” (Gibson 1974). Designing data in Kairos, then, seeks to create an environment that offers humans the affordance to perceive, subsequently interpreting, a dataset’s story and meaning by visually travelling through it.

05

Color Score

A color version of the score was generated using a kmeans machine learning model that grouped each beat of the music by similarity in musical notation and sequential progression. Each theme is color coded and the impressions' background color represents which visual score theme we are in.

06

Enter the Experience