Audio Aura

The work "Audio Aura" is an ambient sonfication installation inside an office space. It is designed "to provide serendipitous information via background auditory cues, that is tied to people’s physical actions in the workplace"[1]. Metrics gathered through various sensors (e.g. infrared tracking of people inside the office) are being turned into an ambient soundscape to give workers inside that space a constant overview of what is going on inside the office space and the business.

Though "Audio Aura" focuses more on the actual physical office space, rather than internal business and process metrics, it still very closely related to the research and work for the "Listening to the Heart of Business" master project. A lot can be learned from the conducted research of the "Audio Aura" project and their results and take aways should be highly considered during the creation of prototypes.

The following three quotes have been particularly seen as important. Those learnings about sound design for such a particular system must not be ignored and have to be kept in mind while creating the intended soundscape that will be implemented inside the DataShaka office space:

"Because we intend this system for background interaction, the design of the auditory cues must avoid the “alarm” paradigm so frequently found in computational environments." [1]

"Another idea (...) is imbedding cues into a running, low-level soundtrack, so that the user is not startled by the sudden impingement of a sound. The running track  itself carries information about global levels of activity (...). This “group pulse” sound forms a bed within which other auditory information can  lie." [1]

"One useful aspect of the ecological approach to sound design is considering frequency bandwidth and human perception as limited resources." [1]

Furthermore, two interesting sources are mentioned that will be further investigated:

• I&ii, H. and Ullmer, B., (1997) “Tangible Bits: Towards Seamless Interfaces between People, Bits, and Atoms,” in Proceedings of CHl’97, ACM, March 1997.
• Weiser, M. (1991) The Computer of the 21st Century. Scientific American 265(3):94-104.

Sources:

D. Mynutt, E., Back, M.,, Want, R., Palo, X., Baer, M., B. Elli, J., "Designing Audio Aura", Alto Research Centre, University of Stanford, Gorgia Institute of Technology, 1998 

Gantt Chart Update

While catching up with the initial Gantt Chart, it turns out that some changes are necessary to respond to the current status of the project. As suggested by the project's supervisors, more time has been reserved for initial research and theoretical work. Therefore, the "Structuring Research & Hypothesis" and "Writing" activity have been moved forward by two weeks. Additionally, the monthly goals had to be restructured. "Live Data & App hosted online" has been moved to December and "Prototype II & DISQ implemented" has been canned.

The main focus still lies on having a good user studies session in November, as the outcome will be essential for the written thesis. While evaluating the results from those user studies and writing the thesis, the goal for December is now to implement the evaluated prototype inside the office and observe how employees react to it.

Below is the updated Gantt Chart for the project.

What do I want to know?

Before starting the user studies and going into deeper organisation and prototyping, it is the most important thing to make clear what one wants to know and what the desired knowledge is, that is supposed to alight from these studies. During a quick brainstorm, these were the initial questions and metrics of interest, that could be monitored and evaluated during those studies:

The User's

• performance 
• action
• reaction 
• monitoring bandwidth 

_______________________________________________

• while executing an unrelated task
• for different use cases/"data-stories"
• for different sound design approaches

_______________________________________________

• get user experience feedback on 
• fatigue
• transition
• notification level
• disturbance
• annoyance

The Sonification Handbook mentions a lot of areas that still lack documented researching. The following paragraphs are the ones that seemed mostly related to the thesis:

Chapter 2
Theory of Sonification
Bruce N. Walker and Michael A. Nees

"Surprisingly few studies to date have considered how the addition of a secondary task affects performance with sonifications." (Page 21)

"The successful deployment of sonifications in real-world settings will require a more solid base of knowledge regarding these issues." (Page 22)

"Researchers have only scratched the surface of possible context cues and their configurations, and we need to implement and validate other, perhaps more effective, methods (see, e.g., Nees & Walker, 2006)." (Page 26)

Chapter 14
Earcons
David McGookin and Stephen Brewster

"Similar studies [20] have been carried out on progress bars (...), where the user may wish to remain aware of an ongoing operation but not constantly visually monitor it. Beaudouin-Lafon and Conversey [4] proposed the use of Sheppard-Risset tones for auditory progress bars. (...) Unfortunately, Beaudouin-Lafon and Conversey did not evaluate their technique." (Page 353)

Chapter 18
Sonification for Process Monitoring
Paul Vickers

In all, four ecologies were constructed:

1. Voice world— vocal speech labels;

1. Sound effects world— beach noises: an auditory icon/soundscape set;

1. Music world— tonal musical motifs: a structured earcon set;

1. Rich world— a composite set of musical motifs, sound effects, and vocal messages.

Unfortunately, no formal studies have been published to discover how well the ecologies worked and which of the four was better received by users. (Page 479)

Obviously, not all research suggestions can be executed during the user studies. The major focus will lie on measuring the performance of sonification during a secondary task, also in comparison or addition the visual input. How context cues can be transported and comparing different ways through various sound design approaches will also be investigated. In that context, looking at the use of Sheppard-Risset tones to communicate statuses of processes will be evaluated. The observation of a sonification tool in a live working environment will not be part of the initial user studies. It is however still planned during later studies as part of this thesis. Investigating different ecologies as suggested in the "sonification for Process Monitoring" chapter will not be part of the user studies.

In conclusion, below are the metrics and topics that will be investigated during the user studies:

• comparing performance, action, reaction and monitoring bandwidth with a sonification prototype for process monitoring while executing a second task (for different "data scenarios") in comparison to visual only approaches
• investigating the use of different possibilities to transport context through sound (Sheppard-Risset tones a.o.)
• Interview after user study to get detailed user feedback (fatigue, transition, notification level, disturbance, annoyance)

Calm Technology

"We have struggled for some time to understand the design if calm technology, and our thoughts are still incomplete and perhaps even a bit confused. Nonetheless, we believe that calm technology may be the most important design problem of the twenty-first century, and it is time to begin the dialogue."

As mentioned in a previous blog post, the article "Calm Technology" by Mark Weiser and John Seely Brown has been further investigated.

The paper describes the problems of today's information overload and how calm technologies could solve these problem, introducing a new way of presenting digital information to people. It is distinguished between a person's center of attention and periphery of attention, meaning "what we are attuned to without attending to explicitly." As an example, a car driver is described, and how he might not pay his entire attention on the car engine's noise constantly, but still be able to quickly recognize and maybe take immediate action once the sound changes. The ability to "move easily from the periphery of our attention, to the center, and back" is pointed out as a major attribute and advantage of calm technologies.

The attributes and abilities described to calm technologies in this paper are in line with the design vision for the "Listening to the Heart of Business" project. The information presentation is supposed to stay ambient and perceived only in a subtle way by the "periphery of our attention", but still be able to quickly catch our full attention when necessary. A related metric to this could be the so called notification level, often being described as a typical dimension of an ambient display taxonomy.

"It seems contra-dictionary to say, in the face of frequent complaints about information overload, that more information could be encalming. It seems almost nonsensical to say that the way to become attuned to more information is to attend to it less."

(...)

"As we learn to design calm technology, we will enrich not only our space of artifacts  but also our opportunities for being with other people. Thus may design of calm technology come to play a central role in a more humanly empowered twenty-first century."

Skype meeting with Supervisors

A second meeting took place with the two supervisors of this master project happening at the Hochschule Darmstadt. This time, the meeting took place via Skype from the Media Campus in Darmstadt/Dieburg and the DataShaka Office in London. Topics have been the classification of  metrics, taxonomies for ambient displays and sonification, user studies and kicking off of writing the actual thesis.

Following points are the major take aways and action:

• identify and define possible intersections and transformations between the three metric classes
• find or create own taxonomy for
• Ambient Display
• Sonification
• => Ambient Auditory display
• "What do I want to get out from my user studies?"
• Start using Citavi
• Create preliminary Table of content
• add early content
• Create a bibliography

Starting to write the actual thesis by beginning with a table of contents and filling in early content has the highest priority, directly followed by the organization of the user studies, as the results of these studies will feed largely into the thesis.

Thanks again two my supervisors for the great support of the project over such a long distance!

The Handbook of Sonification summary

The "Handbook of Sonification" has been read completely during the first research period of the master project. In the following days, all  notes and take aways from this extraordinary work about data sonification will be gathered and evaluated.

The book is structured into four main sections:

1. Fundamentals of Sonification, Sound and Perception
2. Sonification Technology
3. Sonification Techniques
4. Applications

Each section contains various chapters. Before the first section is an introduction, pointing out the potential of Sonification, the major vision of the field and gives suggestions how to read the book. Each section and their chapters will be evaluated in future blog posts.

The book provides a tremendous amount of information about data, sonification and psycho acoustics from a theoretical and practical perspective as well as practical evaluations of auditory displays. Moreover, very valuable information is given on the gaps that have not been covered by scientific researches yet.

Below is the full table of contents of the book, including the authors of each chapter:



1 Introduction
Thomas Hermann, Andy Hunt, John G. Neuhoff


I Fundamentals of Sonification, Sound and Perception


2 Theory of Sonification
Bruce N. Walker and Michael A. Nees

3 Psychoacoustics
Simon Carlile

4 Perception, Cognition and Action in Auditory Displays
John G. Neuhoff

5 Sonic Interaction Design
Stefania Serafin, Karmen Franinovi´c, Thomas Hermann,
Guillaume Lemaitre, Michal Rinott, Davide Rocchesso

6 Evaluation of Auditory Display
Terri L. Bonebright and John H. Flowers

7 Sonification Design and Aesthetics
Stephen Barrass and Paul Vickers


II Sonification Technology


8 Statistical Sonification for Exploratory Data Analysis
Sam Ferguson, William Martens and Densil Cabrera

9 Sound Synthesis for Auditory Display
Perry R. Cook

10 Laboratory Methods for Experimental Sonification
Till Bovermann, Julian Rohrhuber and Alberto de Campo

11 Interactive Sonification
Andy Hunt and Thomas Hermann


III Sonification Techniques


12 Audification
Florian Dombois and Gerhard Eckel

13 Auditory Icons
Eoin Brazil and Mikael Fernström

14 Earcons
David McGookin and Stephen Brewster

15 Parameter Mapping Sonification
Florian Grond, Jonathan Berger

16 Model-Based Sonification
Thomas Hermann


IV Applications


17 Auditory Display in Assistive Technology
Alistair D. N. Edwards

18 Sonification for Process Monitoring
Paul Vickers

19 Intelligent auditory alarms
Anne Guillaume

20 Navigation of Data
Eoin Brazil and Mikael Fernström

21 Aiding Movement with Sonification in “Exercise, Play and Sport”
Edited by Oliver Höner

Reading the "Handbook of Sonification" while commuting to work

Source Control for first Prototype

Screenshot of the SourceTree project

To work more effective and secure on the prototype for the user studies in November, the code project of this prototype is now in source control. The software used is GIT and SourceTree. Every change to the prototype is now tracked and can be rolled back. A visualization in SourceTree shows all added features and every single commit to the code project. Also, it is now possible to amend to the same code from different machines and keep all devices up to date from the same source.

User Studies Planning Kick Off

To kick off the organisation of the user studies, a list of required items has been created and missing items have been identified.

Required days for the user studies are about 4-5 as the booking meetings with all prefered test persons will probably not be all possible on the same day.
For the interaction with the prototype, a laptop is required. To display graphs and visual feedback when required a second screen and a second laptop would be useful.
To play back sound, speakers are needed. Recording the studies visually, a camera needs to be installed and ways to record audio properly will need to be in place. Obviously, a room to hold those studies is required as well.

Meeting rooms are available for booking inside the office space. The bookings still have to be done. A personal laptop can be used for the user studies, a second one however will have to be organized. A large screen is available in every meeting room. For audio play back, speaker should be acquired and headphones should not be used, to later be able to easily evaluate the audio and video material. An audio recording device is available (Zoom H4), a video camera still has to be borrowed.

Here is the resulting list. Red marked items are not available yet.

• 4-5 days time
• 1st Laptop for interaction
• 2nd Laptop and Monitor for graphs
• Audio Speakers
• Video Camera
• Audio Recorder
• Meeting Room

Ambient Technology Research

Further research has been conducted in the area of ambient information system. The major insights for this project are presented below, referring to the specific papers that have been examined closer so far.

Most works define Ambient Displays by referring to already existing work in this area. They mostly stress the aspect of visual ascetics and minimal data representations. It is also often mentioned, that the information of an ambient display should be non-critical, as for this type of data should be represented in a more detailed manner. This statement is standing a bit in contrast with the intended work of this project. Hence, it will be discussed and challenged.

The paper "Towards a Taxonomy for Ambient Information System" by Martin Tomitsch, Karin Kappel, Andreas Lehner and Thomas Grechenig from the Vienna University of Technology introduces the term "Ambient Information System", intending to broaden the horizon of ambient displays, as there are not only visual approaches to ambient technology and ambient information. The term "Display" however does not necessarily describe a visual display, but could also refer to others means of information representation, such as audible feedback for information as in an "Auditory Display". Still, broadening the area of Auditory Display from visually focused projects is very important and eye-opening,
Furthermore, the paper compares different taxonomies for the term "Auditory Displays" and concludes with their own taxonomy for auditory information systems:

"abstraction level, transition, notification level, temporal gradient, representation, modality, source, privacy and dynamic of input"

Additionally, various dimensions are identified, such as "Abstraction" or "Notification Level" (among others). The amount of abstraction mainly results how much the represented data has been aggregated and minimized. The notification level is the amount of times "a system alerts the user or even forces him to interrupt his primary task". (Adjusting and using this notification level in a smart way can possibly challenge the non-critical data aspect of Ambient Devices).

Here's a list of all dimensions being identified by the authors:

• abstraction level
• transition
• notification level
• temporal
• gradient
• representation
• modality
•  source
• privacy
• dynamic of input

The paper "Intrusive and Non-intrusive Evaluation of Ambient Displays" by Xiaobin Shen, Peter Eades, Seokhee Hong and Andrew Vande Moere from the University of Sydney and the University of Melbourne focuses on evaluation techniques for Ambient Displays:

"Intrusive Evaluation — is where the user’s normal behavior is consciously disrupted by the evaluation experiment. This kind of evaluation often consists of usability tests in a laboratory environment for a short period. Most such experiments are conducted using well established evaluation techniques in information visualization (for example, questionnaires and interviews)."

"Non-Intrusive Evaluation — is where the user’s normal behavior is not consciously disrupted by the evaluation experiment. This often focuses on actual use in a general environment (in situ) over a long period. Currently, few existing evaluation techniques can be applied successfully in this manner."

Furthermore, a practical user study is presented in high detail, providing very useful information. Two major conclusions are being stated at the end of the paper:

• "Users need better support information from ambient displays"
• "Non-intrusive evaluation cannot be tested until the display integrates into the environment."

Some of the papers mention the work "Designing calm technology" by Mark Weiser and John Seely Brown from 1995. This paper appears to be an important body of work dealing with the idea of ambient technology and will be investigated during further research. This paper can be found here.

The paper "Ambient Display using Musical Effects" by Luke Barrington, Michael J. Lyons, Dominique Diegmann and Shinji Abe from the University of California and the ATR Intelligent Robotics and Communication Labs in Japan mentions a particular practical work named "Audio Aura" which shows similarities with the planned ambient auditory display for DataShaka. This work will be investigated further. Their research paper can be found here.

Both works will be further investigated.

References

Barrington, L., Lyons, M., Diegmann, D. and Abe, S. n.p. Ambient Display using Musical Effects. [e-book] Electrical & Computer Engineering, University of California, San Diego & ATR Intelligent Robotics and Communication Labs, Kyoto, Japan. [Accessed: 2 Oct 2013].

Shen, X., Eades, P., Hong, S. and Vande Moere, A. 2007. "Intrusive and Non-intrusive Evaluation of Ambient Displays", paper presented at Pervasive '07 Workshop, Toronto, Ontario, Canada, 13 May. Toronto, Ontario, Canada: Pervasive '07 Workshop: W9 - Ambient Information Systems.

Tomitsch,, M., Kappel,, K., Lehner, A. and Grechenig, T. 2007. "Towards a Taxonomy for Ambient Information Systems", paper presented at Pervasive '07 Workshop, Toronto, Ontario, Canada, 13 May. Canada: Pervasive '07 Workshop: W9 - Ambient Information Systems.

Priorities

The online article "Postmortem of a Venture-backed Startup -  Lessons Learned from the rise and fall of @Sonar" that I have read recently, contains a particular sentence that made me reflect on the current planning of my master thesis:

You do not have 20% time. Identify your top three priorities. Throw away numbers two and three.

Putting this strategy to work, an hour has been spent on identifying the major priorities of the project and which would get the most valuable outcome for a successful research thesis:

1. Holding a detailed user study with
• early stage prototype
• staged data
• different user / user-groups
• detailed recordings and evaluation
2. Having a fully working application to
• present
• deploy
3. Deploying an application inside a live business environment to
• observe worker using the piece
• evaluate the additional value to the business
• prove the concept

Afterwards, priorities two and three have been excluded.

As hard as this proactive decision is, it is a valuable thing to do and important to focus on the one and most important priority for the project.

The really interesting article can be found here.

Finally, here's another famous quote that can also be found in this article: 

“Focus is saying no to 1,000 good ideas.” — Steve Jobs