Final Project

I'm happy to post (very late) that this Master project had successfully completed in March 2014 and has been given a First Class Honours.

The abstract, table of contents and introductory chapter of the thesis is below. If you're interested in full thesis, please let me know and I will send it to you! Also very excited to hear any comments or feedback on the project!

LISTENING TO THE HEART OF BUSINESS

by Clemens Anzmann at the DataShaka Ltd. office  

1. Introduction

2. Ambient Auditory Displays

1. Definitions
1. Sonification
2. Ambient Display
2. Taxonomy and Dimensions
3. Related Projects

3. Concept

1. Metric Classification and Manipulation
2. Sound Design

4. Implementation and Evaluation

1. Technical Research
2. Company Internal Research
3. User Studies
1. Intrusive User Studies
2. Non-Intrusive User Studies
1. Observational Methods
2. Reports and Evaluation
4. Retrospective

5. Outlook and Conclusion

6. References

Abstract

Every business today generates data through its everyday processes. The look of this data might vary, depending on the nature of the company. However, they all equally produce data. This data can, if used correctly, provide valuable information and crucially support important decision making. In fact, no decision should be made and no action should be taken that is not data driven or justifiable and backed up with data. This is commonly achieved through the continuous tracking of all business metrics and their retrospective analysis. A different approach is to continuously observe these metrics in real time, to gain a constant awareness on their development and the ability to take quicker action when necessary. The main goal of the “Listening to the Heart of Business” project was to develop and evaluate techniques and applications that produce ambient data sonification to provide non-disruptive data monitoring of business relevant data inside a working environment. Office workers should only be distracted from their main task when critical events occur. The “Listening to the Heart of Business” project states that the use of ambient auditory displays will enable companies to subtly stay on top of their data as well as empower them to take immediate action a lot faster when necessary. The hypothesis, ambient sonification would not add value to the awareness of business relevant metrics has been challenged through research in the fields of ambient displays and auditory displays as well as several user studies and their evaluation, including the implementation and evaluation of an auditory display inside a working environment. All research has been conducted at the DataShaka Ltd. office in London.

1. Introduction

Today’s companies are facing classic big data problems when trying to observe and evaluate their internal data. With the ability to track almost everything that is happening inside and outside the company, people are lacking tools to observe all these relevant metrics for continuous awareness and faster data driven decision making. The three major challenges in big data (also commonly known as the three V’s) are volume, velocity and variety [25]. Regarding the growing amount and accelerating shifts in internal and external metrics, it is especially the problem of volume and velocity that businesses have to cope with. With the overwhelming amount of data that can have a potential impact on the business, the desire to act proactively is challenged as these large amounts of data can only be consumed retrospectively.  
Particular data representation devices that addresses the problematic of information overload are so called ambient displays. Ambient displays destruct data to its most basic information to transmit this information in ways that can be subtly perceived by its observer. A common example for ambient displays are lights that are smoothly changing colour or intensity in relation to the data stream they represent. The major goal of ambient displays is to minimize the mental effort of the observer to monitor the data stream and present data in a non-disruptive way. 
For situations where it is not convenient for the observer to be continuously distracted by a visual display, auditory displays are a suitable solution to provide information without the necessity of visually focusing on a particular artefact, as “unlike visual perception, perception of sound does not require the listener to be oriented in a particular direction.” [22] Most current practical applications of auditory displays for these situations use a classic alarm-focussed approach and often trigger highly disruptive sounds.

A classic example for these types of auditory displays deployed for such use cases are alarms in stores triggered by a sensor that identifies stolen items. Disruptive auditory alarms can also be found in pilot cockpits, signalling critical values of several instruments. Furthermore, a common deployment of auditory displays is found in hospitals for the continuous observation of vital signs of numerous patients. Usually, the loudspeakers and sounds are specifically designed to make a strong noise audible on very sensitive frequencies. 
When looking at current problems and needs companies are facing due to their high data volume and velocity and comparing it to the use cases ambient and auditory displays are trying to solve, it appears that ambient displays and auditory displays both address and potentially satisfy different aspects of these needs. To elaborate and present their potential for this use case, the following research project investigates ambient and auditory displays and their effective use in working environments to enable company personnel to have a permanent awareness over business developments without being distracted from their everyday work. It furthermore investigates the term “Ambient Auditory Display” and proposes a taxonomy. User studies and a live deployment of a prototype in a working environment have been conducted. The prototype has been evaluated in regards to the proposed taxonomy. Evaluations and conclusions have been conducted after the successful completion of all user studies. 
To structure the limited research time and define goals and deadlines for the research period (September 2013 until February 2014), a detailed Gantt chart had previously been created. The outlined strategy on the chart had been evaluated and restructured continuously and went through different iterations during the research period. The following paragraphs presents the final version of the Gantt chart that can be found in appendix A.

During the first month, completing the technical research to produce potential sonification prototypes and create initial sketches and tests have been the main focus. At the end of the second month, a preliminary sonification prototype created from these sketches was planned to be ready and able to process and present a data stream. Simultaneously, it was planned to conduct all necessary theoretical research within the first two months. During the third month, user studies had been scheduled. Evaluating and structuring all results was timed to be completed by the end of the month, so the development of a second prototype respecting all results from the previous user studies could be started. During December, this new prototype was scheduled to be deployed in a live working environment, to be evaluated at the end of the month. The completion of the presentation and writings of all the noted results from both user studies in the written documentation was set for the end of January. Additional stretch goals had been defined for each month. 
All research has been conducted at the DataShaka office in London. DataShaka Ltd. is a data unification platform specialised to solve the variety problem in big data. As the volume problem has become manageable with decreasing prices for digital storages, variety is now a major problem in big data and companies have strong desires to be able to have all data in one place. All internal and external business relevant data comes in various different formats and all datasets are structured differently, depending on their content. DataShaka is solving this problem for their clients by harvesting and unifying all of their client’s data in an agile data format. Many internal metrics exist, that require observation and attention by employees of DataShaka. Ambient auditory display prototypes have been evaluated, by testing and deploying them inside the DataShaka office. Data from occurring events, feedback as well as scheduled interviews with DataShaka personnel has been gathered for additional evaluation.

The following body of work presents the research project “Listening to the heart of business” and its results. In chapter 2, definitions for the terms ambient display, auditory display and data sonification are derived and a taxonomy is proposed for the term “Ambient Auditory Display”, combining the attributes of these data representation approaches. Related projects are investigated and compared. Chapter 3 describes ways to simplify and extract valuable information from raw company metrics. Furthermore, sound design approaches for data sonification are discussed. Further on, chapter 4 presents all user studies that have taken place during the research period. Technical research in the area of data sonification achieved through online applications using JavaScript libraries is propounded. A detailed description, report and evaluation of all user studies is further presented and future research paths are being discussed. 

A research paper by NASA about Data Sonification

A research paper by NASA about data sonification and its use in the practical field has been studied for further insight into sonification applications in different research areas.
It describes sonification as a tool that "can improve and increase the bandwidth of the interface 'human-computer' and can find a lot of applications in the wide range of information technology."

Furthermore, the strengths and possible features of sonifcation are being pointed out:

• "uncovering patterns masked in visual displays
• identify new phenomena current display techniques miss
• improving data exploration of large multi-dimensianal and multi-dataset
• exploring in frequency rather than spatial dimension
• analyzing complex, rapidly, or temporally changing data
• complementing existing visual displays
• monitoring data while looking at something else (background event finding)
• improving visual perception when accompanied by audio cues"

Background event finding would be the most relevant use case for the Listening to the Heart of Business research project.

In the introductory chapter, the papers provides a very detailed research history of the field of Sonification and points out very interesting early works in this field from the following authors:

• Pollack, I., and L. Ficks. “Information of Elementary MultidimensionalAuditory Display”. J.Acoustic SocietyAmerica (1954)
• Speeth, S. D. “Seismometer Sounds”. J.Acoustic SocietyAmerica (1961)
• Chambers, J. M., M. V. Mathews, and F. R. Moore. “Auditory Data Inspection”. Technical Momorandum no. 74-1214-20,AT&T Bell Laboratories (1974) 
• EdwardYeung, “Auditory Display”. Santa Fe Institute (1994)

Further in the paper, the application xSonify is introduced and explained in much detail and how it can be used for successful data sonfications. The software provides different sonification modi and instruments, also including speech support for blind people. Different data sets can be imported into the application and mapped to different sonification techniques and instruments. After setting everything up, the user hits the play button and can listen to the data. xSonify is one of the most famous sonification applications.

Towards the end, a very interesting piece of work by Sylvain Daude and Laurence Nigay is presented, introducing a sonification process and its data transformation within (from Sylvain Daude, Laurence Nigay, “Design Process For Auditory Interfaces”, Proceedings ICAD 2003).

From Data To Data View: Data Transformation 

(...) the raw data will be mapped value by value into a range of values between0 < x < 1. This process can be considered as a kind of standardization of values whichis necessary to make the data available independent from their unit and scale.

 From Data To Abstract Sound Space: Sonification Transformation

(...) the data are prepared according to the chosen Sonification modus. Every value will be assigned to a certain position at a “time line”. This “time line” isrepresents the time line of the Sonification sequence which will be played (...).

FromAbstract Sound Space To Sonic: Auditory Display Transformation 

(...) where the signal is finally displayed on a physical device.

The picture below visualizes the transformation of the data's journey from raw data to sound.

Sonification Process from the NASA Sonification Paper

At the end of the paper, difficulties and problems regarding sonification are being pointed out:

• low resolution of some auditory variables

• limited spatial precision 

• lack of absolute values 

• absence of persistence 

• no printout

More about NASA's research on sonification can be found here. 

Choir.io

"Listing to the Heart of Business" tries to find ways to give businesses a peripheral awareness over their data. Chior.io is a project that takes a very similar approach of presenting continuous data flows:

Chior.io is a web based service to allow to monitor processes acoustically, to provide "ambient awareness  without effort" [1]. Through an API, events can be streamed to the service which turns those events into sound. Defining those events happens by the user in the back end. Different languages are documented to send events to chior.io, such as Shell languages, Python, Ruby, Node.Js, PHP, C#, Java and Clojure. The resulting sonification is available online and therefore can be accessed from any machine. 

The sound design is given by chior.io and specifies three different event types: good, neutral and bad. Two different sound sets are available ("submarine" & "bloob"). 

Sounds available for the three emotions given by choir.io

It seems that individual sound design is not yet possible to implement. All auditory events appear to be static and sound icons don't seem to be able to change dynamically to an events' severeness. Hence, all events are technically treated as binary events. 

An impressive example and prove of concept is available on the project's website: A live sonification of GitHub events.

GitHub events sonification by choir.io

References

1. Choir.io. 2013. Choir.io. [online] Available at: http://choir.io/ [Accessed: 28 Nov 2013].

Interview with a DevOps

An interview with the person responsible for DevOps has taken place at the DataShaka office.

Main questions during the interview where what the term DevOps means, how it is related to the agile movement, how it is different from classic operations and what are the tools currently available.

One topic during the interview was the agile moving, where the term DevOps emerged from. One major drive in the agile movement was to bring developers closer to people, rather than having them isolated inside the company. Similarly, the DevOps movement intends to bring developers and operations closer, as both departments used to be quite separate. In larger companies, entire DevOps teams are now responsible to link development and operations together (also see previous blog post here).

Software being used by operations or DevOps have been mentioned:

• Etsy
• Stats D
• DataDog
• SNMP

A term mentioned very often during the interview was "pro-active". It is important for a person responsible for DevOps, to be pro-active rather than re-active. This means specifically, to be able to interfere before something breaks and to be able to act with hindsight. Practically (and already in acoustically) speaking, it means reacting early to strange noises rather than to the alarm noise, when it is already too late.

There seems to be a lot of interesting history about operations and it appears that this field of work is still facing similar problems as it used to have in the past. With the ability to monitor all the things simultaneously in real-time, operations is facing classic big data problems, such as volume and/or velocity. Additionally, operations as well as DevOps departments want to be able to act pro-actively, interfere before things breach and provide a non-disruptive service. With the rising amount of data, this desire is challenged by what is often referred to as the "bandwidth problem". Not resolving this problem, will ultimately this hinder scaling. And therefore, better tools will be needed for DevOps departments.

The content and information received during this interview has been of very high value for the research project as the challenges faced by the DevOps sector match entirely with what the "Listening to the Heart of Business" project is trying to solve: Being able to be on top of a large amount of metrics with minimal disturbance of your primary task.

---------------------------------------------------------------------------

Below can be find excerpts from the interview that have been found as particularly valuable:

"DevOps introduced a numbers of new tools, processes and operations bringing operations into the twenty-first century"

"[It's about] making sure things are pro-active, not re-active"
"monitor everything"

"People have an expectation now, that things will just work all the time. That means you need to be an top a lot more."

"You need to be able monitor a lot of things at one time with through a limited amount of real estate."

"We can see broad brushes, we can see if something broken or if something stopped, but actually getting this pro-activity of 'is it operating well, how is operating' is difficult, because the bandwidth is lower."

"You might have an ambient display for temperature alarm. (...) You wouldn't have an ambient display for a fire alarm. (...) A fire alarm is designed to disrupt you."

"People don't make enough use of non-disruptive monitoring. That's what pro-activity is about. (...) Pro-activity is the understanding and having the nuance to be able to dig into what's happening before it's a problem and before it's breaching."

"It's also a scaling problem. You suddenly got that need to monitor much more than you're able to perceive. You need to be on top of an awful lot of stuff. And that's when people start to look into different ways of data presentation rather than purely a number. What can we do to help increase that leverage? For large organisations, it's about being able to get that higher bandwidth.""The ingenuity of human brains and look at a problem and find a lateral solution is always going to be involved. What it is is about giving humans better tools."

User Studies Data Analysis

The data collected from the first user study has been harvested and analysed. First thing that comes to eye is, that there do not appear to be many clear differences in the data and it will be difficult to extract valuable information or conclusions, due to the fairly small sample size.

Initially, the performance over time while keeping track of moving metrics has been examined. When comparing the average performance of users working with the visual only approach to the sound approaches, no clear performance increase can be identified. There appear to be performance differences between users working with the "sound alarm only" and the "sound alarm and audification" approach. However, the small sample size does not allow to draw a solid conclusion from this.

In all cases, the performance decreased in a fairly similar amount when an event occurred. Though there where performance differences between each user, the average performance decrease while fixing events was fairly consistent. A notable issue in the data is however, that performances with the visual approach showed a  more consistent performance whereas the audio approaches showed more fluctuations.

A non-intrusive evaluation with a similar prototype installed in a live office environment is planned. This way, the factor time and surprise would take a larger role, as users would not constantly expect event to occur and fully concentrate on their primary task.


In the context identifying tests of auditory alarms, artificial sound design approaches appeared to be clearly more identifiable then recordings of natural sounds. 16 different sound alarms where triggered, eight of them being natural and the other eight being artificially created sounds.

Amount of users guessing context correctly for each context type separated by natural and artificial sounds

Artificial sounds appeared to be more familiar to users whereas natural sounds often caused surprise, confusion or where sometimes completely ignored. 76% of the artificial sounds where referred correctly to what the sound designer intended the sound to represent, whereas only 24% of the naturally recorded sound's context was identified.

This outcome is very insightful and will be further investigated. The focus on further investigations will lie on artificial sounds and different techniques how to transport context.



The analysis of transcript of the interviews is still work in progress.

DevOps

"DevOps is a response to the growing awareness that there is a disconnect between what is traditionally considered development activity and what is traditionally considered operations activity. This disconnect often manifests itself as conflict and inefficiency."

-- Damon Edwards, "What is DevOps?" [1]

In agile business environments, DevOps teams are often responsible to ensure change and stability, and most importantly deal with the conflict that lies within those two areas. The term DevOps is put together from the two terms "Development" and "Operation", which are two classic departments in software companies. Development is responsible for all amendments, bug fixes and constant further development of the software. Operations is responsible for the software's stability. Both departments have interest that often conflict. The area around DevOps is supposed to solve this conflict and bring back agility to the static and sluggish current "Development - Operations" structure, which is continuously blocking itself.

Image from Dev2Ops.org

To ensure constant stability without constantly being distracted with monitoring and still being able to remain agile and open to change, new ways of metric measurements are necessary. 
Therefore, a major part of the DevOps movement is related to metric monitoring of so called KPIs (Key Metric Indicators) [3].
Ambient displays and Auditory Displays inherit potential to benefit for DevOps teams, as they provide a constant peripheral awareness of important metrics but are not necessarily a continuous distraction. 
DevOps is an area the research project could make a major contribution to and it proves to be a potential use case for the deployment of an ambient auditory display.

Image from Dev2Ops.org

Further research will be conducted through organizing an interview with the person responsible for Lead DevOps at the DataShaka office. After the evaluating the interview, installing an ambient auditory display triggering subtle sound events will be the next priority. Its impact on employees, operations and development as well as the office space in total will be evaluated.

References:

1. dev2ops. 2013. What is DevOps? - dev2ops. [online] Available at: http://dev2ops.org/2010/02/what-is-devops [Accessed: 17 Nov 2013].
2. Hüttermann, M. 2012. DevOps for developers. [New York]: Apress.
3. Reh, F. 2013. Key Performance Indicators or KPI. [online] Available at: http://management.about.com/cs/generalmanagement/a/keyperfindic.htm [Accessed: 17 Nov 2013].

First User Study Session

The first user study has been conducted at the DataShaka office. Six individuals have been tested using two different prototypes with different settings while executing a primary task. The user group consisted of five male and one female person between 27 and 49 years of age. All reported normal hearing abilities. All participants work with computers on a daily basis on a similar amount of time.

The studies took place in a lab with a size of 16 square meter, equipped with a large round table, several chairs, a large screen on the wall, one laptop mounted on a stand, speakers and an audio recording device.

The approach to the evaluation of the prototypes was an intrusive evaluation, "where the user’s normal behavior is consciously disrupted by the evaluation experiment", as described in the  paper "Intrusive and Non-intrusive Evaluation of Ambient Displays" by Xiaobin Shen, Peter Eades, Seokhee Hong and Andrew Vande Moere.

The primary task consisted of a simple mouse clicking game, where the user had to follow a box on the screen and click it. The amount of clicks every 10 seconds had been recorded to have a unit to measure the user's performance. While executing this task, users had to monitor four different metrics that where represented in different ways:

• visual representation only
• sound alarms only
• sound alarms and metric audification

Every time, one of the metrics exceeded a particular threshold, users had to interrupt their primary task and switch to the other screen to resolve the issue (by clicking a appearing button).

The visual only data representation of the four metrics was represented on the large screen mounted to one of the walls of the lab. The screen was clearly visible from the position. Once a metric exceeded a particular threshold, a large red bubble would appear on the screen. Additionally, all four metrics where visualized by a bar chart.

The "sound alarms only" approach would only trigger a alarming noise once a value exceeded a particular threshold.

The "sound alarms and metric audification" approach contained the alarming noise as well as four different wave and noise generator mapped to the four values. Through that, users could constantly hear the values and the value's changing.

Every user went through two of these three settings after a short warm up with the application. The sound approaches where always prior to the visual only approaches.

In the second setup, several different auditory icons and earcons where played. Users had to decide for each alarm sound what context it might inherit. There where four different solutions:

• Data In
• Data out
• Process Step Complete
• Process Step Failed

Different design approaches have been tested and evaluated through this method.

After interacting with the prototypes, users have been questioned during an interview about their experience.

References

Shen, X., Eades, P., Hong, S. and Vande Moere, A. 2007. "Intrusive and Non-intrusive Evaluation of Ambient Displays", paper presented at Pervasive '07 Workshop, Toronto, Ontario, Canada, 13 May. Toronto, Ontario, Canada: Pervasive '07 Workshop: W9 - Ambient Information Systems.