Drawing with Sounds, an Interactive Musical Drawing App Created with Processing

This is my first creation using the Drawing with Sounds application. This cause and effect application was created with Processing 1.5.1 and produces musical sounds and patterns, along with random abstract shapes that are triggered through movements of a mouse. The application has been adapted for switch users, making it accessible to those who have motor impairments.

Processing is an open-source programming and integrated development environment that was build on the Java programming that has been easy to learn by people who have limited technical backgrounds. It is used by people from from a range of fields, such as art, music, journalism, and data visualization.

The fact that Processing is easy to use makes it great for people with just a little bit of coding knowledge to adopt a variety of open-source applications and tweak a few of the variables to according to the need.   Using the Drawing with Sounds app as an example, I wanted to change the size of the interactive canvas.  By typing in a few keystrokes,  I changed the original size of the screen from 800 x 800 to something larger, as shown below:

I then typed in another variable for the background color, and was pleased with the results:

The application was developed from Andrew R. Brown's tutorial on "SoundCipher", based on the SoundCipher library for Processing.  According to the description of the application, "The sketch is intended for use....as a sensory stimulus using sound, shape and color to create engagement". 

Below is a videoclip created by Keith Manville, demonstrating what happens when you interact with Drawing with Sounds: 

I look forward to sharing more on this topic in future blog posts.  I'll be sure to include basic "how-to" information for my "low-tech" readers and colleagues who desire to learn a bit more about using and creating basic interactive applications that appeal to young people with special needs.  

RELATED

Note:  SEN is the UK acronym for "Special Educational Needs"

Drawing with Sounds Processing sketch

James Winchester, SEN Classroom Blog, 10/31/13

Drawing with Sound

Keith Manville, opensen blog, 10/27/12

"Open-source, Sensory and Interactive Technology in SEN"

ProcessingSEN Wiki

ProcessingSEN Drawing with Sounds Download Page

(An accessible switch-enabled version of Drawing with Sounds can be downloaded from the above link.)

Free sensory applications built in Processing 1.5.1"Download links for sensory applications built in Processing 1.5.1, designed to assist SEN students in engaging in learning through the the use of ICT, interactive “Smart” boards, touchscreen or tablet technologies.  Our experience is that using applications on these devices or with natural user interfaces such as the Microsoft Kinect can increase the opportunities for engagement and social communication with many students."

I will be posting more information about emerging interactive technologies for special needs in the future. In the meantime, take a look at the following resources:

The SHAPE Project, technologies to enhance learning for young people on the Autism Spectrum

Keith Manville, opensen blog, 11/10/12SEN Classroom "Ideas and Tech used in a SEN class"

iPadSEN

GestureSEN Wiki

LeapSEN

KinectSEN Wiki
"Using Kinect in Special Schools for Pupils with Severe Learning Difficulties"
iGazeSEN
OPENSEN Videos

Link to "Who Works with Creative Coders", by Tim Stutts

Who Works With "Creative Coders"?
Tim Stutts, Interaction Designer/Programmer + Founder at PushPopDesign


RELATED
Tim Stutts' DataGlam Flickr Set; "An on-going series of data-inspired graphic and video works programmed in OpenFrameworks, Processing and Three.js"

Creative Applications.Net
"CreativeApplications.Net reports innovation and catalogues projects, tools and platforms at the intersection of art, media and technology"

Christian Bannister's Interactive Multi-touch and Gesture-based Subcycle Project. Use your hands to shape sound, create, and manipulate music. Wow!!

I came across a link to Christian Bannister's Subcycle Labs website when I was taking a look at the Creative Applications Network website.


If you have an interest in music technology and innovative gesture/multitouch applications, you'll appreciate the details that are shared on the Subcycle website.  In the meantime, take the time to watch a few of Christian's videos.  Enjoy!


Blip Shaper Walkthrough

Blip Shaper Walkthrough from christian bannister on Vimeo.
"a) creating percussive patterns with monome b) shaping the individual sounds that make up the patterns with multitouch gestures c) recording touchscreen gestures as automation d) storing, duplicating and navigation patterns e) recording the resulting audio to a dynamic buffer f) manipulating the buffer with a multitouch cut-up approach g) visualizing everything with dual screens"
Subcycle Walkthrough

Subcycle Blip Shaper from christian bannister on Vimeo.

"a) creating percussive patterns with monome b) shaping the individual sounds that make up the patterns with multitouch gestures c) recording touchscreen gestures as automation d) storing, duplicating and navigation patterns e) recording the resulting audio to a dynamic buffer f) manipulating the buffer with a multitouch cut-up approach g) visualizing everything with dual screens"


The following information describing the Blip Shaper is from Christian's Subcycle website/blog:
"For the drum sounds I have Drumaxx running for synthesized sounds and Battery running for sampled sounds.  These are running in parallel so for each voice there is a separate patch running in each VST.  The Parameters are modified with the touchscreen independently but in all cases a single touch gesture on the X-Axis will cross fade between the sampled version of the sound and the synthesized version of the sound.  I love this because I have never seen this before and I can never decide which technique I like better.  The synthesized drums are more malleable and have more interesting parameters to play with but the sampled sounds seem more substantial.  I will post a detailed list of parameters and gestures in the future."   


Christian currently uses Max for Live (ableton), and codes with Processing/Java, using Eclipse for organization.

Autopilot - Subcycle
autopilot - subcycle labs from christian bannister on Vimeo.
"sound visualization, multi-touch interface, break beat performance engine, autopilot, spacialized sound, dsp, max/MSP, fm synthesis, sonic navigation, sound storm visualization, time machine, granular, interactive sound sculpture, joystick array, more at subcycle.org"

SOMEWHAT RELATED 
Community Core Vision
NUI Group

BTW,I'm wondering if Christian or other NUI group members would like to do something with some of my ideas that incorporate interactive music. To learn more, see my post, "It must be  Voronoi: Looking for ideas for my music+art+dance+tech dream...."
...


....

Interactive Information Visualization for the Kinect? Something like Jer Thop's "Just Landed-36 Hours" might work nicely if revamped!

I follow the O'Reilly Radar blogs and came across a recent post about an information visualization created by blprnt two years ago using Processing. I think it would have great potential if it was re-purposed for use on the Kinect! In the article, Edd Dumbill discusses the advantages of using Processing to create data and information visualizations.  


One example of the power of Processing is an information visualization, "Just Landed -36 Hours, created by Jer Thorp.  Jer gathered tweets from Twitter that included the statement, "just landed", along with location information for each tweet, within a 36-hour period, to create the visualization.


36 Hours- Just Landed is a great 3D visualization of air travel on our planet.  I especially lik the different views that the application provides. As soon as I watched the Just Landed video, I thought it would be great if it could be revamped for use on the Kinect!   (Leave a comment if you know of anyone working on a project in this area.)


Just Landed - 36 Hours from blprnt on Vimeo.


Information about the video from blprnt's Vimeo site:


"I was discussing H1N1 with a bioinformatics friend of mine last weekend, and we ended up talking about ways that epidemiologists model transmission of disease. I wondered how some of the information that is shared voluntarily on social networks might be used to build useful models of various kinds...I'm also interested in visualizing information that isn't implicitly shared - but instead is inferred or suggested...This piece looks for tweets containing the phrases 'just landed in...' or 'just arrived in...'. Locations from these tweets are located using MetaCarta's Location Finder API. The home location for the traveling users are scraped from their Twitter pages. The system then plots these voyages over time...I'm not entirely sure where this will end up going, but I am reasonably happy with the results so far.   Built with Processing (processing.org) You can read more about this project on my blog - blog.blprnt.com"


RELATED
Strata Gems:  Write your own visualizations:  The Processing language is an easy way to get started with graphics
Edd Dumbill, O'Reilly Radar, 12/3/10

Impress: A cool flexible interface project by Silke Hilsing

Impress, a cool flexible interface project by  Silke Hilsing:


impress - flexible display from Silke Hilsing on Vimeo.

From Sike Hilsing's website:

"Impress is the deliverance of the touch screen from its technical stiffness, coldness and rigidity. It breaks the distance in the relationship of human and technology, because it is not any longer the user which is subjected to technology, but in this case the display itself has to cave in to the human. Impress is a chance of approach of user and technology, above all, from technology.


It is a matter of a flexible display consisting of foam and force sensors which is deformable and feels pleasantly soft. Impress works with the parameters position and time like other touch screens as well, but in addition to that, it reacts, above all, on the intensity of pressure."

The application was created with Aruino and Processing.






Thanks to Richard Van Tol for the link!

The Interdisciplinary World of Dance and Interactive Technology:

After I finished my post "What happens when engineers and musicians get together? They get Calvin Harris "Ready for the Weekend" with the Humanthesizer and Bare Ink", I thought I'd catch up on the world of dance and technology.

I took a look at an on-line community, Dance-Tech.NET which focuses on the "interdisciplinary explorations on the performance of motion". I was happy to find that two people I know are members of this vibrant group of people:

Celine Latulipe

Dr. Celine Latulipe is a Human-Computer Interaction researcher at the University of North Carolina-Charlotte. She is the lead in the Dance.Draw project, a collaboration between the Software and Information Systems department, the Department of Dance, and a digital artist.

“A Mischief of Mus Musculus”

"Dance.Draw: Exquisite Interaction"

As Celine puts it, "You are more than your eyes and one hand. Why should you have to be less than you are when confronted with a digital device? Exquisite interaction is a collection of research projects...that aim to enrich your expressive creativity in the digital realm by allowing you and your collaborators to use more of your body in that digital interaction"

Visualization code was designed by Mike Wirth, using Processing, to create the art displayed on the Art from Dance page of the Dance.Draw project.

Doug Fox

Doug Fox created and maintains the Great Dance and Kinetic Interface blogs. When he was 42, he decided to study dance - modern, ballet, and jazz. He is interested in the intersection of dance and technology, and more recently became involved in the study of animation related to dance and movement.

The following is from Doug's "About" page - the videos and links are worth exploring:

"As a starting point, I'd like to encourage readers to visit the Movement Is at the Heart of Scientific and Technological Change background page. Here you will find 16 videos (plus links to more videos) that show in very concrete terms how new computer interfaces and digital devices being used in a range of fields are, in essence, body-centric and movement-centric..."

Here is a sample of Doug's topics:

Dance Theater Workshop's Twitter Community Choreography
Dance Vlogging, Will this Video Genre Increase in Popularity?
Shoot Dance Videos with the new iPhone 3GS
Prodigy Warrior's Dance Combines Stop-Motion Animation and Puppetry
Choreographing Gesture Controls for Interactive Devices (be sure to read the comments and link to Arizona State University: School of Arts, Media and Engineering)

Doug Fox writes about topics that are truly interdisciplinary:




Dance Your Ph.D. Contest - A Wonderful Merging of Dance and Science
This contest was funded by the American Association for the Advancement of Science. The objective of the contest? "Using no words or images, interpret your Ph.D. thesis in dance form".

Contest winners were paired with choreographers, who created a new dance work based on a peer-reviewed article in a scientific journal. (The Science Dance Match-Up Challenge)

I just had to post the videos here - you can read detailed descriptions about each video on YouTube:

The role of Vitamin D in beta-cell function


Cerebral activation patterns induced by inflection of regular and irregular verbs with positron emission tomography: A comparison between single subject and group analysis


A Molecular Dance in the Blood, Observed


Popular Choice: Physics Tango "Single Molecule Measurements of Protelomerase TelK-DNA Complexes."


For a look at the 2008 "Dance Your Ph.D." videos, and the the videos of the 2009 contest entries, visit the 2009 AAS/Science Dance Contest web page.

The four videos generated by the professional choreographers can be accessed on Vimeo.

Links to the scientific articles, the bios of the choreographers and scientists, and videos of the choreographer's renditions of the scientist's work can be found on the Science/AAAS website:

The Gonzo Scientist: The Science Dance Match-Up Challenge
John Bohannon, Science, 4/17/09

SOMEWHAT RELATED
Usually I reserve this section for links and information from external sources, but this time, I thought I'd share a few of my opinions that are somewhat related to this topic.

Why do I think interdisciplinary pursuits are important?

In my opinion, to move forward, the arts and other disciplines need to embrace the interdisciplinary way of thinking. There is much that is mixing and converging as I type these words. There is less emphasis for young people to pick one little corner of a field of study and make it their life's obsession. I have always had an "interdisciplinary" approach to life, ever since I can remember. I attribute this to my parents, who nurtured me as a musical, dancing, artistic child to pursue my talents at a young age, and when faced with choosing a college major, to go for a double major.

My interdisciplinary nature has fueled my journey into the world of technology, and my early background in the arts probably explains why I'm excited about interactive multimedia, extending into the realms of emmersive games, multi-touch and gesture interaction, and technology-supported interaction that takes place in larger public spaces.

Note:

As many of my readers know, I work as a school psychologist in my "day job". I DO miss the time when I was working part-time and taking graduate classes at UNC-Charlotte, but when the economy went downhill, It was necessary for me to return to work full-time.

The upcoming academic year will be busy! With the recent budget cuts to school districts in the in the state of North Carolina, I will have another school added to my schedule. I'm excited that it is a high school for technology and the arts, and that the school has a strong dance program!

I've posted quite a bit recently, since I have plenty to share. Soon I will only have time to post about 3 times a week.

Digital Storytelling Platforms and Multiple Perspectives: A look at the work of Jonathan Harris - Food for Thought for Interactive Timeline Design

I'm in the process of creating an interactive timeline, and as I revisited my links and bookmarks, I came across a link to a video of Jonathan Harris discussing his ideas regarding digital storytelling, overlapping threads, and multiple perspectives.

Jonathan explores real-life stories and celebrates the interconnections between events, ideas, feelings, and people. Linear narrative and linear time lines do not do justice to the richness and complexities of human experience.

"Combining elements of computer science, anthropology, visual art and storytelling, Jonathan Harris designs systems to explore and explain the human world."

@ Yahoo! Video

"Jonathan Harris is redefining the idea of what it means to tell a story. Take a ride through an arctic whale hunt and plunge headfirst into the feelings Harris finds running rampant in cyberspace as he describes what he calls “storytelling platforms.” "

Below are links to two story-telling platforms described in the presentations. The Whale Hunt is organized so that the user can explore the story through a variety of perspectives and interfaces, and at different points in time.

The first screenshot shows how the user can select one of the "cast" members to see how the story unfolds from that person's point of view.

The Whale Hunt

"Every few minutes, the system searches the world's newly posted blog entries for occurrences of the phrases "I feel" and "I am feeling". When it finds such a phrase, it records the full sentence, up to the period, and identifies the "feeling" expressed in that sentence (e.g. sad, happy, depressed, etc.). Because blogs are structured in largely standard ways, the age, gender, and geographical location of the author can often be extracted and saved along with the sentence, as can the local weather conditions at the time the sentence was written. All of this information is saved...The result is a database of several million human feelings, increasing by 15,000 - 20,000 new feelings per day. Using a series of playful interfaces, the feelings can be searched and sorted across a number of demographic slices, offering responses to specific questions..."

Jonathan Harris also presented at the December 2007 EG Conference. The video and related information can be found on the TED website.

Jonathan Harris: The art of collecting stories

If you have some time on your hands, explore Jonathan's Universe project:

This photo depicts the nine stages of the Universe environment.

"Universe is a system that supports the exploration of personal mythology, allowing each of us to find our own constellations, based on our own interests and curiosities. Everyone's path through Universe is different, just as everyone's path through life is different. Using the metaphor of an interactive night sky, Universe presents an immersive environment for navigating the world's contemporary mythology, as found online in global news and information from Daylife. Universe opens with a color-shifting aurora borealis, at the center of which is a moon, and through which thousands of stars slowly move. Each star has a specific counterpart in the physical world — a news story, a quote, an image, a person, a company, a team, a place — and moving the cursor across the star field causes different stars to connect, forming constellations. Any constellation can be selected, making it the center of the universe, and sending everything else into its orbit."

Universe was created using Processing, which is an open-source software that is used by people from all sorts of disciplines to create interesting interactive information visualizations and more. The data used in Universe is from Daylife. For more information about Daylife, visit the Daylife Labs.

Sidenote:
Jonathan Harris collaborated with Sep Kamvar on the We Feel Fine project. Sep Kamvar teaches classes like "Social Software" and "Computational Methods in Data Mining" at Stanford University. Sep is part of the Stanford Human Computer Interaction (HCI) Group.

Stanford's HCI group's weekly seminars highlight a variety of interesting speakers. Current and previous talks are available via Stanford OnLine. You can link to current presentations and videos from the Human Computer Interaction Seminar website. If you are curious, past presentations/abstracts can be accessed on-line alphabetically or by date, going as far back as 1990.

Dan Saffer, author of Designing Gestural Interfaces, presented "Tap is the New Click" on January 23rd, 2009 at Stanford.  You can access the video directly.

Pop! Tech PopCast

Visualization software used to create Radioheads "House of Cards"- no cameras used.

Take a look at my recent Technology Supported Human-World Interaction post, "We were rolling computers all day: The making of Radiohead's House of Cards Video". The music video was made without the use of cameras, and relied on visualization software and scanner/sensors. Follow the links from the post for additional information. ( I came across the video on the Kinetic Interface blog)