This project presents the full integration of visual representation elements and use of sonic-generated data using ‘Java’ language and explores the nature of audio-visual interaction. It provides a holistic approach to the implementation of weighted assignment of frequency bandvalue in order to portray different aspects of sounds and how it can show its waveform behaviour via visual representation of data .
“Space Trance” is a generative audiovisual that was created using java language with the help of the ‘Processing3 IDE’ and ‘minim’ library which will be explained in greater detail below. There is a rich history of artists representing the visible world through imagery and sounds. But there has always been a barrier between the both. The aim was to bring about a unity and a harmonious relationship between these two opposite spheres. We know that the speed of light exceeds speed of sound by roughly a million times which leads to a delayed audio in response to the visuals. This is a commonly known phenomenon. The aim of the project was to create an interactive audiovisual art project for generative and simultaneous manipulation of motion graphics in response to the frequencies it receives, in a way that is coherent, flexible, entertaining and engaging to experience.
The idea for the project arose from a game called Avicii – Gravity, a mobile platformer that consists of a spaceship moving along a path and has progressive music that adds instruments and beats as the game continues to progress. I looked at this from a critical design viewpoint, in a way that I could combine space-time jump, better known as ‘hyperspace jump’ concept from Star Wars series while the setup looks like a 3D game . The initial idea was to design a minimal wireframe styled elements reacting to sound in such a that it creates an audio-reactive hyperspace jump kind of scene as shown in Star Wars and Star Trek movies where the spaceships suddenly accelerate to lightspeeds and reach another corner of galaxies within seconds.
Aesthetic and Technical Research
There have been several other works relating to my project, one of the most notable being from the ProcessingCubes (2016) by Samuel Lapointe. In this audiovisualisation project, the creators explore the use of spectrum values to assign values to colours and change them, amplitude to control the speed of motion graphics and colour change and the use of array and random to make a random audio-reactive matrix of cubes floating in space being translated from a point perspective.
Another artist I took my inspiration from is Raven Kwok. Raven Kwok (aka Guo, Ruiwen) is a visual artist, animator and creative programmer, whose artistic and research interest mainly focus on exploring generative visual aesthetic brought by computer algorithms and software processes. His collaboration project with electronic music artist ‘Karma Field’ called The HEX was the aesthetic inspiration behind the project. The HEX is a live audiovisual experience and collaborative installation project between visual artist Raven Kwok and musical artist / producer Karma Fields (soundcloud.com/karmafields). The installation includes a surrounding set of screens forming a hexagonal prism, with generative visuals projected onto each vertical surface, making it possible for the audience to experience the piece from different perspectives. Karma Fields performs live in the center of the HEX. The interesting aesthetic of the HEX is the use of minimal graphics, lines constructing and deconstructing into geometric structures and the rotation speed of the motion graphics that were manipulated by the amplitude of the audio being played.
I began by outlining the kind of interactivity that I wanted the audiovisual to have, with the idea of star wars and hyperspace jump in mind. The goal was to create something that a viewer could immediately associate with and want to interact with. Not only this, the viewer could also somehow control the acceleration /movement of spaceship.
Firstly, I wanted to actually show the spaceship accelerating through space passing luminous stars and planets. I also wanted the viewer to understand the nature of interaction of visuals with the changes in sonics. I thought that these could be accompanied by extra visual outputs such as changing sizes of elements, colours and acceleration of graphics to portray the behavioral aspect of the audiovisual in an effective way.
How does the visual transition into these changes? For this, I needed a number of variables so the elements could react to its input. I wanted it to react to the spectrum and amplitude values of the audio; it could accelerate when the value is too high and vice versa. The space and objects could change colours when spectrum bandvalue is changed.
In order to give the audiovisual its functionality, I had to programme its behaviour in the java programming language and give it an audio input to which it could give a feedback. The coding involved assigning the moving elements, the stars and planets, a variable related to sonics which would influence its displacement velocity in the x-z and y-z coordinate system and a variable to accept the size, stroke and fill of the moving elements. In this way, the audiovisual was able to take input from the audio it was provided with. In order to make stars and planets appear scattered across the space, ‘array’ and ‘random’ commands were used .
The first step in the coding aspect of the project was to create an array of numerous 3d stars and a few planets using P3D functions for making 3d shapes. Depending on the bandvalue of audio spectrum, five variables get manipulated that affect the displacement of elements and their colour, its intensity, stroke intensity and size. For example, if the audio frequency is high and the bandvalue of the spectrum is also high, the moving elements will emit a green colour and move at a greater speed; their stroke will get thicker as well. The use of audio variables was made available using the ‘minim’ library functions.
The audio input was bifurcated into two variables : spectrum and score (power of sound). Both spectrum and score were further divided into 3-3 variables : low, medium and high. As spectrum goes from low to high, frequency goes from low to high, which I used to distinguish bass, kicks and high frequency sounds in the audio output. As the score goes from low to high, volume of frequencies go from low to high, which helps distinguishing which frequency sounds the loudest at a particular point of time. All these values are obtained using fft.getBand() function which returns the amplitude of the requested frequency band. This value is obtained as a result of fft.specSize() and spectrum variable being multiplied together; fft.specSize() returns the size of the spectrum created by this transform. Another variable scoreGlobal was introduced as a function of low, medium and high scores combined to get the volume for all frequencies at this time, with the highest sounding higher. This allowed the animation to go faster for the higher pitched sounds, which is more noticeable. Score value also affects the colour of the background (space) as well as the objects which changes as the value of score varies from low to high to low.
The most important element, the spaceship, was then created using beginShape(). The 3D structure of the spaceship was composed of 12 triangles with vertices denoted with variables sx, sy and sz. sx, sy and sx were map based on the variables x, y and z which were derived as a multiplication function of two vector values : angle and amplitude. Since I wanted my spaceship to be hovering in space leading to constantly changing its position, I chose vector variables to denoted the vertices and to smoothen out the to and fro hovering movement, I used law of simple harmonic motion as a function of x and y coordinates while z being kept as absolute since the hovering was occuring only in x-y coordinate system. Since the value of amplitude is seeded with random() function, the displacement of spaceship while hovering will be different everytime we run the code. The display of the spaceship is also affected by the same five variables that affected the display star and planets in the same way.
In order to move the spaceship across the space, I used the help of processing forum to find the best suitable way to translate my spaceship object in x, y and z axis using the mouse. I used 3D CAD-style navigation with mouse. The use of variables camX, camY and camZ affect the position of the spaceship when mouse button is clicked and dragged across the screen. The object is translated on the x-y plane when left mouse button is clicked and held while dragging in any direction and in z-axis when right button is clicked and held while dragging up and down.
The concept of the hyperspace jump being taken from the star wars and other space fiction movies involved three basic elements in space, stars, planets and a spaceship. Since I wanted the setup to look 3D but somehow related to music , my idea was to make stars in the same of 3D diamonds, or more like di-tetrahedral structures and planets to look like a disco ball. After a bit of a research, I figured out this can easily be achieved by using P3D functions to draw 3D shapes. The star and planets both are derived from sphere() function. The difference lies in the different values assigned to sphereDetail() function which controls the 3D details of the sphere being drawn.
The design concept of the spaceship was influenced by the game Avicii -Gravity which has the initial spaceship as an arrow shaped object as shown in the figure 1.
Figure 1. Avicii- Gravity game screen
I designed the spaceship on Google sketchup first as shown in the figure 2. Then I found out the coordinates of the spaceship using coordinate geometry and used beginShape() function to draw the spaceship in 3D.
Figure 2. Design of the spaceship made in Google Sketchup
The variable scoreDecreaseRate was used to soften/smoothen the colour and movement transitions to give a more aesthetically pleasing factor to the audiovisual display.
The aim of this project was to create an eye-pleasing and entertaining generative audiovisual display that people could relate to the concept of hyperspace they have witnessed in Star Wars movies. The result of this project was quite interesting as I created something that reacts to the audio input. One of the most intriguing aspects of this product is the reaction of a person when the movement speed suddenly increases for the first time and it kind of feels like an adrenaline rush. Another interesting aspect from viewer’s perspective is the free control over the spaceship’s movement to make it feel like we are actually flying over spaceship in space. This makes the project more interactive and engaging. The additional things I wanted to do with this project were being able to shoot laser beams from spaceship and destroy stars and planets which I might try to add in the future.
 Richard Marquand (1983). Star Wars Episode VI: Return of the Jedi, Lucasfilm Ltd. and 21st Century Fox.
 Samuel Lapointe (2016). ProcessingCubes.
 Raven Kwok (2016). The HEX. LOT 613, Los Angeles, CA, US. October.
 Processing Forum Discussion (2013). 3D CAD-style navigation with mouse. https://forum.processing.org/two/discussion/1396/3d-cad-style-navigation-with-mouse
 HelloThereAB, Avicii (2018). Avicii-Gravity HD. https://www.youtube.com/watch?v=vnfoFG8L618