We were inspired to translate visual information to auditory information. Try it out at http://acsweb.ucsd.edu/~scongero/main.html.
What it does
Music To My Eyes is a website that allows users to hear certain lines through an image as music. Frequency data in the image is converted into melody, allowing users to experience image and color in a new way. Users can choose an image (or upload their own) and hear these melodies play, as well as view the sheet music.
How we built it
Challenges we ran into
Deciding how to extract music lines from an image was our first hurdle. We chose to use a Sobel operator to understand the locations of the largest horizontal color differences along a row through the image, and then convert the brightness of the pixels in this region into a frequency. The x-axis location of each note in the image dictates its timing. Other challenges included converting the notes extracted from the image into actual sound using the Tone.js library, and also quantizing them in both time and frequency to render them as sheet music using the VexFlow library.
What's next for Music To My Eyes
One future direction for Music To My Eyes is creating more settings and controls, to allow the user to create more complex music. As one example, we could have location information around a note in the image to control a low-pass filter for that note. We would also like to allow the user to draw arbitrary curves through the image which could be converted into music.
Try It out