The idea is to write a program which takes a black and white silent movie and generates an interesting sound track for it.
How to accomplish it:
I had originally planed to generate pitches based directly on some sort of analysis of the movie, but so far this sounds really bad. I thought maybe in stead, I would use a 1/f generated sound (using the pseudo algorithm from Martin Gardener). That way the melody would sound “nice” (or better at least). I could then use the movie analysis to “adapt” the melody.
Part where it gets fuzzy:
Here is the fun part. 1) I was thinking it would be nice if the speed, note durations, and volume reacted to the movie. 2) I was also thinking it would be cool if I designed some “mutator” methods that were called in reaction to the movie. The idea is to generate several short melodies using 1/f, and then create variations using the “mutator” methods, so that there is a certain cohesiveness to the music. So something sort of genetic algorithm inspired (as in no fitness function). 3)It would also be cool if I could figure out how the filters work so I could have the timber change with response to what is going on in the movie. 4)It would also be nice if some kind of harmony was going on… But I kind of doubt I will have much time for that.
As for the actual movie analysis, I plan on using openCV (of course). Things to consider would be the number of blobs detected, the and the size of blobs. I think I might be able to pass the sum of the areas of blobs off as “the total movement” and use that for something. Since the movies I am focusing on are black and white (or gray scale) it might be good to track how dark/light a given frame is (I envision this being useful for controlling the timber).
Where I am currently at: 1) I can successfully import, run, and do blob detection on movies with processing. 2) I can send pitches (values) to PD 3) I believe I have a functional 1/f pitch generator. I used Gardner's pseudo algorithm with some tweaking. I limited the range and forced it to map into a predesignated three octaves of a C major scale. However, it sounds pretty good, so I am sticking with it.
What I need to do (in no particular order): 1) Create mutator methods. 2) Decide how best to analyze the movies 3) Decide how to use that analysis (to call mutator methods, etc.) 4) figure out how to make the timber do what I want in PD 5) Put it all together 6) Time permitting, think about harmonization