User Tools

Site Tools


NSF Proposal Brainstorming

This page is for working out ideas related to a potential NSF proposal/collaboration.

Research Question

Students of art, digital media, and multimedia programming are becoming interested in mobile handheld devices as a platform for interactive media. Programmable mobile devices such as the Apple iPhone can be used to host image, audio, and video applications, but can also be used as remote control devices. A handheld device could be used to control an interactive digital art installation, or a presentation running on a computer connected to a projection display. An even more intriguing possibility is the potential of using multiple devices collectively over a network for applications and experiments in communication, collaboration, interaction, and emergent behavior.

This research will explore two specific aspects of designing applications for mobile devices. First, how is programming for a mobile device different from programming a desktop computer? Can software design methodologies developed for creating desktop applications be used or adapted directly to creating applications for mobile devices? Second, what possibilities in presentation and digital media can be realized when one or more participant can interactively control or create content from their own handheld device? Related to this, how can content designers from the art and digital media world effectively create for these devices without having to become software developers?

Learning goals

The learning goals of this study are: 1) learn how to program mobile handheld devices and how this process differs from programming desktop computers, 2) learn a methodology for designing robust and maintainable program code, and 3) learn techniques for communication between wireless mobile devices and computers on a local network or over the Internet. These goals will be addressed first by our study of the iPhone Software Development Kit (SDK), which uses a novel architecture and development language (Objective-C) not normally covered in traditional computer science curricula. We will gain experience programming for mobile platforms by developing an application that turns the device into a remote controller for interactive applications, using an established design methodology. Next, we will evaluate the ability of various digital art installation applications and presentation software packages to be configured for external control via handheld devices (either directly or by making modifications). An example of a system for developing digital media which can be programmed to accept external control information is the Processing system ( Our application will be developed to communicate with this and other packages.

Another important goal is to find ways to bridge the gap between computer scientist and digital media designer, exploring ways for those with little or no programming experience to create content for mobile devices. To this end we will be evaluating existing multimedia development systems that can be targeted to the Apple iPhone and iPod Touch devices, providing an entry point for artists and digital media designers who may find the novel architecture or the iPhone SDK a significant obstacle to approaching this platform as a development vehicle.

Roles of the faculty and student researchers

The CS faculty advisor(s) will guide the student through available resources on learning the iPhone SDK. As a registered Apple developer, the advisor acts as liason to Apple through the iPhone Developer University Program. This allows students to get access to the full resources for iPhone SDK development provided by Apple without cost, though they must register as team members specially through the University program. This is because Apple controls the distribution channel for all official applications developed to run on the iPhone or iPod Touch mobile computing devices, even those made available for no charge.

The CS faculty advisor will also participate in software design and development, provide a host computer and lab space for the student, and interact with other digital media designers on campus in order to evaluate the impact and usefulness of our work to their students and projects.

The Art faculty advisor(s) will assist in digital multimedia content creation, give insight into issues such as how artists work with technology, and provide classroom laboratories so that students in digital arts can experiment with the emerging software and provide feedback useful in the development of the new technologies employed.

The student researcher(s) will create software designs and implement these using Objective-C, OpenFrameworks, and the iPhone SDK. The student(s) will also be responsible for designing experiments and lab experiences for digital art students with assistance from the faculty advisors…

{TODO add more to Art advisor and student roles here}

Impact on CS General Education Courses

In 2007, enrollment in our introduction to computer science for non-majors course jumped from 7 to 24. While this may have been driven by everything from difficulties in the scheduling process for first- and second-year students to better-informed advisors, we do not have a clear idea of why this jump occurred. We do know, however, that enrollments have stayed high, which bodes well for the future of the program. Enrollments since spring 2007 have been 24, 33 (two sections open), 8 (for a summer session), and 18 for the coming fall. Also starting this fall, the introductory course will begin to count as a “general education” course, so demand will likely surge again. While we have seen a larger number of students continuing to courses that count for the CS major, the effect has been less-pronounced. One reason for this may be a relatively larger number of upperclassman registering for this course. Another may be that the course doesn't impact them directly, because the content does not seem related to their lives or future work.

The idea behind this proposal is to involve mobile devices that student now use, such as cell phones and MP3 players, so that they have a hands-on way of seeing and sharing the results of their programming efforts directly.

The implementation of the proposal involves 1) getting a system working on the iPhone SDK platforms that can run JavaScript and/or Processing and 2) get these devices into the students' hands (either a classroom set, buy them as part of a course fee - but would that put up too much of a barrier to a gen-ed course?). Of course, if they already own them they can use their own.

This proposal will have an interdisciplinary component. One important result of this effort for the Art program, for example, would be the introduction of more programming-literate students into their classes, particularly the Digital Media and Advanced Sculpture courses, which often involve programming for animation, collecting data from sensors, and using microcontrollers.

There are several potential problems with this idea, which the project will attempt to track. One is that students who enjoy the simplified programming interface to their mobile devices may find that their interest wanes when they move on to learning object-oriented programming in all of its curious complexity, producing programs that run only on desktop or laptop systems. We would like to answer the question “Does the use of mobile devices in introductory classes attract students to CS in general, or only to using mobile devices?”

(now talk about how to answer the question)

Impact on CS Core and Electives


Impact on Digital Art courses



What form would a non-traditional programming environment take? We would create a Domain-Specific Language as described in the following CFP (from the IFIP Working Conference on Domain Specific Languages, July 15-17, 2009, Oxford, UK):

Domain-specific languages are emerging as a fundamental component of software engineering practice. DSLs are often introduced when new domains such as web-scripting or markup come into existence, but it is also common to see DSLs being introduced and adopted for traditional domains such as parsing and data description. Developing software using DSLs has many benefits. DSLs are often designed based on existing notations that are already in use by experts in a given domain. As such, successful DSLs often reduce or eliminate the effort needed to transform the concept or innovation produced by the domain expert into an executable artifact or even a deliverable software product. DSL implementations can capture and mechanize a significant portion of the repetitive and mechanical tasks that a domain expert traditionally needed to perform in order to produce an executable. DSLs can in many cases capture and make widely available special expertise that only top specialists in a given domain might have. By capturing expert knowledge and reducing repetitive tasks, DSLs often also lead to software that is significantly more portable, more reliable and more understandable than it would otherwise be.

DSLs can be viewed as having a dual role to general-purpose languages: whereas general purpose languages try to do everything as well as possible, DSLs are designed to find a domain where they can solve some class of problems – no matter how small – in the best possible way. The ultimate goal of using DSLs is to improve programmer productivity and software quality. Often, this is achieved by reducing the cost of initial software development as well as maintenance costs. These improvements - programs being easier to write and maintain - materialize as a result of domain-specific guarantees, analyses, testing techniques, verification techniques, and optimizations.

See also WG211 online

An alternative introductory paragraph (after brainstorming):

Digital artists and media programmers, not to mention students in CS and Digital media programs, are intrigued by the idea of using platforms such as the Apple iPhone as a way of bringing unique content and interconnected installations to the world in non-tradition formats. Such users would benefit from the ability to develop applications for this platform using a non-traditional programming environment that shields the new iPhone developer from details of interacting with Cocoa, Objective-C, and the UIKit while providing a path for the motivated user to approach these industrial-strength frameworks and technologies.

DIY / Art approaches to Technology

From Hans Christoph Steiner:

Have an iPod you want to repurpose into a hackable music machine? Install iPodLinux and make music with Pd! Want to play Doom on your iPod Video? Come install Rockbox! Did your trusty iPod's battery finally die, or disk give in? Come and break open your iPod, either the software, the hardware or both. Give it new life and new purpose. Also, we have on hand our stash of old iPods for harvesting parts, and the newfound skills for frankensteining dead iPods into living ones. Join Hans-Christoph Steiner and Chris “the Widget” DiMauro with your own iPod, or try your hand to see if you can create a living one from our pile.

This workshop is a place to get exposed to the possibilities. We've been hacking on some iPods to see what's possible. Lots of this software is kind of raw, but you can do lots of fun stuff now. We are just getting started, we are having fun exploring the options. Now we want to help you do the same.

Check out our wiki to see if your devices are supported.

References and Resources to Explore

iPhone Dev Center

iPhone Developer University Program

iPhone Developer Library


nsf_proposal_making_mobile_platform_development_accessible.txt · Last modified: 2009/10/02 14:18 by scarl