Transformations: Students find creativity at intersection of art and engineering

Inspired by the desire to help broaden boundaries for vision-impaired people, three Princeton University students created an armband device that allows a wearer without the ability to see to interpret color. 

The project emerged from a new class offered for the first time this spring, "Transformations in Engineering and the Arts," and lived up to the name of the course. In addition to transforming the sensation of color from a visual to a tactile experience, the students transformed an idea born on a whiteboard into a product interweaving engineering concepts and artistic practices.

"Having a forum where our imagination is the limit has been an incredible experience," said Nora Bradley, a sophomore majoring in mechanical and aerospace engineering who worked with senior music major Noah Fishman and sophomore Sunny He of electrical engineering.

The team's device uses a sensor to identify whatever color passes under it, then "displays" the hue by turning a rotary dial labeled in Braille. Two motorized pointers slide along metal bars to indicate the relative saturation and brightness of the color. 

"[In this class,] you tend to work with people with different backgrounds, which is something you don't generally see," Fishman said. "I had a lot of fun working with everyone trying to work out the best way to streamline the device and solve problems together." 

He, who worked on the coding and color sensor portion of the project, said the class built a community of faculty and students that created a positive environment to explore different ideas. 

From left, students Noah Fishman, Nora Bradley and Sunny He prepare a device they designed that allows visually impaired people to experience color.

A partnership between the Council on Science and Technology, the School of Engineering and Applied Science, the Department of Music, and the Lewis Center for the Arts, the class stemmed from several conversations with faculty and students interested in the intersection of arts and engineering.

"We were focused on leveraging a combination of the creative processes we use in engineering and the arts, with 'transformations' as our framework," said Naomi Ehrich Leonard, who led the development of the course and is a professor of mechanical and aerospace engineering and director of the Council on Science and Technology. "For example, the students designed artful visualizations of sound and compositions that were translations of movement into sound," she said. "It's about working across disciplinary boundaries and not being confined to just one academic identification."

Computer science major Drew Wallace demonstrates to Daniel Day, assistant vice president for communications, a system that captures the movements of a person's hands and interprets and portrays them in stylized computer graphics settings.

The class, which had 11 students, was organized around four modules in the first half of the semester: visuals, sound, structure and movement. The modules included lectures, hands-on activities, discussions of aesthetics in pieces everyone watched or heard, mini-design challenges, and tutorials on tools and resources available in a newly created teaching space called StudioLab.

Adam Finkelstein, a professor of computer science, led off with the module on visuals; Jeff Snyder, associate research scholar in the music department, taught the unit on sound; civil and environmental engineering faculty members Maria Garlock and Sigrid Adriaenssens led the unit on structure; Leonard led the unit on movement. They were joined by faculty members from the Lewis Center for the Arts: Jeff Whetstone and Martha Friedman from visual arts; Aynsley Vandenbroucke from dance; Jane Cox from theater; and A.M. Homes from creative writing.

"We all became students again and sat in one another's modules," Leonard said. "We can't help but wonder how this affects our own creative work. It's incredibly generative that way."

Sophomore Kathleen Ma and two classmates created a project called "Mother Womb," a portable tent that envelops the user with calming visual, auditory and tactile sensations.

The second half of the semester was dedicated to independent or group design projects in which students defined their own questions. Jonathan Zong, a sophomore majoring in computer science, designed a system in which a wheeled robot tracks a human performer's movements through a motion-capture system and translates that data onto paper with a pen that the robot carries. His idea was to study interactions of the formalized behaviors of a robot with the creative and expressive movements of a person, as well as to investigate the transformation of physical movements that occur over time into a two-dimensional drawing that is viewed at a single point in time.

"The person is doing whatever they want but also regulating their movement to imprint some intent on how the robot is moving," Zong said, "so they have to formalize their own movement. This whole interaction is mediated by the feedback loop, so you get a weird blending of behaviors."

Zong said the faculty and expert support staff gave students the confidence and support they needed to be creative.

"I've hit a few snags where I couldn't figure out how to work the mechanics of my robot, but staff was available to help brainstorm solutions," he said. "It’s great to know we can problem-solve together."

Sophomores Zong (left) and Kathleen Ma (right) work with Aatish Bhatia (center), associate director for engineering education at the Council on Science and Technology.

Several other projects include a motion-capture game that pits one player moving in a physical space against another operating a computer; an hourglass that uses water instead of sand and continually refills itself (but slowly leaks) to play with different concepts of time; and a system that captures the movements of a person's hands and interprets and portrays them in stylized computer graphics settings.

Junior Drew Wallace, the computer science major who made the hand-interpreting system, said the project pushed him to satisfy aesthetic goals, which made it different from typical code-writing projects that are tightly focused on achieving a functional goal. "This is more of an exploratory process," he said. "I wound up throwing out a whole lot of code. There were a lot of visual effects that were a lot of work, but I didn't like the way they made it look. Working from an aesthetic perspective was really exciting because there were a lot of things I wouldn't have explored technically."

Helen Lin, a Program Two major in the Department of Art and Archaeology, created "Mother Womb," a project with Mariah Wilson, a comparative literature major, and Kathleen Ma, who is majoring in the Woodrow Wilson School of Public and International Affairs. The three sophomores wanted to craft a full-sensory experience while evoking imagined memories and escaping reality. They built a portable tent that envelops the user with visual, auditory and tactile sensations.

"This was a literal transformation of synthetic materials used for the 'womb' to create a natural environment," said Lin. "As a visual artist, I don't normally work with physical things. So I learned a lot by working on this physical environment, figuring out what works and what doesn't."

Junior Max Greenwald demonstrates a game he created in which one player interacts with lights controlled by another player and a computer. At right, he discusses his project with Susan Marshall (center), director of the Program in Dance, and Naomi Leonard, a professor of mechanical and aerospace engineering, who led the course.  

One big aspect of getting the class started was acquiring a sufficiently versatile space. The StudioLab was renovated to support the course. It is highly customizable with a dance floor, motion capture, theater lighting, sound system, 3-D printer, laser cutter, CNC machine (a computer-controlled cutting device), and virtual reality and projection equipment.

The role of the Council on Science and Technology was to formulate the concepts for both the course and the StudioLab space and to bring those ideas to reality, said Evelyn Laffey, the council's senior associate director. She emphasized that the council is opening the StudioLab to the University community with the goal of spurring further creative ideas for courses and collaborations among faculty members of diverse disciplines who might not otherwise work together.

"It's thrilling because students are often forced to decide to go with science or art when they're choosing their majors," said Cox, who taught a taught a mini-module on lighting design for the course. "We need to think of the world differently to see the bigger picture and I hope this class will help facilitate that."

After he demonstrated his game that combined motion-tracking technology, theater lighting and computer game design, Max Greenwald, a junior with an independent concentration in computer science and public policy, said the best part of the class was the opportunity to play. "This project really energized me to be a maker and a doer and physically create something," he said. "I couldn't be more thankful to the class for allowing me to create something like this."

The course will be offered again next spring.

Senior Hannah Miller demonstrates the continuous, water-fed hourglass she constructed with senior Neeta Patel to Axel Killian, assistant professor of architecture.

The class took place in a newly developed teaching space called StudioLab in Fine Hall. The space allows students to integrate studio lighting, motion-capture technology, performance space, 3-D printers and other technology. The Council on Science and Technology is making the space available for other teaching and research initiatives.