When we share ideas through conversation, we convey a specific mental image to another person. However, mental images don’t always match, and the effort it takes to stop and achieve common ground would interrupt conversational flow. We are currently investigating the design of a system that translates a speaker’s gestures and speech into a visualization of their idea. The challenge of designing such a system is complex, as conversation is composed of many intricate factors. We focus the results in this paper on one factor in particular: speaker of reference. We ran a study with 26 participants, with a prototype meant to handle gestured descriptions of object size and noted the effects of speaker frame of reference. From our analysis of reference frames used during size descriptions, we draw implications for how our proposed system may detect and translate frame of reference to produce visualizations of the user’s mental image.