Most commercial virtual reality applications with self avatars provide users with a “one-size fits all” avatar. While the height of this body may be scaled to the user’s height, other body proportions, such as limb length and hand size, are rarely customized to fit an individual user. Prior research has shown that mismatches between users' avatars and their actual bodies can affect size perception and feelings of body ownership. In this paper, we consider how concepts related to the virtual hand illusion, user experience, and task efficiency are influenced by variations between the size of a user’s actual hand and their avatar’s hand. We also consider how using a tracked controller or tracked gestures affect these concepts. We conducted a 2×3 within-subjects study (n=20), with two levels of input modality: using tracked finger motion vs. a hand-held controller (Glove vs. Controller), and three levels of hand scaling (Small, Fit, and Large). Participants completed 2 block-assembly trials for each condition (for a total of 12 trials). Time, mistakes, and a user experience survey were recorded for each trial. Participants experienced stronger feelings of ownership and realism in the Glove condition. Efficiency was higher in the Controller condition and supported by play data of more time spent, blocks grabbed, and blocks dropped in the Glove condition. We did not find enough evidence for a change in agency and the intensity of the virtual hand illusion depending on hand size. Over half of the participants indicated preferring the Glove condition over the Controller condition, mentioning fun and efficiency as factors in their choices. Preferences on hand scaling were mixed but often attributed to efficiency. Participants liked the appearance of their virtual hand more while using the Fit instead of Large hands. Several interaction effects were observed between input modality and hand scaling, for example, for smaller hands, tracked hands evoked stronger feelings of ownership compared to using a controller. Our results show that the virtual hand illusion is stronger when participants are able to control a hand directly rather than with a hand-held device, and that the virtual reality task must first be considered to determine which modality and hand size are the most applicable.