Why do some homogeneous groups face backlash for lacking diversity, whereas others escape censure? We show that a homogeneous group’s size changes how it is perceived and whether decision makers pursue greater diversity in its ranks. We theorize that people make different inferences about larger groups than smaller ones—with consequences for diversity management—due to Bayesian reasoning. This can produce sensitivity to a lack of diversity in large groups and limited sensitivity to a lack of diversity in small groups. Because each group member represents the outcome of a hiring decision, larger homogeneous groups signal a diversity problem more strongly than smaller homogeneous groups. Across three preregistered experiments (n = 4,283), we show that decision makers are more likely to diversify larger homogeneous groups than smaller ones and view larger homogeneous groups as (i) more likely to have resulted from an unfair selection process; (ii) less diverse; (iii) more likely to face diversity-related impression management concerns; and (iv) less open to the influence of newly added underrepresented members. Further, (i)–(iii) mediate the relationship between homogeneous group size and decisions to diversify. We extend our findings to S&P 1500 corporate boards, showing that larger homogeneous boards are more likely to add women or racial minorities as directors. Larger homogeneous boards are also rarer than expected, whereas smaller homogeneous boards are surprisingly abundant. This suggests that decision makers neglect homogeneity in smaller groups, while investing extra effort toward diversifying larger homogeneous groups. Our findings highlight how group size shapes diversity-related perceptions and decisions and identify mechanisms that kickstart diversification efforts.