Predicting how communities change in space and time requires an understanding of how mechanisms such as environmental filtering and biotic interactions shape distributions and abundances. However, ecological communities are often context dependent, so mechanisms that are important in certain environments may not be in others. To understand how context dependency affects the prediction of community structure, we asked: Can the environment predict the outcomes of community assembly? Using fish association networks estimated with Markov random fields for over 700 lakes in Ontario, Canada, we tested if species association patterns, representing potential community assembly mechanisms, varied as a function of the environment. We examined the effect of the environment at two scales: pairwise and community level, summarizing potential mechanisms between species and across whole communities. The environment was a strong predictor of community level species association patterns but not of pairwise patterns, suggesting that the cumulative outcome of mechanisms structuring communities can be explained by the environment. We then tested if community level patterns were associated with the uniqueness of a lake’s species composition. We found that as species association patterns became stronger, they lead to lakes with more common species compositions. Taken together, our results show that variation in the outcome of community assembly can be explained by the environment using community level patterns, offering a way for community ecologists to study context-dependency in community structure across differing environmental gradients and species compositions.