You hear many opinions on how to match colours and complexion and patterns and size. Advices on matching fabric and build are neglegted. That is weird, since the right fabric for you has a lot to do with your build and the way you move.
Fabric for small guys and big guys
In general, the larger the size of your body the more stress you put on the fabric, and the stronger you need the fabric to be. Just think of boxing, and how much the weight influences the power transfer. A 340 grams business suit cloth might be a robust choice for a small guy, whereas a big guy would need a 380 grams cloth or 400 grams cloth to get the same solid drape and wear.
Body weight and knitwear is also an area to consider. Again I would say that small guys can more easily wear, say, a fine cashmere pullover than 220 punds rugby fellow, who perhaps should go for a cashmere cable jumper instead or a merino wool pullover. They expose the knit to wear very differently.
Matching fabric to your acitivity
I would also argue that your daily physical activity level can influence, which fabrics suit you best. Some people are moving around more than others during daytime. They might jump on a bike, when they go for a meeting downtown, or they walk a lot. That means the clothing fabric (and knitwear) must take more wear and tear, and those men should go for a stronger fabric that better fits their physical activity level.