Every one should be either casual or dressy. If mom is in a ball gown, but dad in T-shirt and shorts, the photos will look strange and not cohesive.  Very out of place. All the family members in a dressy casual works well. 

I have also done family photos were the family loved snowboarding together. The family wore shirts from different favourite  mountains. It was special to them. 

Colours opposite each other on the colour wheel are complimentary, they work well together. 

Colours beside each other are harmonious and also work well together. 

Depends on the look you are going for. If photos are taking place in grassy park, pink will pop and stand out against the background of green. On the other hand, blue tones beside green are harmonious and in a more subtle way, also look great together. 

Notice the photo above of the kids. The colours are harmonious with the yellows of the grass, greens of trees, blues of boys shirt and girls dress. But then the girls dress also has a pop of pink that adds a touch of contrast and depth. 

One person in pattern is great. You can then pull colours from that pattern and have others wear solids in the colours. 

Take this photo of me and my girls for example. The girls dress and rompers are each colours in the flowers of my romper. We are each wearing something we love, while co-ordinating!

Using the first 3 tips gather a few options for all members of the family.

Lay out the clothes on the floor or bed. You can then swap in other options to see which colours and tones work best with each other.