A good question, and the answers are as varied as the demographics of the group.
IMO, how "large" a group should be is a case-by-case determination. Larger groups may be successful if they have already worked together, know each other well, and/or subscribe to a common, well-defined charter. Or, in the absence of that, lot of freedom and simple, basic, agreed-upon responsibilities.
Examples of natural groups... family, people who work long hours together, workers who have a job that demands character, workers blessed with a "family" job environment, strong ideological or religious charters related to practical life, etc. Contributing factors are trust, morality, ethics, adherence to "legal" framework, work ethic, experience, leadership, submission, reasonable elements of democracy, etc. You tailor the above factors for what you personally consider a bare minimum for your style and size of group.
I believe the group should define a charter or constitution with the above taken into consideration. Absolutes should be defined in areas of security, command structure, property, and support/treatment of each other. Provision should be made for agreed-upon disciplinary considerations... it's a fact of group life.
Some personal experiences...
Our family of 7 was/is very close, excluding one parent. We can trust each other with anything. Trust, moral code, work ethic, some resources, knowledge, etc. are all there. Some extended family is also trusted, but not many.
I grew up in a very tight knit church environ. Laziness, hypocrisy, etc. was not tolerated. Some of the best people in the world, 95% would work together with minimal problems. Command structure, trust, moral code, work ethic, etc. already there. The remaining 5% were high maintenance, but would be useful. "sister" churches I visited or lived among also had similarly high ratio of great people.
I work in law enforcement, and 40% of my fellow line officers could immediately band together without serious problems. Another 40% could be convinced to pull their weight, albeit with a lot of strife about losing rank. 15% would be high maintenance. Command structure, legal/ethical/moral code enough to work together, a basic level of trust, etc. The remaining 5% I wouldn't want anywhere near me.
I had jobs in small business. I'd take about 20% of the small biz people. Command structure, work ethic was there, legal/ethical/moral would be problematic for the group.
I had jobs in Fortune 500 companies that developed and made their own electro-mechanical or computer products. I'd take about 5% to 10% of the Fortune 500... the rest were a lot of hide-outs, 'fraidy cats, kowtowers, lazy asses, rule nazis, politicians, butt kissers, etc., and lacked real, applicable life experiences that would at least half prepare them for group survival. It would be a bit volatile, but we could reasonably make it work with about 5-10%. The rest would have to be fended off, I'm afraid. My military experience falls somewhere between small business and Fortune 500. Some platoons worked great together, others had the reputation of being skates.
No matter the rules, if the S ever really HTF, many in your group will want to bring others in. Many of them will be a legitimate asset. Vetting them in a time of tension is very tricky. It brings up a lot of problems and deceits... you need to have a process for vetting and voting.
One of my biggest concerns, having been exposed to the ass end of humanity at all social levels, is how to handle those who unfortunately cannot function in a group environment. Abusive, sexual deviates, lazy, intemperate, liars, thieves, addicts, those who are completely irresponsible in some areas of their life, malcontents, evil spirited, etc. A lot of it can remain hidden... until 24/7 living exposes it. Someone can be a high functioning member of a group but eventually make you realize you have to discipline or ostracize them. How to discipline? How to separate them? They know all your weaknesses and you are forcing them to desperation. They expect a due share of resources--proportional to what they have contributed. Tough situations.
Anyway, got to git for now... very interesting topic!