Suspicion ruled every corner of the quarry worker — employer wage and labor movement struggle in the 19th Century. And the Cape Ann Quarry workers & quarry owners were no different. The common language for the immigrant populations which streamed to Cape Ann seeking works was the language of the hand tool.
Not the words that came from their mouths.
Over the course of the several strikes related to hours and wages, the workers often could not even organize themselves enough to be able to get to a meeting at an agreed upon time. This resulted in great disdain on the part of the employers. Over time, as English was acquired by the immigrants, they became more efficient in presenting their demands. And this increased the violence of the response by the quarry owners. They were losing the upper edge in a business which in fact, had not produced a single millionaire quarry owner anywhere in the United States. The owners themselves then organized into the Granite Manufacturers Association, in which the Rockport Granite Company played a major organizational role.
“Someone stood up and started to speak in a foreign language…”
-Charles Rogers, on the occasion of the 1906 Babson Farm Quarry Strike
Break: Stone Water Heart 
One has to be careful limiting the organization of industrial action to within the unions, for the immigrant social and benevolent societies played a significant role as well and allowed quarry members to organize under the strength of ethnic maintenance and in their own language. Much that Swede and Finn granitecutters learned about unions and industrial action came from experiences in the “Old Country,” where labor unions were a more integrated part of working class life and politics (something that continues to this day).
You are correct. However the greatest social and benevolent societies were located in Barre, VT.
At least as far as I have researched.