It's hard to word it right. It's more "except members of a faction other then monsters and ruffians". So... A wildling belong to the "Denizens of the Wildlands faction, or a wildling belonging to Magnar's faction could both attack, but a wildling belonging to "Monsters and Ruffians" could not. I think what is confusing is not really the wording, as you probably naturally assumed that a wildling belonging to Magnar could attack you, it's more the realization that those wild lands have their own independent factions different from the regular "filler" monsters that I think escapes people.
