The delete button is there, iirc, but it's grayed out.  Deleting Unit Designs is a planned but NYI (not-yet-implemented) feature.

Edit: if you really want to get rid of some of them, you should be able to delete the corresponding files in the "C:\(UserName)\Documents\My Games\Elemental\Units\" folder (on Vista, if that stuff is on your c drive).  I'd advise not having Ele running while you delete those files.

I believe the original post was referring to deleting unit designs, not actually-built-units.  There's also no way currently to disband a built unit, or at least there wasn't last I checked.

I have no idea if they plan on recycling the people into the civilian population, or whether that will depend on the unit actually being in a city.