Aside that it's not realistic option to kick anyone out - I guess it'd be easier to break up NATO completely and make a similar alliance to replace it, but then probably few others would drop out - Turkey is one of countries with most strategically important location. Getting thrown out would cause Turkey to find new allies and those wouldn't be ones we like. As bad as it looks now, I'm pretty sure Turkey would be more trouble outside NATO than it is now.
But we ("we" being EU and NATO in general) need to start systematically punishing harder countries that use requirements for unanimous decisions for blackmail. Greece got away with it till Macedonia added the "North" and now we have these two. Either they need to see clear loss of future support and benefits or we need to start changing the rules so that a single county can no longer do this kind of sabotage, ramp it up to at least two or three required for a veto.