The problem is with your when statements. The two when statements are specifying that either leaf a or leaf b can be modified only when the other one isn't configured. If you add the mandatory true statement to both leaf a and leaf b, you will end up not being able to configure either leafs as the when statements will never be fulfilled if one of the two leafs is being configured.
You appear to have conflicting requirements for your YANG model. Perhaps you can elaborate on what you are trying to achieve.