The problem is that ASP.NET does not get to know about this extra or removed listitem. You got an number of options (listed below):
Disable eventvalidation (bad idea, because you lose a little of security that come with very little cost).
Use ASP.NET Ajax UpdatePanel. (Put the listbox in the Updatepanel and trigger a update, if you add or remove listbox. This way viewstate and related fields get updates and eventvalidation will pass.)
Forget client-side and use the classic postback and add or remove the listitems server-side.
I hope this helps.
or
<%@ Page EnableEventValidation="false" %>