Cannot change the Incident State from "In Progress" to "New"

It is going back to In Progress

There is an out-of-box (OOB) business rule "Incident State Active on Assignment" which checks this field.

If the Assigned to has a value and state is "New", then it sets the State and Incident State back to "In Progress"

Make sure that the Assigned to field is empty when you want to set it back to "New"