How would you go about -succesfully- and systematically eliminating a specific group of people from a society? I believe what Hitler did wrong was basically give the Jews a head start by warning them. Of course, the gradual degradation of the Jewish people to a sub-human level was necessary to turn the German people against them, since I imagine just deciding overnight to kill off/banish them would have upset a lot of otherwise morally sound people.
But I believe the element of surprise would play an important role in the process of a succesful genocide. That is, from the perspective of the (to be) victimised population group, there may be no suspicion of the governing power's intentions. As a government however, preparation is key. I can only sympathise with the nazi law to remove Jews from civil service in this particular instance, for obvious reasons.
Chapter I: Data Collection
Let's say we'd want to purge the Jews again, but we want to do it right this time. We'd have a pretty comprehensive list of state files about our Jewish population at our disposal. Of course there is the question of where do we draw the line. The Nazis were pretty harsh in this aspect, that is to say that if you were Jewish by ethnicity but completely westernised in every way then you would still be counted as a Jew. (See Nuremberg race laws)
This raises many more questions about non-Jews married to Jews, the degree of Jewsihness by each generation etc.. Which is very difficult to figure out.
So let's say for our convenience that we want to purge all first and second generation Jews, meaning mommy and daddy who go to the Synagogue still, and their children, but not their children's children. Provided they convert to christianity or islam of course. We would go aboutmaking a list of all Jews provided to us by the state's ethnographic data. That's the easy part. The difficult part is making sure that no one slips through the cracks. Surveillance of Synagogues is an obvious next step, time consuming but not infeasible.
The main problem with this issue is that the more people start working on this, the larger the chance that your element of surprise will be lost somehow. From here I conclude that in order to commit a succesful, as complete as possible genocide, you will need to take a slow and long-term approach to compile all data as opposed to the quick, hurried, get-them-out-asap approach which leaves way too much room for error.
This concludes the first chapter of my genocide hypothetical. The next chapter will fast forward a few years towards rounding up the to be eliminated group of people in one fell sweep, ensuring that no one slips through the meshes of the net.