This is what I've done to transform an enemy after it reaches zero hit points.
The first thing to do is set the enemy to Immortal in the Troops tab of the database. Right-click on the enemy in the battle window, and select Immortal. With Immortal turned on, the enemy can get hurt even to 0 HP, but can never die.
Now add a new Battle Event for the troop. Set the Condition to Enemy [X]'s HP 1% or below, where X is the enemy you want to transform. Set the Span to Turn.
The first event command should be a Conditional Branch. On the last tab of Conditional Branch, select Script, and enter this:
$game_troop.enemies[X].immortal
X should be one less than whatever number the enemy is in the troop. If it's the first enemy you added to the troop, then X would be 0. If it was the third enemy, then X should be 2. Why do I add this conditional branch? Well, when I tested the battle without it, sometimes the enemy would transform, I would kill it, and then it would simply reappear. Adding the conditional branch here fixes that.
If you want any special effects, go ahead and add a Show Battle Animation inside the conditional branch. Next add the event command Enemy Transform. Then add Enemy Recover All and select your transforming enemy. This will restore all HP, SP, and remove any status effects.
We have to add another scripting event command. After we transform the enemy, it's still set to Immortal. Add a new event command and select Script so that we can turn off the Immortal flag. For the script, enter this:
$game_troop.enemies[X].immortal = false
Again, replace X with the appropriate number.
That should be it! The battle event should like something like this: