Just like real life, I practice avoidance so the question becomes: Which is the most difficult to avoid, even with Speed Buffs?
Magma Sparker
Honorable Mention to their little siblings the Magma Sprites but they don't have the debuff and are quicker to kill.
In early game the Shadow people in the dark levels of the Mines creep me out and me do NOT like.
I have trouble with the Flying Serpents but I can always just toss down more Stairs to evade them. And everything else in Skull Cavern.
Slimes are a pain but the Slime Charmer Ring completely eliminates them.
Lava Lurks are annoying but pretty easily avoided as their fireballs are slow.
The Spiders and Mushroom thingies in the Dangerous Mines are very annoying with their high speed boinging but again, Staircases. Frankly, the same goes for
everything on the lower Levels of the Dangerous Mines, but always the solution is Staircases.
Avoidance is key.