Silo removal?

You go back to Robin and open the panel to pick farm buildings (like you did to build the silo). On the bottom are several icons that tell you what they do when you hover the cursor over them. One allows you to destroy any building and another allows you to move them (the space they're moving to needs to be cleared of everything but grass).
 
Top