Some inventory items (including your original Drill I and Slugpistol, and any Battery) are "cursed" -- you cannot discard them (normally done by bringing up your inventory and dragging the unwanted item anywhere outside the inventory boxes).
This might be a game safeguard to make sure you don't lose these important items, but if you make a better drill or pistol you probably don't want the old ones.
However, you can get rid of your old pistol when you make a new pistol. Use the Assembler to create a new pistol, and drag the newly-created pistol onto the old pistol. Your old pistol will be gone, replaced by the new one. The same technique works on drills. The new items are not "cursed" and can be thrown away.
If also want a second drill as a "backup", and if you have enough resources, Assemble another one and drag the second drill into a different empty spot in your inventory. Two identical drills cannot be combined (the way you normally would do with materials), but you can throw unwanted ones away.