The paper, straw, tape, and paper clip can be sourced at most office supply stores. The copper tape, LED, and coin battery can be found at the following places online:
A local craft store might stock copper tape. Just make sure that it uses conductive adhesive.
A local hobby electronics store should stock batteries and LEDs. This activity uses CR2032 coin cell batteries and 3mm LEDs.