One of the best sources for inexpensive, used hobby supplies is Craigslist. Really, Craigslist is the best source for just about anything used, because it is one of the first places people look when they are shopping for a bargain, and sellers know this.
Thrift stores are also a great place to buy used supplies, from woodworking materials to craft supplies. I like to get sewing patterns and yarn at the thrift store, because they are often in great shape and cost pennies sometimes.
If you have the time and the inclination, yard sales are probably the least expensive way to get outfitted for a hobby, but it takes a lot of time and dedication to find the good deals. Yard sale shopping can be a fun pastime though, so if you enjoy that sort of thing, you can save money on all kinds of things – not just hobby supplies.
Buying used hobby supplies makes sense for almost every hobby, and at least some of your supplies can be gained this way for pennies on the dollar.