What is a smoking gun? Where does the phrase come from?

A smoking gun is a weapon that has recently been fired. It is used metaphorically to describe incontrovertible incriminating evidence. 
The phrase was first used in the Sherlock Holmes story, The Gloria Scott (1893). 

We rushed into the captain's cabin . . . there he lay while the chaplain stood with a smoking pistol in his hand.

William Safire the identifies the first contemporary use as during the Watergate scandal in 1974.