Gold is one of the least reactive metals, sitting at the very bottom of the reactivity series. Because of this extremely low reactivity, gold does not combine with other elements under natural conditions. It therefore occurs in the earth's crust as the free (uncombined) metal, often found as nuggets or flakes in alluvial deposits.
Since gold already exists in its elemental form, there is no need for chemical extraction from an ore. Metals higher in the reactivity series (such as iron, aluminium, or sodium) form stable compounds with oxygen, sulphur, or other elements and must be chemically reduced to obtain the pure metal. Gold does not form such compounds naturally, so it is simply recovered by physical methods like panning or washing.
The phrase free in nature specifically means the metal is found uncombined. While gold is indeed unreactive (inert), the reason it does not require extraction is that it occurs free in nature, which is the more precise and direct answer to the question. Being "inert" describes a property; being "free in nature" describes the consequence of that property that directly answers why extraction is unnecessary.