Gift taxes are only assessed on gifts given above a certain dollar amount (the "exclusion" amount), per recipient, per year, that total more than the exemption amount. If you give one person more than the exemption amount during the tax year, you must report the gift to the IRS on the IRS Form 709. You are required by law to report the gift, and if you don't, it could come out in an audit. This is how the IRS determines whether you owe gift tax.
The amount you can gift to one person during one tax year is called your exclusion amount. You can give any one person money up to the exclusion amount, and you will not have to report it. If you give one person more than the exclusion amount during the tax year, you must complete the Form 709 and report it to the IRS, but you will not be taxed at that time. Throughout your life, you can continue to gift up to the exclusion amount every year, and you must report the amount you go over the exclusion every year.