Can they use my silence against me?

The timing distinction here is real and counterintuitive.

After warnings and an invocation, using your silence against you is generally prohibited. The government can't tell you that you have a right and then argue that exercising it shows guilt.

Before any warnings, in a non-custodial setting, the picture is different. The Supreme Court has held that a person who simply stays quiet without expressly invoking the privilege may have that silence used against them.

That's the practical argument for saying the words rather than just going quiet. A person who declines to answer without invoking anything may be in a worse position than one who states clearly that they're asserting the right.

Related: selective answering creates a problem of its own. Answering ten questions and then going quiet at the eleventh draws attention to the eleventh, and prosecutors do argue from that pattern.