You sign with your private key. You’re using the public key as an identifier to tell your git client which key to use, since git can use the public key to confirm it’s using the right private key.