There are two sets of keys shared between IDP and SP. SP share their Public Key to IDP and IDP also share their Public Key. The SSO Flow happens like this
SP (Service Provider - Our Application Signs the SAMLRequest using SP Private Key). SAML Request is never Encrypted.
IDP validates the signature using SP Public Key. At IDP it needs to be configured to use this signing key.
IDP then encrypt the Response using SP Public Key and Sign it using IDP Private Key. Encryption needs to be configured at ADFS side.
SP validates the signature using IDP Public Key and decrypt it using SP Private Key