First check the SMTP server is installed in the server. You can check that on Server Manger and Features section.
If it is not installed first, you need to install the SMTP in the server. SMTP server needs port 25 opens in the firewall in default.
After setting up the SMTP server, you need to open the IIS Manger.
Before stating the SMTP Virtual server needs to configure some properties. Right click the SMTP virtual Server and click on the Access Tab. Click on Relay.. button on the Relay restrictions.
This section specified which server can use the SMTP for mail relay. So either you need to select Only the list below and add IP Addresses of all server which used to send the emails (ex:-WFE, APP) or let all servers to send emails by All except the list below and putting nothing on the list.
Then click on Authentication... button on Access Control section and make the checkbox for Anonymous access.
Then click ok and go to the Delivery Tab.
Outbound Security section allows you to provide credential for receiving server. Since we are using local SMTP server, you can make this as a Anonymous Access.
Outbound Connection section you can change the default port using.
Then start the SMTP virtual server.
After that, you can find the fully qualified name in General Tab and proceed with the Outbound SMTP settings as Part 1.