A similar question was asked before, it might be useful to look at the answers:
How do I setup make for Sqlserver 2014 with install shield?[
^]
As I don't use Installshield but the free Inno Setup installer, I can't help you with Installshield, but can tell that installing SQL Server is a complex matter with lots of pitfalls:
Usually I check first if SQL Server is already installed.
If not, check if a 32 or 64 bits version must be installed.
Run the SQL Server (Express) installer from a temp directory with the correct parameters.
If you want to keep things simple, you can use Windows Authentication (no username and password asked).
Install your application and let it run an SQL script to create the database.
Working with .mdf files directly is asking for trouble.
Here is a simple Inno Setup script:
Inno Setup Script to Install SQL Server 2008 R2 with Tools[
^]
The best advice I can offer however is: do not use SQL Server !
I got so fed up with the complications and bloated size of SQL Server that I decided to use
PostgreSQL (with Npgsql in C#) instead.
Here is a conversion program to convert SQL Server databases to PostgreSQL:
Convert SQL Server Database to PostgreSQL[
^]