Kumar, why do you write code to open a popup window in business logic layer.
Purpose of Business Logic Layer is different. Be clear with what is 3-tier/n-tier architecture is about. At the end of it all, you must be able to appreciate why a tier (or Layer) is required and what each one does specifically.
Coming to you question, I assume that you are required to open modal/nonmodal popup window. Since this is purely a UI thing, you will not be required to go to Business logic layer and open it from there. You can very well do that in UI itself.
In Main.aspx.cs, in the button click event write some thing like below.
void Button_Click(object sender, EventArgs e)
{
OpenNewWindow("Sub.aspx");
}
public void OpenNewWindow(string url)
{
ClientScript.RegisterStartupScript(this.GetType(), "SubWindow", String.Format ("<script>window.open('{0}');</script>", url));
}
This shoould do for you. Else, try out for different ways of opening a modal dialog in asp.net.