Save displayed text from Web Page in C# Pin

Question

0.00/5 (No votes)

See more:

Hi
I need save to file User Names from this web: http://worldoftanks.eu/community/clans/500027296-BERU/#wot&mt_order_by=-name

I have tried this with WebClient, WebBrowser, SHDocVw.WebBrowser etc. and nothing work because in downloaded html page code has not filled that informations, what I need. As I undersrand they are showing via AJAX or somethig like this.

WebBrowser has method ShowSaveAsDialog, but I need WITHOUT this dialog

Could someone help me?

P.S. Sorry for my bad english.

Posted 10-Nov-13 1:31am

triXerr

Add a Solution

Comments

ridoy 10-Nov-13 7:33am

trigger an event that could perform like Ctrl+C and Ctrl+V.

OriginalGriff 10-Nov-13 7:41am

Don't post the same question in multiple places:
http://www.codeproject.com/Messages/4698227/Save-displayed-text-from-Web-Page-in-Csharp.aspx
It's rude and it wastes effort.
Bear in mind it is a Sunday, and most of the western world is not at work. Have a little patience!

Abhinav S 10-Nov-13 7:55am

Repost from the forums section.

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Sergey Alexandrovich Kryukov · Answer 1 · 2013-11-10T06:23:00

Solution 1

It does not matter if they use Ajax or something else. If the information you need is ever provided and can be seen in any other way, you can do the same in your code.

Or, maybe, you cannot, just because of one your problem: you are using try-and-error approach instead of trying to understand how things works. Such approach is utterly inefficient. I can see it by "WebBrowser" and "SHDocVw.WebBrowser" — browser components can be used, but are irrelevant to the problem. Using WebClient would make perfect sense, but the more universal tool is the class System.Net.HttpWebRequest:
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest%28v=vs.110%29.aspx[^].

This is all you may possibly need. See also my past answers:
How to get the data from another site[^],
get specific data from web page[^].

[EDIT]

I found that the problem is somewhat more difficult. Please see my comment below, about possible Ajax use.

Here is some very different advice: find out and install some plug-in which works as the "HTTP spy". I, for example, use a Mozilla browser (Seamonkey, or you can use Firefox), and my plug-in is called HttpFox. Enable and start it. Then load you page. Wait until the list you need is show. Stop the plug-in. See all the requests and resources downloaded. Using HTTP requests provided, download them all. Find out in which one your data resides.

I just tried it, it shows a good hundred of downloaded items for just one page you referenced. Loot at them, find out where your data is.

Do the same thing in on step, using the code you already have. I will be a considerable shortcut in your work.

—SA

Posted 10-Nov-13 6:23am

Sergey Alexandrovich Kryukov

Updated 11-Nov-13 7:11am

v3

Comments

triXerr 10-Nov-13 12:54pm

Via WebClient or HttpWebRequest I cant download necessary data. Because in downloaded string are not that data (User Names) from Web Page :(

Sergey Alexandrovich Kryukov 10-Nov-13 12:58pm

This is a matter of particular Web page. Is this data accessible if you do it throw a Web browser? IF it is, you can do it in code. If not, it is not available to you. As simple as that.
—SA

triXerr 10-Nov-13 13:15pm

Yes, When I will Navigate that page in WebBrowser, then I can see that data.

Sergey Alexandrovich Kryukov 10-Nov-13 13:56pm

Great. It means that you should do exactly the same sequence of requests programmatically. Do you have to authenticate? Then you need to do it, too.

As I can see, the URL you provided does not show anything, you also need to click on something. When you click, this is another URL, you should request it. Or it can be some JavaScript, at it will eventually redirect you to where you should find the information you need. Then you should look at this JavaScript and see what it does. And so on...

—SA

triXerr 10-Nov-13 14:19pm

If it is correct... filling that table wich contains User Name, etc. doing this script: http://worldoftanks.eu/static/3.15.0.4/clans/js/helper_members.js

But I'm not able to analyze it, because I never work with JavaScript or whatever is it.

Sergey Alexandrovich Kryukov 10-Nov-13 14:31pm

Nobody will do it for you. Depending on your goals, you will have to analyze it to some extent. Or you have to get back to a browser which handles JavaScript inside the browser control and gives you the redirection you would need to save...
—SA

triXerr 10-Nov-13 15:35pm

So, if I understood correctly, it is possible save that data somehow via WebBrowser in C#?
there is a method ShowSaveAsDialog, that is doing exactly what I need, but I cant save it without that dialog. Is there some way to bypass and save it directly ?

Sergey Alexandrovich Kryukov 10-Nov-13 15:59pm

Yes, but this might be not a good business. And you don't want SaveAs dialog. Do you see the contradiction? Then you need to use some other event anyway, probably on the control which is outside the browser control. How it can be better?.. Or you need to use some preexisting knowledge of what should be on that site and save the page when the page with appropriate content is reached. Does not look nice anyway. The whole problem looks questionable. Perhaps you can explain why doing it all...
—SA

triXerr 10-Nov-13 16:45pm

Okay. I will try to explain.
I want create application which will be checking every hour teamsepeak 3 server (that I have already completed) and compare it with Users, which are true members of clan. These Users are displayed on Web Page. I do not know how if there easier way for that

Sergey Alexandrovich Kryukov 11-Nov-13 0:45am

I just loaded the page and immediately saw the table "User Name - Position - Joined", 43 rows of data. Is that the data you need it's all there. I saved the page and found all those names there. Is that what you wanted to see? So, you can do it programmatically and extract those names from the HTML file.
—SA

triXerr 11-Nov-13 5:44am

That is exactly what I want, but maybe I dont understand... How I could extract that names from HTML file, when in saved HTML file, table are not filled ?

Sergey Alexandrovich Kryukov 11-Nov-13 10:28am

I don't see a problem. I simply saved the file from your URL. How to extract? You would need some HTML parser. Ideal situation if the page is presented in HTML which is well-formed XML. If not, you would need a parser which could work with "dirty" HTML.

Please see the last of my past answers referenced above. Last link. On that answer, last paragraph and last link.

—SA

triXerr 11-Nov-13 11:23am

This is my code:
WebBrowser wb = new WebBrowser();
wb.Navigate("http://worldoftanks.eu/community/clans/500027296-BERU/");
wb.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(wb_DocumentCompleted);

private void wb_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
WebBrowser wb = (WebBrowser)sender;
StreamWriter sw = new StreamWriter("Page.html");
sw.Write(wb.DocumentText);
sw.Flush();
sw.Close();
}
And in saved Page.html, are not filled data what I want. - that is a problem.

Sergey Alexandrovich Kryukov 11-Nov-13 11:37am

Why? I just tested that downloading a file will immediately give you the content you need. Use HttpWebRequest.
—SA

triXerr 11-Nov-13 12:41pm

There is code which I using HttpWebRequest
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://worldoftanks.eu/community/clans/500027296-BERU/");
request.Method = "GET";
request.KeepAlive = true;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();

Stream stream = response.GetResponseStream();
var encoding = new System.Text.UTF8Encoding();
StreamReader sr = new StreamReader(stream, encoding, true);
StreamWriter writer = new StreamWriter("Page.html");
writer.Write(sr.ReadToEnd());
sr.Close();
writer.Close();

And still data are not in saved Page.html
I do not know what I am doing wrong

Sergey Alexandrovich Kryukov 11-Nov-13 12:51pm

From the first glance, it looks correct. So, what do you get in Page.html.

I just found that the problem is different, please see below...

—SA

Sergey Alexandrovich Kryukov 11-Nov-13 12:57pm

Okay, I tried it. Probably your assumption about Ajax was correct. You need to see what happens on page load. Maybe some script works after the page is already load, load additional data via Ajax and shows it dynamically. Pretty usual technique. You need to find out the URL used by Ajax and use it, too.

You need to recursively "bootstrap" the technique. Download all JavaScripts used by the page. Find out what they do. Find out where the URL providing user list is. It could be somewhere under http://worldoftanks.eu//ajax_info/... (please see the first script).

Look, this is a good amount of work you should do.

—SA

Sergey Alexandrovich Kryukov 11-Nov-13 13:12pm

Please see my updated answer, after [EDIT].
—SA

triXerr 11-Nov-13 14:06pm

I thing I found nescessary sript. It is probably this: http://worldoftanks.eu/static/3.15.0.4/clans/js/helper_members.js
So now I dont know how to call this sript for recive data.

Sergey Alexandrovich Kryukov 11-Nov-13 15:05pm

Great. Will you finally accept my answer formally then?
—SA

triXerr 12-Nov-13 10:41am

Looks like I am not able to complete it. I dont know how to work with JS in C#, call JS and download that data.
Anyway Thank you for help.

Sergey Alexandrovich Kryukov 12-Nov-13 11:06am

All right, did you try my technique with HTTP spy? I think this is the most promising. You can do the following: obtain the list of all resources downloaded (probably you can copy and paste all URLs), try to download each one. Before downloading, remember a couple of user names you see on the page and after downloading, try to find them in the downloaded files using text search.
—SA