Free Online Courses for Software Developers - MrBool
× Please, log in to give us a feedback. Click here to login
×

You must be logged to download. Click here to login

×

MrBool is totally free and you can help us to help the Developers Community around the world

Yes, I'd like to help the MrBool and the Developers Community before download

No, I'd like to download without make the donation

×

MrBool is totally free and you can help us to help the Developers Community around the world

Yes, I'd like to help the MrBool and the Developers Community before download

No, I'd like to download without make the donation

How to Retrieve Web Pages with HTTP in ASP.NET

This tutorial helped the developers to find an easy way to request pages over the internet making use of the HttpWebRequest and HttpWebResponse classes.

This is done in order to obtain the web page and comprises of a GetResponse() method for obtaining a HttpWebResponse object. With the help of a HttpWebResponse object, we fetch a stream to read bytes from. Iterating until all the bytes of a Web page are read, translating bytes to strings, and holding the string, makes it possible to obtain the entire Web page.

Introduction

The basic transport means in order to communicate with resources over the internet is HTTP. As far as developer is concerned, he/she will always require to fetch or retrieve the web pages for several reasons that comprises of:

  • searching engine page caching
  • obtaining info on a particular page
  • Implementing browser-like capabilities.

In order to get assistance on this task, we have .NET Framework that comprises of classes and it makes the job look very simple.

Retrieve a HTTP Page

Talking about the HTTP classes in the .NET framework, we have HTTPWebRequest and HTTPWebResponse.

Let us look at the steps involved to accomplish the task.

  • The first step is to require specifying a web page to get with a HTTPWebRequest object,
  • Next would be to perform the actual request and making use of an HTTPWebResponse object in order to receive the page.
  • Finally, one would be making use of the stream operations to fetch the details of the page.

The below listing illustrates the process to accomplish the task.

Listing 1: Retrieving a Web Page: WebFetch.cs

using System;
using System.IO;
using System.Net;
using System.Text;


/// <summary>
/// Retrieves a Web Page
/// </summary>
class WebFetch
{
	static void Main(string[] args)
	{
		// utilizes to create the whole input
		StringBuilder sb  = new StringBuilder();

		// used on each read operation
		byte[]        buf = new byte[8192];

		// prepare the web page we will be asking for
		HttpWebRequest  request  = (HttpWebRequest)
			WebRequest.Create("http://www.mrbool.com");

		// run the request
		HttpWebResponse response = (HttpWebResponse)
			request.GetResponse();

		// data will be read with the help of the response stream
		Stream resStream = response.GetResponseStream();

		string tempString = null;
		int    count      = 0;

		do
		{
			// fill the buffer with data
			count = resStream.Read(buf, 0, buf.Length);

			// make certain we read some data
			if (count != 0)
			{
				// translate from bytes to ASCII text
				tempString = Encoding.ASCII.GetString(buf, 0, count);

				// continue building the string
				sb.Append(tempString);
			}
		}
		while (count > 0); // any more data to read?

		// print out page source
		Console.WriteLine(sb.ToString());
	}
}

Let us see what the program listed above will do. It will actually request the main page of a web site. Hereafter it will exhibit the HTML on the console screen. For the reason that the page data will be returned in bytes, we will be setting a byte array by the name buf, to hold results.

The very first step that needs to be done in getting a web page is to instantiate a HttpWebRequest object. When does this occur? This actually occurs at the time of invoking the static Create() method of theWebRequest class.

The parameter to the Create() method is a string that reflects the URL of the web page you desire. The Create() method returns a WebRequest type hence we require to cast it to anHttpWebRequest type prior to assigning it to the request variable. The below lists the code to create the request object:

Listing 2: Creating the request object

// prepare the web page we will be asking for
		HttpWebRequest  request  = (HttpWebRequest)
			WebRequest.Create("http://www.mrbool.com");

After attaining the request object,make use of the same to obtain a response object. The response object is created by using the GetResponse() method of therequest object that we just created. The below code reflects the way to obtain theHttpWebResponse object.

Listing 3: Obtaining theHttpWebResponse object

// execute the request
		HttpWebResponse response = (HttpWebResponse)
			request.GetResponse();

The response object is utilized to fetch a Stream object that is a member of the System.IO namespace. The GetResponseStream() method of the response instance is invoked to obtain this stream as follows:

Listing 4: Obtaining Stream

// we will read data via the response stream
		Stream resStream = response.GetResponseStream();

We will now make use of the byte array in the Read() method, of the stream we just retrieved in order to fetch the web page data.

The Read() method accepts three arguments:

  • The first is the byte array to populate
  • The second is the beginning position to begin populating the array
  • The third is the maximum number of bytes to read.

This method gives back the actual number of bytes that were read. The web page is read as follows:

Listing 5: Web page Ready

// fill the buffer with data
		count = resStream.Read(buf, 0, buf.Length);

Now we have an array of bytes that contains the web page data in it. But it is a good technique to transform these bytes into a string. Using this, we can make use of all the built-in string manipulation methods that are available with .NET. We have used the static ASCII class of the Encoding class in theSystem.Text namespace in order to accomplish this task.

The ASCII class has a GetString() method that accepts three arguments and are similar to the Read() method.

  • The first parameter is the byte array to read bytes from, which we pass buf to.
  • Second is the beginning position in buf to begin reading.
  • Third is the number of bytes in buf to read.

We have passed count here that was the number of bytes given back by the Read() method, as the third parameter. This ensures that only the required number of bytes were read. The below lists the code that translate bytes in buf to a string and appends the results to a StringBuilder object.

Listing 6: StringBuilder Object

// translate from bytes to ASCII text
		tempString = Encoding.ASCII.GetString(buf, 0, count);

		// continue building the string
		sb.Append(tempString);

This is all for today's article, hope you learnt how to Retrieve Web Pages with HTTP and see you next time.



Software developer with more than 5 years of development on Java, HTML, CSS.

What did you think of this post?
Services
[Close]
To have full access to this post (or download the associated files) you must have MrBool Credits.

  See the prices for this post in Mr.Bool Credits System below:

Individually – in this case the price for this post is US$ 0,00 (Buy it now)
in this case you will buy only this video by paying the full price with no discount.

Package of 10 credits - in this case the price for this post is US$ 0,00
This subscription is ideal if you want to download few videos. In this plan you will receive a discount of 50% in each video. Subscribe for this package!

Package of 50 credits – in this case the price for this post is US$ 0,00
This subscription is ideal if you want to download several videos. In this plan you will receive a discount of 83% in each video. Subscribe for this package!


> More info about MrBool Credits
[Close]
You must be logged to download.

Click here to login