Download a Web Page using the WebClient

Introduction

In the past you might have used the Microsoft Internet Controls (SHDOCVW.dll) and the Microsoft Internet Transfer Control 6.0 (SP4) in VB6 applications. The WebBrowser Control (SHDOCVW.dll) gives you the ability to display a web page in a Visual Basic Application. The Inet Control (Internet Transfer Control) gives you the ability to download and upload files to URLs. Please review the tutorial on the WebBrowser control used for downloading web pages. Given the URL, the two controls together can show the web page and its source at the same time.

This basic tutorial shows how the same functionality is now available to .NET framework users by calling the methods and properties of the WebClient class, which belongs to the System.NET Namespace. In order to use this class, the Microsoft Visual Basic 2005 Express Edition (also Visual Studio 2005) can be used. The program can be downloaded from the Microsoft website. This beginners oriented tutorial also shows how to use the WebClient Control as well.

You can find the WebClient in the .NET Framework Components page in the Choose Toolbox Items pop-up when you access it using the Tools-->Choose Toolbox Items... hyperlink. You can also find the WebBrowser Control in the same folder. You can also access these controls in the same exact manner in Visual Studio 2005, and the Visual Web Developer 2005 Express Edition.

Creating a Windows project to use the WebClient Class

Create a Windows Application project by accessing the New Project from the File menu item. This opens the New project window. Choose the Windows Application as shown and change the default WindowsApplication1 name to something different. In the present case it is called ExploreWebClient.

Change the default Form1.vb to WebPage.vb. Using the WebClient class we will download a web page, in this case, a blog from the site http://hodentek.blogspot.com/. This is a blog that requires no authentication; you may use any other web page as well. To the WebPage.vb form add a button, a textbox, and a WebBrowser control. If the WebBrowser control is not already in the Toolbox, you may add the control by going to Toolbox--> Choose Toolbox items...

and click the hyperlink to open the Choose ToolBox Items Window as shown.

This adds the control to the Toolbox and appears immediately when you click OK to the above after placing a check mark against the item.

With these controls added, your page should appear as shown in the next picture. The area above the button is the textbox which has been set in design to show multi-lines and to show both scroll bars. It does not show the horizontal scroll bar for reasons which are not clear. The clear area below the button is the resized WebBrowser control.

To the click event of the button add the following code as shown:

Imports System.Net

Imports System.IO

Public Class WebPage

Dim wclient As New WebClient

    Private Sub Button1_Click(ByVal sender As System.Object, _

    ByVal e As System.EventArgs) Handles Button1.Click

        Dim sr As New StreamReader(wclient. _

        OpenRead("http://hodentek.blogspot.com/"))

        Dim s As String = sr.ReadToEnd

        TextBox1.Text = s

        WebBrowser1.Navigate("http://hodentek.blogspot.com/")

    End Sub

End Class

The Imports System.NET allows us to use the WebClient class and spawn a new instance by calling the New method. The newly created instance can call one of its methods, OpenRead() to read the contents of the URL.

This gets directly read into the Stream by the StreamReader, and all that is read is sent to the textbox. Since you are using the StreamReader, you need the second imports statement, System.IO.

The WebBrowser1 ActiveX control is straightforward to code. Given an URL, it displays the web page.

Know your options

Visual Basic 2005 Express Edition has all the nice features of the full Visual Studio 2005; that includes intellisense. You should make use of this feature to code so that you will understand the various possibilities as shown in the next picture. As you can see, the number of overloaded options you have, and in fact most of what you can code, is spelled out and laid out neatly.

Click on Project and from the drop-down click on the ExploreWebClient Properties... hyperlink to open the following window, where you can set the project to run the page you choose from among other forms in this application, as shown by the drop-down, Startup Object.

When you run the form, you will see the form displayed as shown. The textbox shows the source of the blog and the one below it shows the web page.

View from the Object Browser

The Object Browser provides access to the tree structure of the various Namespaces and classes used in the project. The next picture shows a part of the class WebClient which has a large number of methods and some properties. The OpenRead() method is highlighted.

Create a form using the WebClient Control

Another way of opening an URL to read its contents is to use the WebClient Control, which may be added to the Toolbox using the same procedure used for adding the WebBrowser control. Add a new form, WebClientControl.vb, to the project. From the Toolbox double click the WebClient control when the form is in the design view. Add a command button and a textbox whose multi-line property is set to true, with both scroll bars showing. The WebClient control gets to the component tray area as shown.

To the click event of the command button add code as shown:

Imports System.IO

Public Class WebClientControl

    Private Sub Button1_Click(ByVal sender As _

    System.Object, ByVal e As System.EventArgs) _

    Handles Button1.Click

        Dim wc As New System.Net.WebClient

        Dim strm As New StreamReader _

        (wc.OpenRead("http://localhost/TestXMLHttp/TestClient.htm"))

        Dim s As String = strm.ReadToEnd

        TextBox1.Text = s

    End Sub

End Class

This time the OpenRead() method reads the TestClient.htm file on the local intranet. You are not referencing the class directly by the imports statement, rather you are instantiating a New WebClient. Otherwise using the StreamReader is similar to the previous case. When you run this form, your display appears as shown.

Summary

WebClient Class and the WebClient Control are very useful for interacting with web sites. The many properties and methods can be put to good use for importing any part of a web site. By creating a COM wrapper the controls can be called from VB 6. However the Inet and WebBrowser controls can continue to be used in VB6 programs. What is achieved using several lines of code in the Visual Basic 2005 Express Edition program can be achieved by two lines in a VB6 project, however one is object-oriented and the other is object-based. Whether Microsoft will continue to support VB 6 or not is another point to consider.