The Net.WebClient class doesn't support HTTP compression, eg, when you add the Accept-Encoding: gzip,deflate header to your request:
Dim wc As New Net.WebClient '-- google will not gzip the content if the User-Agent header is missing! wc.Headers.Add("User-Agent", strHttpUserAgent) wc.Headers.Add("Accept-Encoding", "gzip,deflate") '-- download the target URL into a byte array Dim b() As Byte = wc.DownloadData(strUrl)
What you get is a gzipped array of bytes. It's pretty easy to add the missing gzip support, though. First, download the SharpZipLib and add a reference to ICSharpCode.SharpZipLib to your project. Then it's only a few more lines of code..
Dim gz As New GZip.GZipInputStream(New MemoryStream(b)) Dim intSizeRead As Integer Dim unzipBytes(intChunkSize) As Byte Dim OutputStream As New MemoryStream While True '-- this decompresses a chunk '-- remember the output will be larger than the input (one would hope) intSizeRead = gz.Read(unzipBytes, 0, intChunkSize) If intSizeRead > 0 Then OutputStream.Write(unzipBytes, 0, intSizeRead) Else Exit While End If End While '-- convert our decompressed bytestream into a UTF-8 string Return System.Text.Encoding.UTF8.GetString(OutputStream.ToArray)
And voila, the bandwidth, you have saved eet! How do I know this actually works? Using my network sniffer of course..