Coding Horror

programming and human factors

Net.WebClient and GZip

The Net.WebClient class doesn't support HTTP compression, eg, when you add the Accept-Encoding: gzip,deflate header to your request:

  
Dim wc As New Net.WebClient  
'-- google will not gzip the content if the User-Agent header is missing!
wc.Headers.Add("User-Agent", strHttpUserAgent)  
wc.Headers.Add("Accept-Encoding", "gzip,deflate")  
'-- download the target URL into a byte array
Dim b() As Byte = wc.DownloadData(strUrl)  

What you get is a gzipped array of bytes. It's pretty easy to add the missing gzip support, though. First, download the SharpZipLib and add a reference to ICSharpCode.SharpZipLib to your project. Then it's only a few more lines of code..

  
Dim gz As New GZip.GZipInputStream(New MemoryStream(b))  
Dim intSizeRead As Integer  
Dim unzipBytes(intChunkSize) As Byte  
Dim OutputStream As New MemoryStream  
While True  
'-- this decompresses a chunk
'-- remember the output will be larger than the input (one would hope)
intSizeRead = gz.Read(unzipBytes, 0, intChunkSize)  
If intSizeRead > 0 Then  
OutputStream.Write(unzipBytes, 0, intSizeRead)  
Else  
Exit While  
End If  
End While  
'-- convert our decompressed bytestream into a UTF-8 string
Return System.Text.Encoding.UTF8.GetString(OutputStream.ToArray)  

And voila, the bandwidth, you have saved eet! How do I know this actually works? Using my network sniffer of course..

Written by Jeff Atwood

Indoor enthusiast. Co-founder of Stack Exchange and Discourse. Disclaimer: I have no idea what I'm talking about. Find me here: http://twitter.com/codinghorror