One of my customer came with a problem of speed for varnish.

Long debug, short :
When varnish get an object from his cache (memory), it's really sluggish (> 5 seconds),
when varnish need to get the object from the apache backend, no speed problem (< 1 second).

Exemple of a slow request (from varnishlog) :

193 ReqStart     c <client ip> 59490 1329239608  
193 RxRequest    c GET  
193 RxURL        c /<my_url>/toto.png  
193 RxProtocol   c HTTP/1.1  
193 RxHeader     c Accept: */*  
193 RxHeader     c Referer: <my_referer>  
193 RxHeader     c Accept-Language: fr  
193 RxHeader     c User-Agent: <client_useragent>  
193 RxHeader     c Accept-Encoding: gzip, deflate  
193 RxHeader     c Host: <my_vhost>  
193 RxHeader     c Connection: Keep-Alive  
193 VCL_call     c recv lookup  
193 VCL_call     c hash  
193 Hash         c /<my_url>/toto.png  
193 Hash         c <my_vhost>  
193 VCL_return   c hash  
193 Hit          c 1329136358  
193 VCL_call     c hit deliver  
193 VCL_call     c deliver deliver  
193 TxProtocol   c HTTP/1.1  
193 TxStatus     c 200  
193 TxResponse   c OK  
193 TxHeader     c Server: Apache  
193 TxHeader     c Last-Modified: Mon, 18 Jun 2012 08:57:46 GMT  
193 TxHeader     c ETag: "c330-4c2bb5c0ef680"  
193 TxHeader     c Cache-Control: max-age=1200  
193 TxHeader     c Content-Type: image/png  
193 TxHeader     c Content-Length: 49968  
193 TxHeader     c Accept-Ranges: bytes  
193 TxHeader     c Date: Tue, 16 Oct 2012 06:54:03 GMT  
193 TxHeader     c X-Varnish: 1329239608 1329136358  
193 TxHeader     c Age: 391  
193 TxHeader     c Via: 1.1 varnish  
193 TxHeader     c Connection: keep-alive  
193 TxHeader     c X-Cache: HIT  
193 TxHeader     c X-Cache-Hits: 210  
193 Length       c 49968  
193 ReqEnd       c 1329239608 1350370443.778280735 1350370480.921206713 0.372072458 0.000045538 37.142880440   

If I'm true, the problem is on th

Edit
Report