I do the same request twice - One with https, one with simple http.
The https-Request requires much more time than the http-Request - I think it is about 4 times longer for https-Request.
And I'm getting no X-LiteSpeed-Cache:hit header flag.
Which application do you use? Have you installed proper cache cache plugin? Have you enable cache lookup in .htaccess?
Basiclly check your cache enabling process to see if there is any step missed.