Virtual host scalability

[zellster]

Hello,

I am in the position where I need to support 1-2 million virtual hosts. The obvious way to handle this in lsws (or Apache) I am guessing, is to throw everything into one huge XML config file. Given that approach, what type of algorithmic lookup behavior (O(n), O(1), etc.) is used internally for Host: --> vhost mapping? At what point can I expect the vhost mapping to not scale?

Any suggestions appreciated.

 

[mistwang]

It depends on how the mapping is configured, for explicit domain name matching, hash map is used in LSWS, so it should be O(1), for wild card domain name (with * or ?) matching, regex is used and each pattern has to be processed one by one, so it is O(n).

Scalability of virtual mapping is not my biggest concern to those kind of mass hosting configuration, but memory usage is. As each virtual host data structure will take certain amount of memory to store parsed configurations. Assume each one takes 1KB memory, you will be using 1-2GB memory just for vhost configurations. We can add a new class of virtual host which uses minimum memory for each virtual host.

However, there is one solution for LSWS (and Apache) right now, to use URL rewrite for mass hosting, only a domain name to document root mapping is required, it is fast (o(1) in lsws), memory efficient but lacking the flexibilities in configuration comparing to standalone virtual host.

To host 1-2 million virtual hosts on one server is not a easy job no matter what server software is used, I think. LSWS has not added any special feature for that kind of applications yet, but we can if there are a lot of interest in that.

George

 

[zellster]

Hello,

Thank you for the reply. From some quick web searches, it seems that using URL rewrites for mass virtual hosting is being phased out of Apache in favor of mod_vhost_alias, which looks to be fairly flexible in its hashing format.

Does LSWS support the same notation:
http://httpd.apache.org/docs/mod/mod_vhost_alias.html

Thanks.

 

[mistwang]

Yes, we can add somthing similar. Our next goal is to make litespeed be able to run off apache's configuration with same functionality.

 

[zellster]

Hello,

Let me add one more wrinkle, and see if it's doable in LSWS without a lot of pain. As mentioned above, I would like to support a large number of virtual hosts of the form '*.foo.com'.

Ex: bar.foo.com, baz.foo.com, goo.foo.com, etc.

If the url is for static content, I would like LSWS to serve the content directly. Otherwise, I would like LSWS to forward the request to a Java back-end for processing via AJP.

It would be easier to host static content directly on another domain (img.foo.com), but that may not be possible short-term. If I stick to the same consistent layout in the vhosts for static content (*.foo.com/img), would the rewrite module be a complete solution?

Any thoughts appreciated...

 

[mistwang]

Acutally, it is our current AJP implementation, LSWS serves the static contents and forwards request to dynamic content to backend servlet engine. There are two way to define dynamic content, one is via script handler (for jsp files), another one is via servlet context (for servlets).
It should work well with URL rewrite.
You just need to make sure the servlet engine scale well.