nginx serving large static files

General Discussion of atomic repo and development projects.

Ask for help here with anything else not covered by other forums.
premierhosting
Forum Regular
Forum Regular
Posts: 257
Joined: Wed Aug 04, 2010 2:52 pm

nginx serving large static files

Unread post by premierhosting »

Howdy,

Anyone using nginx? I'm serving some 10-30mb files at peak times up to 250-300 requests per second, if I'm reading these RDD'd /status logs correctly. These are on VPS servers, which may mean some network settings aren't available.

It mostly works, but sometimes the massive simultaneous connections overwhelms the system, and I'm wondering if anyone here is doing something like that and has suggestions for nginx.conf or sysctl.conf.

Most info out there is on using nginx as a proxy, and tuning the proxy settings. I'm not using it as a proxy, I'm using nginx as a web server.

Turning sendfile on seemed to help dramatically. Setting sendfile_max_chunk to 1m helped too. Right now I'm looking at the TCP settings like tcp_fin_timeout to see if optimizations can be had at that level. There's a lot of conflicting info out there so I figured if someone had it dialed in, they'd be lurking here :)
biggles
Forum Regular
Forum Regular
Posts: 806
Joined: Tue Jul 15, 2008 2:38 pm
Location: Sweden
Contact:

Re: nginx serving large static files

Unread post by biggles »

Almost sounds lika a mission for a CDN...
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: nginx serving large static files

Unread post by faris »

biggles wrote:Almost sounds lika a mission for a CDN...
I second that.

To be honest I'm not (yet) sure what all the fuss is about regarding nginx. A CDN would seem like the best option in this case.

Ooh! I've just thought of a new project: An auto-CDN Plugin for Plesk (auto-uploads changed files in specified directories ever X minutes)

I'm aware that some CDNs can connect via FTP and do this themselves, but Amazon (AFAIK) doesn't, and that's what I use.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
dayo
Forum Regular
Forum Regular
Posts: 158
Joined: Sun Jul 12, 2009 1:33 pm

Re: nginx serving large static files

Unread post by dayo »

I tried to post a response but it seems something in the setup here doesn't like Nginx :). The ASL threat mitigation measures kicked in as soon as I submitted the post and I got frozen out. I am now using a proxy server to post this as I am no longer able to browse the site otherwise

As I don't know what exactly it was in my post that triggered the alarm, I'll just say the best place to get information on Nginx is the mailing list.

Good luck!
premierhosting
Forum Regular
Forum Regular
Posts: 257
Joined: Wed Aug 04, 2010 2:52 pm

Re: nginx serving large static files

Unread post by premierhosting »

Cost to benefit, CDNs still too expensive when lighter resources will do. Assuming the lighter resources work :)
faris
Long Time Forum Regular
Long Time Forum Regular
Posts: 2321
Joined: Thu Dec 09, 2004 11:19 am

Re: nginx serving large static files

Unread post by faris »

I can't argue with that.
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>
dayo
Forum Regular
Forum Regular
Posts: 158
Joined: Sun Jul 12, 2009 1:33 pm

Re: nginx serving large static files

Unread post by dayo »

After thinking about it I reckon the reason I got frozen out was because I had five links in my original post which must be above a certain magic number on the threat/spam mitigation front so I have stripped them all out in the reproduction of the post below
A CDN would obviously help but if you are trying to get more out of your setup, you might want to make sure you are eliminating disc I/O bottlenecks.

In this case, what you want to set up is a system where the files are saved into RAM and served from there. This can be done using a set of modules such as the Memc, srcache and perhaps, the Lua modules (documentation on each on the Nginx wiki site). When a request hits, Nginx can check to see if the file is in the memcache and serve it from there or get it from the filesystem if not and update the cache. This can be local or on (a) separate VPS installation(s) serving as (a) memcache server(s)

The company behind what is apparently one of the largest e-commerce sites in China have been behind some interesting developments with Nginx with these modules along with others that make it more of a web application framework instead of just a webserver and they have been releasing these modules as open source. Basically, you can look at Nginx as a scriptable application that can interact directly with data management systems such as MySQL, PgSQL, Memcache, Redis, MongoDB etc. A lot of stuff don't need to hit a backend application and can be run swftly and with low memory impact directly within the webserver space itself. Your imagination is your limit and people are doing all sorts of wonderful things. That is what the fuss is all about.

There are some basic things you need to look into as well such as the Worker Processes and Worker Connections settings etc.

The best place to find information and advice is the Nginx Mailing List as the developers themselves are very active there.
premierhosting
Forum Regular
Forum Regular
Posts: 257
Joined: Wed Aug 04, 2010 2:52 pm

Re: nginx serving large static files

Unread post by premierhosting »

Good info dayo. I'm going to check into the file caching options. There's a high peak and a long tail for which content is accessed, so if the caching system could prioritize into RAM the most often accessed files for any given time frame, it could really be slick.

Do you think doing bandwidth limiting per client is helpful? The one area that it looks like I can't optimize is the tcp_fin_wait, as the virtual server environment is relying on the underlying tcp stack, I think.
dayo
Forum Regular
Forum Regular
Posts: 158
Joined: Sun Jul 12, 2009 1:33 pm

Re: nginx serving large static files

Unread post by dayo »

I don't think this is a suitable forum to look for detailed responses on this sorts of issues regarding Nginx at this point and as said, your best bet is to post your queries to the Nginx mailing list.
premierhosting
Forum Regular
Forum Regular
Posts: 257
Joined: Wed Aug 04, 2010 2:52 pm

Re: nginx serving large static files

Unread post by premierhosting »

OK, no problem.
mandi22
Forum User
Forum User
Posts: 11
Joined: Sat Oct 06, 2007 8:01 am

Re: nginx serving large static files

Unread post by mandi22 »

premierhosting wrote:OK, no problem.
Seriously,
Nginx is too slow! I am gonna try installing a second Apache instance for serving static content.
At the moment my actual apache is loaded with PHP and other modules and is faster serving files from the same server than nginx which is configured with no mods at all.
Yes, nginx is very lite. I can barely see it on 'top' but I would like to see how apache plain with no mods behave in comparison to nginx plain.
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: nginx serving large static files

Unread post by scott »

Thats on my list with the apache benchmark thread here: https://atomicorp.com/forum/viewtopic.php?f=1&t=5576. I just finished up the cgi/suphp/fcgi performance thread earlier this week, so I've got a baseline to compare it against.
Post Reply