Discussion:
[lopsa-tech] Mail server: hardware specs
(too old to reply)
Gregory Sutter
2008-08-12 00:33:10 UTC
Permalink
Dear colleagues,

My organization needs a new email server. We have a few specific
requirements. Our current mail server is at disk I/O capacity
limits, due to slow hardware RAID-5, insufficient (5) spindles, and
the inherent chattiness and random-access of IMAP. I am hoping to
get advice here to confirm or improve upon what my own research has
produced.

Here are the non-negotiable aspects of the situation:
- 40 users currently, needs to expand to 60.
- OS: FreeBSD -STABLE
- Software disk partition encryption (GELI/GBDE) is mandatory.
- Mail chain: postfix / amavisd-new / clamav / procmail / dovecot
- Mail storage capacity only has to be ~120GB :)

Here's the current hardware which needs to be replaced:

SuperMicro 833T-550 sata case
SuperMicro X6DHE-G dual gig ddr motherboard
2x Intel Xeon 2.8GHz 800FSB 1M active
2x DDR 1G PC2700 ECC Reg
8x WD 250GB RAID SATA drive
3Ware SATA-150 RAID controller 9500S-8

Basically, it can't handle the random-read, random-write of our IMAP
usage, and is completely bogged down at 100% I/O usage during parts
of the day every day. Only 5 of the 8 disks are in the primary RAID,
and we're using GBDE (disk encryption), both of which slow things
down, but despite its 3-year age, that still seems like some hefty
hardware.

I have been thinking of replacing it with this:

1x SuperMicro SC213A-R900UB 2U case, 16 2.5" hotswap bays
1x Super X7DWU/X7DWE dual xeon motherboard
2x CPU, perhaps Xeon E5420 (quad-core, 2.5GHz, 1333MHz bus)
16GB RAM sounds good
10x Seagate ST973402SS 73GB Savvio 10K.2 2.5" SAS/SATA

I don't believe any RAID is necessary in this setup, as users' home
directories (with mail spools) can be split arbitrarily among
several file systems. At least 2 of the drives would be used solely
for warm backups (via rsync from the main file systems).

Any thoughts on hardware selection and setup? Overall, this is
supposed to be a simple, reliable architecture that is easy for a
single sysadmin to maintain when something fails. Another thing to
note is that we're going to get two of whatever server we choose, so
we have the ability to recover when the hardware fails its saving
throw. Thanks.
--
Gregory S. Sutter It is completely dark.
mailto:***@zer0.org You are likely to be eaten by a grue.
http://zer0.org/~gsutter/
Neil Neely
2008-08-12 01:53:22 UTC
Permalink
I'm a bit concerned that your current server is struggling with 40
users. I've handled thousands of users on hardware weaker than what
you are describing. I have not yet tried to do any of this with an
encrypted filesystem, and that is obviously an important variable. For
the encrypted FS - is that just the mail store, or does that also
include your spool directories and such? If every temp file postfix and
your spam/virus scanners use is getting encrypted that would hurt your
capacity quite a bit.

Since you are doing spam scanning on the same server as delivery/access
you might want to look at what pre-queue filtering you are running and
see if you can do more and keep the junk from hitting you disk. Certain
RBL's (like spamhaus) are very reliable and can go a long ways.

FreeBSD does a good job of using surplus RAM for File System caching -
2GB of RAM might be light for everything you are doing on that box,
simply providing extra RAM may help cut down your disk i/o a good bit.

For building out a new server:

Your goal of easy to administer and not using RAID seem at odds for a
lot of reasons, you really want to maximize the spindles you are
throwing at the lot, by isolating them all you are going to have to
manually balance your I/O loads - and almost certainly end up with some
drives hammered hard and others idle. This is what RAID is good at
helping out with.

If disk I/O is your problem that is where you should focus your
attention. SATA and high volume random read/write is not a good
combination.
15k RPM SAS drives should help considerably.

It may be worth taking a look at an external SAN/NAS solution for your
mail store as well.

Neil Neely
http://neil-neely.blogspot.com
Post by Gregory Sutter
Dear colleagues,
My organization needs a new email server. We have a few specific
requirements. Our current mail server is at disk I/O capacity
limits, due to slow hardware RAID-5, insufficient (5) spindles, and
the inherent chattiness and random-access of IMAP. I am hoping to
get advice here to confirm or improve upon what my own research has
produced.
- 40 users currently, needs to expand to 60.
- OS: FreeBSD -STABLE
- Software disk partition encryption (GELI/GBDE) is mandatory.
- Mail chain: postfix / amavisd-new / clamav / procmail / dovecot
- Mail storage capacity only has to be ~120GB :)
SuperMicro 833T-550 sata case
SuperMicro X6DHE-G dual gig ddr motherboard
2x Intel Xeon 2.8GHz 800FSB 1M active
2x DDR 1G PC2700 ECC Reg
8x WD 250GB RAID SATA drive
3Ware SATA-150 RAID controller 9500S-8
Basically, it can't handle the random-read, random-write of our IMAP
usage, and is completely bogged down at 100% I/O usage during parts
of the day every day. Only 5 of the 8 disks are in the primary RAID,
and we're using GBDE (disk encryption), both of which slow things
down, but despite its 3-year age, that still seems like some hefty
hardware.
1x SuperMicro SC213A-R900UB 2U case, 16 2.5" hotswap bays
1x Super X7DWU/X7DWE dual xeon motherboard
2x CPU, perhaps Xeon E5420 (quad-core, 2.5GHz, 1333MHz bus)
16GB RAM sounds good
10x Seagate ST973402SS 73GB Savvio 10K.2 2.5" SAS/SATA
I don't believe any RAID is necessary in this setup, as users' home
directories (with mail spools) can be split arbitrarily among
several file systems. At least 2 of the drives would be used solely
for warm backups (via rsync from the main file systems).
Any thoughts on hardware selection and setup? Overall, this is
supposed to be a simple, reliable architecture that is easy for a
single sysadmin to maintain when something fails. Another thing to
note is that we're going to get two of whatever server we choose, so
we have the ability to recover when the hardware fails its saving
throw. Thanks.
------------------------------------------------------------------------
_______________________________________________
Tech mailing list
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Schwartz, Elizabeth
2008-08-12 01:59:54 UTC
Permalink
FWIW, I saw an enormous speedup with my spam/virus filtering system when
I gave it a ramdisk as a tmp directory. And definitely a strong second
for spamhaus - no point wasting time filtering stuff from pure spam
sources.
Robert Hajime Lanning
2008-08-12 03:20:47 UTC
Permalink
Post by Gregory Sutter
- 40 users currently, needs to expand to 60.
- OS: FreeBSD -STABLE
- Software disk partition encryption (GELI/GBDE) is mandatory.
^^^^^^^^^^^^^^^^^
That will kill you. Unless hardware acceleration is used. Is the
Seagate FDE not usable?
Post by Gregory Sutter
- Mail chain: postfix / amavisd-new / clamav / procmail / dovecot
- Mail storage capacity only has to be ~120GB :)
SuperMicro 833T-550 sata case
SuperMicro X6DHE-G dual gig ddr motherboard
2x Intel Xeon 2.8GHz 800FSB 1M active
2x DDR 1G PC2700 ECC Reg
8x WD 250GB RAID SATA drive
3Ware SATA-150 RAID controller 9500S-8
^^^^^^^^^^^^^^^^^^
And that is horrible for RAID5 performance. 3Ware was designed as the
"cheap"
hardware RAID. I upgraded my performance by 5 times with moving from
3Ware
hardware RAID5 (9xxx series) 12 drives, to hardware JBOD software RAID5.
Also
make sure TCQ is enabled and the depth is large enough.

Basically the main CPU massively out strips the 3Ware RAID processor.
So, you are
not waiting on IO, but on the RAID processor.
Post by Gregory Sutter
Basically, it can't handle the random-read, random-write of our IMAP
usage, and is completely bogged down at 100% I/O usage during parts
of the day every day. Only 5 of the 8 disks are in the primary RAID,
and we're using GBDE (disk encryption), both of which slow things
down, but despite its 3-year age, that still seems like some hefty
hardware.
1x SuperMicro SC213A-R900UB 2U case, 16 2.5" hotswap bays
1x Super X7DWU/X7DWE dual xeon motherboard
2x CPU, perhaps Xeon E5420 (quad-core, 2.5GHz, 1333MHz bus)
16GB RAM sounds good
10x Seagate ST973402SS 73GB Savvio 10K.2 2.5" SAS/SATA
I don't believe any RAID is necessary in this setup, as users' home
directories (with mail spools) can be split arbitrarily among
several file systems. At least 2 of the drives would be used solely
for warm backups (via rsync from the main file systems).
I would not forgo RAID. Either switch to multiple RAID1 or a RAID10
setup. RAID10 would
be the best as load is shared with all spindles. RAID5 is bad for
random access.
"rsync" works, but with downtime during a failure and latency between
updates.

--
END OF LINE
--MCP
Brad Knowles
2008-08-12 03:40:14 UTC
Permalink
Post by Gregory Sutter
Basically, it can't handle the random-read, random-write of our IMAP
usage, and is completely bogged down at 100% I/O usage during parts
of the day every day. Only 5 of the 8 disks are in the primary RAID,
and we're using GBDE (disk encryption), both of which slow things
down, but despite its 3-year age, that still seems like some hefty
hardware.
If disk performance is an issue, then make sure you have enough RAM
and enable softupdates. For maximum performance, you also want to
use RAID-0 or RAID-10, so that you can get the maximum number of
spindles working for each disk operation.

Performance-wise on the IMAP server side, you can't beat Cyrus.
Dovecot is an improvement over Courier-IMAP, but it still can't beat
Cyrus for speed.
Post by Gregory Sutter
I don't believe any RAID is necessary in this setup, as users' home
directories (with mail spools) can be split arbitrarily among
several file systems. At least 2 of the drives would be used solely
for warm backups (via rsync from the main file systems).
Any thoughts on hardware selection and setup? Overall, this is
supposed to be a simple, reliable architecture that is easy for a
single sysadmin to maintain when something fails. Another thing to
note is that we're going to get two of whatever server we choose, so
we have the ability to recover when the hardware fails its saving
throw. Thanks.
Nothing beats building the system and then benchmarking and
stress-testing the hell out of it. Disk benchmarking tools are one
part of that equation, but you also want to test all your
applications, and there you want to make sure you do IMAP
benchmarking as well. Russ Coker has some good tools in this area,
including bonnie++ 1.93d and "postal". Ignore the "experimental"
label on the latest versions of bonnie++ at
<http://www.coker.com.au/bonnie++/experimental/>, and pay close
attention to the latest version of postal at
<http://doc.coker.com.au/projects/postal/>.

I just did a review for _;login:_ magazine of "The Book of IMAP:
Building a Mail Server with Courier and Cyrus" by Peer Heinlein and
Peer Hartleben. Even if you're not going to be building your IMAP
server with Courier or Cyrus, I would highly recommend this book.
They don't go far enough in terms of some of the subjects they cover
(like benchmarking and stress testing), but it's still light years
ahead of the Mullet book.
--
Brad Knowles <***@shub-internet.org>
LinkedIn Profile: <http://tinyurl.com/y8kpxu>
Phil Pennock
2008-08-12 07:06:21 UTC
Permalink
Post by Gregory Sutter
- 40 users currently, needs to expand to 60.
- OS: FreeBSD -STABLE
- Software disk partition encryption (GELI/GBDE) is mandatory.
- Mail chain: postfix / amavisd-new / clamav / procmail / dovecot
- Mail storage capacity only has to be ~120GB :)
Are you able to split that mail-chain by inserting a second postfix in
there and using two hosts?

If you can afford it, split the incoming mail into MX hosts and
storage hosts. The MX hosts handle validation, spam-filtering,
everything to do with limiting what comes through. The storage hosts
take the pre-filtered mail and store it and provide IMAP/POP. This is
really the first of the "easy" splits as a mail-system grows and you
need to segregate functionality.

For one thing, spam-scanning and virus-checking can hit the file-system
with a large amount of I/O, and a large amount of it will be pure
garbage by the nature of being spam. As others recommend, RAM-based
file-systems for their tmp areas can help here. Sometimes, some clavav
ruleset updates are not as ... optimised as one might hope and clamav
CPU usage goes through the roof until a later update fixes things. This
is the hazard in automatically downloading verification rules. Think
"badly tuned regexes".

Also, with this security holes in ClamAV will still let an attacker get
to mail passing through the system but won't let them get to the history
of all email. It's better to be in a situation where you can say "all
mail received in this time-frame may have been read by an intruder" than
"all of your mail may have been read by an intruder".

This means that more of the RAM on the storage host will be handling
real email in buffer caches. Given the random-access loads, this helps.

Oh, and the temporary filesystem storage where ClamAV works being RAM
will mean that it doesn't need to be encrypted.

The other advantage to this approach is that you're then halfway to
being able to scale the storage layer "horizontally"; the MX hosts can
have logic to map different email addresses to different storage hosts,
so each only handles roughly half the load. You'll still need to make
sure that internal users are using hostnames which let you direct them
to the correct host for their mail, but this is separated from the
public information. For the internal users, the two main options would,
at that later date, be either a service hostname and a Perdition proxy
running on a front-end or internal DNS giving a per-user CNAME for their
mailhost, moving the partitioning logic into a script building a DNS
zone. "fred.mailbox.example.net. IN CNAME mailstore-2a.example.net." or
whatever.

Later, should you need to, use the shared storage resiliency poison of
your choice to scale this up and let you lose a given server without
losing mail. Eg, 2 "local" mail partitions per host plus two
"non-local" partitions, not typically used; 3 hosts. Each of the
mail-partitions is a shared filesystem with another host. You lose one
host, the other two split the load. If the MX hosts can be configured
to have fall-backs per address (I don't know Postfix, sorry) then you
can continue to deliver mail and even survive reboots without
interruption. For the client-side, this works better with the front-end
host and a proxy with fallback configuration than it does with the
mailbox CNAME approach. (The poisons available to FreeBSD for this are
rather limited compared to those for Linux but your local apothecary may
carry a wider selection by the time this is ever an issue for you).

The *real* advantage is that you don't want your continued operations to
be dependent upon ever more expensive hardware, so designing to split
load early is a win.
Post by Gregory Sutter
I don't believe any RAID is necessary in this setup, as users' home
RAID 5 is comparatively slow, even against other RAID options; just
because RAID5 is too slow, that doesn't mean you need to lose all disk
resiliency. RAID5 is not well geared towards use in email, as you end
up needing to rewrite multiple blocks for every little mail received and
for every rename; especially with a format like Maildir where every time
a new mail is read by a user, the mail needs to be moved to a different
directory, so there's *really* more small fragmented I/O work.

If you can afford so many disks, use mirroring for pairs of redundant
disks. Use striping if you need to speed it up and disk I/O capacity is
really the bottleneck. With encryption in there, are you sure that's
the bottleneck?

-Phil
Tom Limoncelli
2008-08-12 08:10:44 UTC
Permalink
Another way to separate out the MX host is to let someone else do the
MX and virus scanning for you. At my last job I was very happy with
messagelabs.com. They were our inbound MX. They collect all the
email, quarantine the virii, and hold the spam. Each user had a
dashboard they could log into view their held spam. It was nice not
having to update SpamAssassin and other programs so frequently. They
did a better job of virus scanning than I could. I wasn't concerned
about our email flowing through their systems because, well, it was
flowing unencrypted over the internet already, right? If the user was
concerned about privacy they should have encrypted the message.

There are other companies that provide this kind of service
(postini.com) but I found MessageLabs.com to be better because it had
less features and configurability, thus was easier for my users to
understand.

Tom
Tom Limoncelli
2008-08-12 08:12:47 UTC
Permalink
Another way to separate out the MX host is to let someone else do the
MX and virus scanning for you. At my last job I was very happy with
messagelabs.com. They were our inbound MX. They collect all the
email, quarantine the virii, and hold the spam. Each user had a
dashboard they could log into view their held spam. It was nice not
having to update SpamAssassin and other programs so frequently. They
did a better job of virus scanning than I could. I wasn't concerned
about our email flowing through their systems because, well, it was
flowing unencrypted over the internet already, right? If the user was
concerned about privacy they should have encrypted the message.

There are other companies that provide this kind of service
(postini.com) but I found MessageLabs.com to be better because it had
less features and configurability, thus was easier for my users to
understand.

Tom
Brad Knowles
2008-08-13 02:25:05 UTC
Permalink
Post by Tom Limoncelli
There are other companies that provide this kind of service
(postini.com) but I found MessageLabs.com to be better because it had
less features and configurability, thus was easier for my users to
understand.
The disadvantage with outsourcing this service is that you're now
completely dependant on that service provider being up and running,
otherwise all your e-mail stops. All providers in this space have
had periodic problems with reliability -- it's tough to build systems
to scale to these kinds of levels.

Moreover, there's a whole ecosystem of spammers whose sole purpose in
life is to craft messages to get through specific outsourcing
providers.


My virtual domain hosting provider provides filtering via Postini for
all their customers -- whether I want it or not. Overall, it's
actually been surprisingly good, and their interface and features
greatly improved after Google bought them.

However, they have found that they need to do their own second layer
of system-side filtering with tools like SpamAssassin, because there
is still so much that leaks through.

And I apply a third level of filtering on top of that, done through
carefully crafted anti-spam rules that I've developed over the years
with my MUA, plus what my MUA has been able to develop for itself
through their automatic learning process.


In my regular life, I help administer the anti-spam scanning system
for The University of Texas at Austin, which uses Ironport e-mail
security appliances.

I've been very pleasantly surprised at what these things can do.
Over the summer, handling rates that are considerably lower than what
we should see during normal Fall or Spring semesters, we've been
rejecting on the order of fifty million messages/delivery attempts
per day, and something on the order of one billion per month.

And we're only a large public research university, with ~50,000
students and ~20,000 faculty and staff.
--
Brad Knowles <***@shub-internet.org>
LinkedIn Profile: <http://tinyurl.com/y8kpxu>
Adam Tauno Williams
2008-08-12 17:50:55 UTC
Permalink
Post by Phil Pennock
Post by Gregory Sutter
- 40 users currently, needs to expand to 60.
- OS: FreeBSD -STABLE
- Software disk partition encryption (GELI/GBDE) is mandatory.
- Mail chain: postfix / amavisd-new / clamav / procmail / dovecot
- Mail storage capacity only has to be ~120GB :)
Are you able to split that mail-chain by inserting a second postfix in
there and using two hosts?
He's only got 40 users! Buy more RAM, 2Gb is ridiculously low.
Post by Phil Pennock
For one thing, spam-scanning and virus-checking can hit the file-system
with a large amount of I/O, and a large amount of it will be pure
garbage by the nature of being spam.
And all without accomplishing much of anything. SPAMHaus + SpamCOP +
Greylisting. Then CLAMAV if you want. Toss everything else as it
provides an imperceptible improvement at a massive cost.
Post by Phil Pennock
Oh, and the temporary filesystem storage where ClamAV works being RAM
will mean that it doesn't need to be encrypted.
I agree completely with using a RAM disk for temporary files. RAM is
cheaper and faster than *fast* disk.
Post by Phil Pennock
Post by Gregory Sutter
I don't believe any RAID is necessary in this setup, as users' home
RAID 5 is comparatively slow, even against other RAID options; just
because RAID5 is too slow,
It is hard to believe a mail server for 40 users can't work around poky
RAID5 by just having sufficient RAM. Unless you are subscribed to
thousands of maillists.

And most importantly, switch your IMAP server to Cyrus.
Atom Powers
2008-08-12 18:12:11 UTC
Permalink
On Tue, Aug 12, 2008 at 10:50 AM, Adam Tauno Williams <
Post by Adam Tauno Williams
Post by Phil Pennock
Post by Gregory Sutter
- 40 users currently, needs to expand to 60.
- OS: FreeBSD -STABLE
- Software disk partition encryption (GELI/GBDE) is mandatory.
- Mail chain: postfix / amavisd-new / clamav / procmail / dovecot
- Mail storage capacity only has to be ~120GB :)
Are you able to split that mail-chain by inserting a second postfix in
there and using two hosts?
He's only got 40 users! Buy more RAM, 2Gb is ridiculously low.
No it isn't. With 40 users, and an average mail size of a 10-100KB, 1GB
should be enough. Your messages aren't spending that much time in RAM
anyway, even when you are spooling to a RAMdisk.
Post by Adam Tauno Williams
Post by Phil Pennock
For one thing, spam-scanning and virus-checking can hit the file-system
with a large amount of I/O, and a large amount of it will be pure
garbage by the nature of being spam.
And all without accomplishing much of anything. SPAMHaus + SpamCOP +
Greylisting. Then CLAMAV if you want. Toss everything else as it
provides an imperceptible improvement at a massive cost.
++ False positives. I prefer a spam filter that I can tune to my business
requirements. And anti-virus on the mail filter is a *must*.
Post by Adam Tauno Williams
Post by Phil Pennock
Oh, and the temporary filesystem storage where ClamAV works being RAM
will mean that it doesn't need to be encrypted.
I agree completely with using a RAM disk for temporary files. RAM is
cheaper and faster than *fast* disk.
Faster maybe, but not cheaper. And last I checked BIGMEM still had issues on
some architectures with FreeBSD 6, so 4+GB RAM may not even be an option.
Post by Adam Tauno Williams
Post by Phil Pennock
Post by Gregory Sutter
I don't believe any RAID is necessary in this setup, as users' home
RAID 5 is comparatively slow, even against other RAID options; just
because RAID5 is too slow,
It is hard to believe a mail server for 40 users can't work around poky
RAID5 by just having sufficient RAM. Unless you are subscribed to
thousands of maillists.
RAM won't help with RAID5, encryption, or disk I/O.
Post by Adam Tauno Williams
And most importantly, switch your IMAP server to Cyrus.
Use RAID10,
Use a RAMdisk for your mail spool;
If you must have encryption, configure a separate partition for your mail
store and only encrypt that partition, get a hardware crypto card if
necessary,
Move your spam filter to a font-end MX host (with lmtp passthrough).

These are all good suggestions, regardless of the specific software/hardware
you choose/are required to use.
--
Perfection is just a word I use occasionally with mustard.
--Atom Powers--
Adam Tauno Williams
2008-08-12 18:19:26 UTC
Permalink
Post by Atom Powers
On Tue, Aug 12, 2008 at 10:50 AM, Adam Tauno Williams
Post by Phil Pennock
Post by Gregory Sutter
- 40 users currently, needs to expand to 60.
- OS: FreeBSD -STABLE
- Software disk partition encryption (GELI/GBDE) is
mandatory.
Post by Phil Pennock
Post by Gregory Sutter
- Mail chain: postfix / amavisd-new / clamav / procmail /
dovecot
Post by Phil Pennock
Post by Gregory Sutter
- Mail storage capacity only has to be ~120GB :)
Are you able to split that mail-chain by inserting a second
postfix in
Post by Phil Pennock
there and using two hosts
He's only got 40 users! Buy more RAM, 2Gb is ridiculously
low
No it isn't. With 40 users, and an average mail size of a 10-100KB,
1GB should be enough.
If he was just running a IMAP/POP/SMTP server sure. I supported 10x
that number of users in 1Gb of RAM for years. But since he also runs
encryption and an anti-SPAM-monster then it certainly isn't.

Encryption means he has to read, decrypt, update, recrypt, write all his
data. He needs the headroom for that.
Post by Atom Powers
Post by Phil Pennock
Post by Gregory Sutter
will mean that it doesn't need to be encrypted.
I agree completely with using a RAM disk for temporary files. RAM
is cheaper and faster than *fast* disk
Faster maybe, but not cheaper. And last I checked BIGMEM still had
issues on some architectures with FreeBSD 6, so 4+GB RAM may not even
be an option.
Then switch to an OS that supports a realistic [modern] amount of
memory.
Post by Atom Powers
If you must have encryption, configure a separate partition for your
mail store and only encrypt that partition,
This is certainly true. There isn't much point in crypting your
meta-data, just do your mail store volume.
Phil Pennock
2008-08-12 19:23:07 UTC
Permalink
Post by Adam Tauno Williams
Post by Atom Powers
Faster maybe, but not cheaper. And last I checked BIGMEM still had
issues on some architectures with FreeBSD 6, so 4+GB RAM may not even
be an option.
Then switch to an OS that supports a realistic [modern] amount of
memory.
BIGMEM is a Linux term.

FreeBSD supports more than 4GB RAM. With amd64, it supports what you'd
expect; if stuck with a 32-bit CPU then you can build a kernel with the
PAE option but there are some caveats there.

The OS handbook, which comes with the OS, describes this.

http://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/compatibility-memory.html

I'm not currently using FreeBSD professionally, only for my own system,
so haven't stayed up-to-date; but the only current limit I know of
(beyond the usual 32-bit limits) is the amount of RAM which can be
allocated as kernel memory.

The "more experienced" *cough* sysadmin here will shudder at the idea of
a _Kernel_ requiring more than 2GB of RAM. Apparently, there are
downsides to using ZFS. So there's work to raise that paltry limit.

-Phil
d***@lang.hm
2008-08-12 21:56:19 UTC
Permalink
Post by Phil Pennock
Post by Adam Tauno Williams
Post by Atom Powers
Faster maybe, but not cheaper. And last I checked BIGMEM still had
issues on some architectures with FreeBSD 6, so 4+GB RAM may not even
be an option.
Then switch to an OS that supports a realistic [modern] amount of
memory.
BIGMEM is a Linux term.
the linux term is HIGHMEM

David Lang
Nathan Hruby
2008-08-12 22:03:05 UTC
Permalink
Post by d***@lang.hm
Post by Phil Pennock
Post by Adam Tauno Williams
Post by Atom Powers
Faster maybe, but not cheaper. And last I checked BIGMEM still had
issues on some architectures with FreeBSD 6, so 4+GB RAM may not even
be an option.
Then switch to an OS that supports a realistic [modern] amount of
memory.
BIGMEM is a Linux term.
the linux term is HIGHMEM
I think the 32bit PAE enabled kernels witth the 4g/4g split patch
redhat uses were/are called "bigmem" which is part of the confusion,
but nevertheless, none of this particular subthread is ontopic for the
OP asking about his FreeBSD mailserver :)

-n
--
-------------------------------------------
nathan hruby <***@gmail.com>
metaphysically wrinkle-free
-------------------------------------------
Brandon S. Allbery KF8NH
2008-08-12 23:03:38 UTC
Permalink
Post by Nathan Hruby
Post by d***@lang.hm
Post by Phil Pennock
Post by Adam Tauno Williams
Post by Atom Powers
Faster maybe, but not cheaper. And last I checked BIGMEM still had
issues on some architectures with FreeBSD 6, so 4+GB RAM may not even
be an option.
Then switch to an OS that supports a realistic [modern] amount of
memory.
BIGMEM is a Linux term.
the linux term is HIGHMEM
I think the 32bit PAE enabled kernels witth the 4g/4g split patch
redhat uses were/are called "bigmem" which is part of the confusion,
Also, (which may explain Red Hat) the original >1GB patchset for Linux
(where you picked what division between kernel and user memory you
wanted, with some pretty major tradeoffs) was called bigmem.

But really, if you want to use lots of memory, you want amd64; x86 PAE
is most charitably called a hack.
--
brandon s. allbery [solaris,freebsd,perl,pugs,haskell] ***@kf8nh.com
system administrator [openafs,heimdal,too many hats] ***@ece.cmu.edu
electrical and computer engineering, carnegie mellon university KF8NH
Aleksandar Ivanisevic
2008-08-13 08:46:14 UTC
Permalink
Post by Adam Tauno Williams
Post by Phil Pennock
Post by Gregory Sutter
- 40 users currently, needs to expand to 60.
- OS: FreeBSD -STABLE
- Software disk partition encryption (GELI/GBDE) is mandatory.
- Mail chain: postfix / amavisd-new / clamav / procmail / dovecot
- Mail storage capacity only has to be ~120GB :)
Are you able to split that mail-chain by inserting a second postfix in
there and using two hosts?
He's only got 40 users! Buy more RAM, 2Gb is ridiculously low.
No it isn't. I'm running a mail server for 20 people in a xen XM with
256 megs of RAM. I don't have antivirus though, but do have spamassassin.
Yves Dorfsman
2008-08-13 14:23:55 UTC
Permalink
Post by Aleksandar Ivanisevic
Post by Adam Tauno Williams
Post by Phil Pennock
Are you able to split that mail-chain by inserting a second postfix in
there and using two hosts?
He's only got 40 users! Buy more RAM, 2Gb is ridiculously low.
No it isn't. I'm running a mail server for 20 people in a xen XM with
256 megs of RAM. I don't have antivirus though, but do have spamassassin.
In theory the more users the more spam one gets, but this is not necessarily
linear, especially on very small setups like what we are talking about here
(my guess is that universities for example are probably comparable in terms
of average number of messages per email account).

Shouldn't we talk volume (nbr. of connections per day) here instead of
number of users ?

A 100 user setup with only emails that have not been publicised will be a
lighter load than a 10 user setup where half of them have been long time
internet users, with their email publicised in usenet, whois etc...

Architecture will also make a difference on where the bottleneck is. I like
to use pathetically low compute power machines (at home, not $work), and my
bottleneck always end up being the cpu (slow cpus are available, but you
can't buy slow disk or slow network card any more, and even if you could,
they do would not have any advantage in term of electric power anyway).
--
Yves.
http://www.SollerS.ca
Doug Hughes
2008-08-13 14:31:38 UTC
Permalink
Post by Yves Dorfsman
Architecture will also make a difference on where the bottleneck is. I like
to use pathetically low compute power machines (at home, not $work), and my
bottleneck always end up being the cpu (slow cpus are available, but you
can't buy slow disk or slow network card any more, and even if you could,
they do would not have any advantage in term of electric power anyway).
I don't mean to be picayune, bou don't consider SATA drives slow? They
are half the speed of drives that were available 4+ years ago in SCSI/FC
format. Laptop drives are 1/3 the speed. I'll agree on slow network
cards unless you scour ebay and take your chances, but buying 2.5" 5400
RPM drives is pretty easy. Even the adapters are rather plentiful and
inexpensive.
SATA drives are a revolution in capacity per unit space and price, but
certainly not speed. I'd say it's increasingly difficult to buy fast
drives, but NVRAM and flash are rapidly filling that niche. The SATA
revolution has adjusted a lot of expectations bars.
Brad Knowles
2008-08-13 02:13:50 UTC
Permalink
Post by Phil Pennock
You'll still need to make
sure that internal users are using hostnames which let you direct them
to the correct host for their mail, but this is separated from the
public information. For the internal users, the two main options would,
at that later date, be either a service hostname and a Perdition proxy
running on a front-end or internal DNS giving a per-user CNAME for their
mailhost, moving the partitioning logic into a script building a DNS
zone. "fred.mailbox.example.net. IN CNAME mailstore-2a.example.net." or
whatever.
You can solve a lot of these types of problems with the proper type
of POP3/IMAP proxy (including SMTP routing). Perdition is one good
choice in this area, but there's also local IMAP connection-cache
proxying which can further help performance issues with ill-behaved
clients, such as most webmail systems.

Note that IMAP connection-cache proxying can help with some other
clients, too -- a fact that was missed by the Peer Heinlein and Peer
Hartleben.


Architecture-wise, I laid out pretty much all this stuff in my two
Invited Talks I've done at LISA.
--
Brad Knowles <***@shub-internet.org>
LinkedIn Profile: <http://tinyurl.com/y8kpxu>
Edward Ned Harvey
2008-08-12 16:43:15 UTC
Permalink
What sort of Internet connection do you have? I have a very hard time
believing you're facing a disk IO bottleneck. How much memory does your
current system have? Maybe it's just thrashing due to low memory...

FWIW, I've had good performance out of the 3ware raid card, but you must
enable Write Back (not Write Through) and enable Adaptive Readahead. And,
if you do this, you really really want a Battery Back Up on the card. The
performance difference is up to 20x faster (as low as 0% faster, for large
sustained i/o).
-----Original Message-----
Of Gregory Sutter
Sent: Monday, August 11, 2008 8:33 PM
Subject: [lopsa-tech] Mail server: hardware specs
Dear colleagues,
My organization needs a new email server. We have a few specific
requirements. Our current mail server is at disk I/O capacity limits,
due to slow hardware RAID-5, insufficient (5) spindles, and the
inherent chattiness and random-access of IMAP. I am hoping to get
advice here to confirm or improve upon what my own research has
produced.
- 40 users currently, needs to expand to 60.
- OS: FreeBSD -STABLE
- Software disk partition encryption (GELI/GBDE) is mandatory.
- Mail chain: postfix / amavisd-new / clamav / procmail / dovecot
- Mail storage capacity only has to be ~120GB :)
SuperMicro 833T-550 sata case
SuperMicro X6DHE-G dual gig ddr motherboard 2x Intel Xeon 2.8GHz 800FSB
1M active 2x DDR 1G PC2700 ECC Reg 8x WD 250GB RAID SATA drive 3Ware
SATA-150 RAID controller 9500S-8
Basically, it can't handle the random-read, random-write of our IMAP
usage, and is completely bogged down at 100% I/O usage during parts of
the day every day. Only 5 of the 8 disks are in the primary RAID, and
we're using GBDE (disk encryption), both of which slow things down, but
despite its 3-year age, that still seems like some hefty hardware.
1x SuperMicro SC213A-R900UB 2U case, 16 2.5" hotswap bays 1x Super
X7DWU/X7DWE dual xeon motherboard 2x CPU, perhaps Xeon E5420 (quad-
core, 2.5GHz, 1333MHz bus) 16GB RAM sounds good 10x Seagate ST973402SS
73GB Savvio 10K.2 2.5" SAS/SATA
I don't believe any RAID is necessary in this setup, as users' home
directories (with mail spools) can be split arbitrarily among several
file systems. At least 2 of the drives would be used solely for warm
backups (via rsync from the main file systems).
Any thoughts on hardware selection and setup? Overall, this is
supposed to be a simple, reliable architecture that is easy for a
single sysadmin to maintain when something fails. Another thing to
note is that we're going to get two of whatever server we choose, so we
have the ability to recover when the hardware fails its saving throw.
Thanks.
--
Gregory S. Sutter It is completely dark.
http://zer0.org/~gsutter/
Robert Brockway
2008-08-13 18:29:58 UTC
Permalink
Post by Gregory Sutter
- 40 users currently, needs to expand to 60.
As others have noted you should be having no problem with 40 users.
Post by Gregory Sutter
- OS: FreeBSD -STABLE
- Software disk partition encryption (GELI/GBDE) is mandatory.
Why is this mandatory?

If you are sending & receiving email in cleartext then there may not be
much point to encrypting it on disk. Do you encrypt all or even most of
your email with GPG or similar? Chances are the vast majority of people
who you exchange email with don't have any facility to encrypt/decrypt
email.

Security is a risk assessment. You may be paying a hefty performance
price for on-disk encryption and it may be buying you very little.

Rob
--
"With sufficient thrust, pigs fly just fine..."
-- RFC 1925 "The Twelve Networking Truths"
Gregory Sutter
2008-08-22 04:20:53 UTC
Permalink
Post by Gregory Sutter
My organization needs a new email server. We have a few specific
requirements. Our current mail server is at disk I/O capacity
limits, due to slow hardware RAID-5, insufficient (5) spindles, and
the inherent chattiness and random-access of IMAP. I am hoping to
get advice here to confirm or improve upon what my own research has
produced.
Dear colleagues,

I want to apologize for not getting back to you in the above thread.
Real life, in the form of other work issues, caught up with me, and
now I am off for vacation, leaving this problem for me when I return
in early September. I'm sorry for my silence, and will reply further
then.

(See you on the playa?)
--
Gregory S. Sutter Fnord.
mailto:***@zer0.org
http://zer0.org/~gsutter/
Continue reading on narkive:
Loading...