Discussion:
[lopsa-tech] Does anybody sell wired (not wireless!) routers anymore?
(too old to reply)
Rick Thomas
2011-09-16 05:50:30 UTC
Permalink
My ancient SMC7008VBR 8-port wired (not wireless) home router died as
a result of hurricane Irene. I've temporarily replaced it with it's
predecessor, a 4-port SMC7004VBR, but that was underpowered and on
it's last legs 8 years ago when I replaced it. So I'm in the market
for a new wired (not wireless) router.

I'd like to find something that can run DD-WRT or OpenWRT.
It needs to be low power (run for a couple of hours on a dedicated so/
ho UPS);
I don't need or want Wi-Fi in this application;
I'd like it to support 2 or more LANs, jointly NATed onto one WAN port;
I'd like it to be inexpensive (about $100 or less, but I'd go as high
as $200 if necessary);
and I'd like to be able to support IPv4-NAT/NPT, IPv6, and simple
firewall rules.

I've googled "wired router" and searched the OpenWRT and DD-WRT
websites for hardware recommendations, but everything I find is
oriented to WiFi, and has at most 2 ethernet ports (one WAN and one
LAN, plus lots of groovy WiFi that I don't want or need)

As a last resort, I'm willing to build my own from parts (e.g. a
routerstation Pro or ALIX board and OpenWRT) but a commercial solution
would be preferable.

Anybody got a suggestion?


Thanks!

Rick
Graham Dunn
2011-09-16 13:04:27 UTC
Permalink
Check out something like http://soekris.com/products/net4501-1.html ?
I have one of these running m0n0wall (http://m0n0.ch/wall/).

Graham
My ancient SMC7008VBR 8-port wired (not wireless) home router died as a
result of hurricane Irene.  I've temporarily replaced it with it's
predecessor, a 4-port SMC7004VBR, but that was underpowered and on it's last
legs 8 years ago when I replaced it. So I'm in the market for a new wired
(not wireless) router.
I'd like to find something that can run DD-WRT or OpenWRT.
       It needs to be low power (run for a couple of hours on a dedicated
so/ho UPS);
       I don't need or want Wi-Fi in this application;
       I'd like it to support 2 or more LANs, jointly NATed onto one WAN
port;
       I'd like it to be inexpensive (about $100 or less, but I'd go as high
as $200 if necessary);
       and I'd like to be able to support IPv4-NAT/NPT, IPv6, and simple
firewall rules.
I've googled "wired router" and searched the OpenWRT and DD-WRT websites for
hardware recommendations, but everything I find is oriented to WiFi, and has
at most 2 ethernet ports (one WAN and one LAN, plus lots of groovy WiFi that
I don't want or need)
As a last resort, I'm willing to build my own from parts (e.g. a
routerstation Pro or ALIX board and OpenWRT) but a commercial solution would
be preferable.
Anybody got a suggestion?
Thanks!
Rick
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
John Stoffel
2011-09-16 13:13:05 UTC
Permalink
Graham> Check out something like
Graham> http://soekris.com/products/net4501-1.html ? I have one of
Graham> these running m0n0wall (http://m0n0.ch/wall/).

I was just going to suggest this too, get a pcengines.ch board and put
m0n0wall on and you're happy. I'd also think about splitting up the
internal switch part and the external router/gateway part. This way
can have GigE internally, but the external router can live happily
with 100mb/s ethernet.
Liyun Yu
2011-09-16 13:51:12 UTC
Permalink
Anyone used the OWC (other world computing) SSD (6G) for RAID Array?
Any suggestions/recommendations on SSD based RAID on the market
or drives good for RAID ? Or any pointers to some reference benchmark?
Yes I do have budget issues. I also visited SSd page on Tom's hardware site
which was more than 8 months old.

I did not access the SSD market for a while. Yesterday one of our colleagues
sent me a link at OWC which indicated that their drives has writing speed above
500MB
for some of their 6G SSD drives. I thought SSD was at 150MB usually therefore
this is very attractive to me for our app which pulls a high volume of dataflow
from
the central RAID.

Thanks

Liyun Yu
d***@lang.hm
2011-09-16 18:07:43 UTC
Permalink
Post by Liyun Yu
Anyone used the OWC (other world computing) SSD (6G) for RAID Array?
Any suggestions/recommendations on SSD based RAID on the market
or drives good for RAID ? Or any pointers to some reference benchmark?
Yes I do have budget issues. I also visited SSd page on Tom's hardware site
which was more than 8 months old.
I did not access the SSD market for a while. Yesterday one of our colleagues
sent me a link at OWC which indicated that their drives has writing speed
above 500MB
for some of their 6G SSD drives. I thought SSD was at 150MB usually therefore
this is very attractive to me for our app which pulls a high volume of
dataflow from
the central RAID.
one thing to watch out for with SSDs, most of them do not have good
behavior in the face of power failures. they all cache writes in ram on
the disk and most of them have no way of getting those writes to flash if
the power fails.

David Lang
Liyun Yu
2011-09-16 18:45:28 UTC
Permalink
Thanks David, it is good to know this.

One of our team members suggested that the SSD also tend to be
wear out every 12-16 months. I was puzzled by that statement since
I never know if the RAM or other chip-based memory device would be
wear out electronically. That is another topic maybe.

Thanks,

Liyun
Post by Liyun Yu
Anyone used the OWC (other world computing) SSD (6G) for RAID Array?
Any suggestions/recommendations on SSD based RAID on the market
or drives good for RAID ? Or any pointers to some reference benchmark?
Yes I do have budget issues. I also visited SSd page on Tom's hardware site
which was more than 8 months old.
I did not access the SSD market for a while. Yesterday one of our colleagues
sent me a link at OWC which indicated that their drives has writing speed
above 500MB
for some of their 6G SSD drives. I thought SSD was at 150MB usually therefore
this is very attractive to me for our app which pulls a high volume of
dataflow from
the central RAID.
one thing to watch out for with SSDs, most of them do not have good behavior
in the face of power failures. they all cache writes in ram on the disk and
most of them have no way of getting those writes to flash if the power fails.
David Lang
d***@lang.hm
2011-09-16 19:19:32 UTC
Permalink
Post by Liyun Yu
Thanks David, it is good to know this.
One of our team members suggested that the SSD also tend to be
wear out every 12-16 months. I was puzzled by that statement since
I never know if the RAM or other chip-based memory device would be
wear out electronically. That is another topic maybe.
Primer on SSD flash technology limits

all of the chip based memory devices that are able to maintain the data
without power have a limited number of write cycles.

In the case of modern flash drives this is in the millions of cycles on an
individual block.

older flash drives wear out faster than newer ones, MLC flash drives wear
out faster than SLC drives (but MLC stores two bits per cell nstead of one
and has further price advantages due to volume production so it's _far_
cheaper)

flash drives write data in chunks much larger than disk sectors (128K per
chunk is common and far from the largest), if the disk doesn't buffer
writes agressivly there is a symptom called 'write magnification' that
makes it so that when you write one 512 byte block of data to the disk it
is forced to write 128K of data to the disk. As far as I know, all modern
drives have large ram buffers to limit this effect.

erasing data on a flash drive is very slow, SSDs get around this by
writing a new copy of the data to a new location and erasing the old copy
sometime later. All modern drives contain more flash than what you pay for
so that they have a large pool of 'blank' space that they can write to
(this also helps spread out the writes across more flash so that you are
less likely to hit the 1m write limit on any one spot). If you are doing a
LOT of writing to the SSD you can run into the situation where the drive
no longer has blank spots to write the data to and it has to slow down to
erase space before it can write any more. you can reduce the likelyhood of
running into this by not using all the space on the disk so it has more
unused space (also by picking a drive, controller card, and OS that
support the TRIM command so that when you free up space on disk you can
tell the drive that you don't care about it amymore and it doesn't have to
preserve it)

most flash drives are slower to write to than a high performance spinning
drive, but they eliminate seek overhead so they can still be a win when
writing, and can be a HUGE win for random reads.


I purchased a couple dozen machines with the Intel x25E SLC flash drives
about three years ago. these machines are log servers and so writing data
continuously (hundreds of gigs per day) I have had a half dozen of them
'fail' on me, and from discussions on the postrgres performance list this
is not an uncommon thing. In most cases the failure seems to be firmware,
not storage failures, but the bottom line is they quit working. In theory
when a drive wears out you should still be able to read the last data
written to it, you just won't be able to write anything. In practice there
are times when you just can't access the disk.

David Lang
Post by Liyun Yu
Thanks,
Liyun
Post by d***@lang.hm
Post by Liyun Yu
Anyone used the OWC (other world computing) SSD (6G) for RAID Array?
Any suggestions/recommendations on SSD based RAID on the market
or drives good for RAID ? Or any pointers to some reference benchmark?
Yes I do have budget issues. I also visited SSd page on Tom's hardware site
which was more than 8 months old.
I did not access the SSD market for a while. Yesterday one of our colleagues
sent me a link at OWC which indicated that their drives has writing speed
above 500MB
for some of their 6G SSD drives. I thought SSD was at 150MB usually therefore
this is very attractive to me for our app which pulls a high volume of
dataflow from
the central RAID.
one thing to watch out for with SSDs, most of them do not have good
behavior in the face of power failures. they all cache writes in ram on the
disk and most of them have no way of getting those writes to flash if the
power fails.
David Lang
Luke S. Crawford
2011-09-16 19:08:14 UTC
Permalink
Post by Liyun Yu
Thanks David, it is good to know this.
One of our team members suggested that the SSD also tend to be
wear out every 12-16 months. I was puzzled by that statement since
I never know if the RAM or other chip-based memory device would be
wear out electronically. That is another topic maybe.
I've heard this as well... but in my experience, SSDs fail like disks,
not like tires, which is to say, the failure pattern seems fairly random,
and more correlated to age than to use. I didn't stick around the
employer with thousands of SSDs in production to tell, but for the short
time I was there, I saw something that looked a lot like the begining of
the regular bathtub curve of hard drive failures.

Of course, these were very early 'enterprise' SSD, so I imagine things
are different now... But my point is that yeah, SSDs fail, but so do
hard drives. Seems to me like you should solve the problem of SSD
failures the same way you solve the problem of hard drive failures;
you mirror two different SSDs from different lots, and you replace the
things when they fail.
Doug Hughes
2011-09-16 19:06:47 UTC
Permalink
Post by d***@lang.hm
Post by Liyun Yu
Anyone used the OWC (other world computing) SSD (6G) for RAID Array?
Any suggestions/recommendations on SSD based RAID on the market
or drives good for RAID ? Or any pointers to some reference benchmark?
Yes I do have budget issues. I also visited SSd page on Tom's hardware site
which was more than 8 months old.
I did not access the SSD market for a while. Yesterday one of our colleagues
sent me a link at OWC which indicated that their drives has writing
speed above 500MB
for some of their 6G SSD drives. I thought SSD was at 150MB usually therefore
this is very attractive to me for our app which pulls a high volume of
dataflow from
the central RAID.
one thing to watch out for with SSDs, most of them do not have good
behavior in the face of power failures. they all cache writes in ram on
the disk and most of them have no way of getting those writes to flash
if the power fails.
This, along with price and performance is one of the reasons that I like
the Intel 320 SSD. I get them on amazon. You can probably get faster
with OCZ, but the reviews on the reliability of the OCZ on amazon and
other places made me a bit nervous. This 320 is about as fast as our
older, smaller X-25E drives, but at significantly higher space for lower
price.

ALso, it has a capaciter to flush the SRAM to flash (small since it just
keeps mapping tables there). This makes it suitable to use as a ZFS ZIL,
among other things.

http://www.anandtech.com/show/4244/intel-ssd-320-review

I did some extensive looking around for about 3 days and various tests
before settling on this one in the Summer time frame.
Anton Cohen
2011-09-17 07:17:26 UTC
Permalink
Post by Liyun Yu
Anyone used the OWC (other world computing) SSD (6G) for RAID Array?
Any suggestions/recommendations on SSD based RAID on the market
or drives good for RAID ?
OWC's SSDs are SandForce-based. OCZ (not OWC) is probably the biggest
SandForce-based SSD manufacturer.
Post by Liyun Yu
I did not access the SSD market for a while. Yesterday one of our colleagues
sent me a link at OWC which indicated that their drives has writing speed
above 500MB
SandForce uses compression and de-duplication. That 500MB/s number is
not really 500MB/s because with compression and de-duplication it
doesn't actually have to right all that data. When tested with random
uncompressible data SandForce drives perform about the same as Intel
drives.

SSDs are great for random reads and writes, like database servers,
they aren't really worth it for sequential writes.
Post by Liyun Yu
Tom's hardware site which was more than 8 months old.
AnandTech has the best consumer SSD articles:
http://www.anandtech.com/tag/storage
Post by Liyun Yu
one thing to watch out for with SSDs, most of them do not have good behavior
in the face of power failures.
The Intel G1 and G2 drives lost data on power failure. The G3s (SSD
320) have a supercap and they do not lose data. There was a firmware
bug the killed the drives on power failure, referred to as the
"Addresses Bad Context 13x Error", that was fixed in August.

The SandForce drives without super capacitors or discrete capacitors
lose data on power failure. SMART Modular makes some SandForce drives
with capacitors.
Post by Liyun Yu
I've heard this as well... but in my experience, SSDs fail like disks,
not like tires, which is to say, the failure pattern seems fairly random,
and more correlated to age than to use.
In my experience consumer SSDs (e.g., Intel and SandForce) fail at a
much higher rate than enterprise HDDs.

Purely anecdotal evidence here, not a scientific study. Given 100
servers with 8 SSDs each and 100 servers with 8 15k SAS disks each,
the difference in failure rate is extreme. For the first 3 years the
SAS HDDs would fail at a rate of 1 every 6 months. After 3 years,
maybe 1 every 2 months. Throughout the life of the SSDs they would
fail at a rate of 1 a week, sometimes the failures clustered, often
leading to multiple failures a week.

The most common failure was sectors becoming unreadable or unwritable.
The firmware didn't seem to correct the bad sectors. Sometimes the
firmware would completely fail, and the drive would be lost. Overall
the firmware of consumer SSDs does not seem to be as mature as
enterprise spinning disks, leading to odd failure scenarios.

To sum it up, I'd trust consumer SSDs in my laptop, I'd trust them for
L2ARC, I wouldn't trust them as primary storage for critical data like
an Oracle database.

-Anton
Luke S. Crawford
2011-09-19 00:47:31 UTC
Permalink
Post by Anton Cohen
To sum it up, I'd trust consumer SSDs in my laptop, I'd trust them for
L2ARC, I wouldn't trust them as primary storage for critical data like
an Oracle database.
But not the ZIL, right? or is there a way to mirror SSDs for a ZIL?
Doug Hughes
2011-09-19 02:08:08 UTC
Permalink
Post by Luke S. Crawford
Post by Anton Cohen
To sum it up, I'd trust consumer SSDs in my laptop, I'd trust them for
L2ARC, I wouldn't trust them as primary storage for critical data like
an Oracle database.
But not the ZIL, right? or is there a way to mirror SSDs for a ZIL?
has been since update 6. zpool add zpool1 log mirror <dev1> <dev2>

I typically carve off a slice on my SSD for ZIL that's about 8G each and
then let the rest be used for OS.
Luke S. Crawford
2011-09-19 07:34:55 UTC
Permalink
Post by Doug Hughes
Post by Luke S. Crawford
Post by Anton Cohen
To sum it up, I'd trust consumer SSDs in my laptop, I'd trust them for
L2ARC, I wouldn't trust them as primary storage for critical data like
an Oracle database.
But not the ZIL, right? or is there a way to mirror SSDs for a ZIL?
has been since update 6. zpool add zpool1 log mirror <dev1> <dev2>
I typically carve off a slice on my SSD for ZIL that's about 8G each and
then let the rest be used for OS.
Sweet! thanks for the info. Why only 8G? I mean, my systems are pretty
much being constantly written to in a very random manner; unrolling that
into sequential writes, which I believe is what a ZIL does, would be
a huge win.
Edward Ned Harvey
2011-09-19 11:50:46 UTC
Permalink
On Behalf Of Luke S. Crawford
Sweet! thanks for the info. Why only 8G? I mean, my systems are pretty
much being constantly written to in a very random manner; unrolling that
into sequential writes, which I believe is what a ZIL does, would be
a huge win.
Here's how it works:
When some application issues a write to disk, by default it's an async
write. The OS is permitted to buffer it in memory. Depending on which
version of ZFS we're talking about, it will buffer up to 30 sec, or up to 5
sec, assembling all the async writes into a single transaction group (TXG)
which is a performance enhancement. Thanks to copy on write and the
requisite block remapping that's necessary to support it, all these small
writes that would otherwise be randomly spread across the disk on an older
filesystem (ext3) are now serialized into a sequential block.

When some application issues fsync(), or opens a file in SYNC write mode,
the OS cannot do the above. It has to immediately store the write on
nonvolatile storage. What the log device does is this... It allows these
small SYNC writes ("small" is configurable, I think by default smaller than
32k) to be quickly written to nonvolatile storage, after which they can be
buffered and optimized along with all the async writes. Also, since the log
device is only being used for sync writes, there is no device competition -
If some other async read or write is currently in progress on the main pool,
no need to wait for it. So the log device greatly improves performance of
small random sync writes.

Since recent versions of ZFS, log device removal is supported, so if a log
device fails, then the TXG is immediately flushed from RAM, and the pool
degrades back to "normal" level of performance without log device. During
that window, a power failure or kernel panic would result in data loss...
As long as you're willing/able to risk this window of a second or two
vulnerability... Then there is no need for ZIL mirroring. It is logical,
but I've never seen anybody test it, that ZIL mirroring makes your system
slower than running without ZIL mirroring. (but still much faster than
running without dedicated log device)

BTW, there are many situations where completely disabling ZIL is also
acceptable. Even if you're doing databases and so forth, there is
absolutely nothing faster than disabling ZIL, and there is no risk of data
corruption. There is only a risk of up to 30 sec data loss in the event of
power failure or kernel panic etc. Depending on the role of your machine
interacting with other machines, this may be acceptable. And usually is
acceptable if your server is a file server.

(And finally, the actual answer to the question...)

Supposing your log device is on a 6Gbit bus, and a TXG will be flushed at
maximum every 30 seconds, and you're absolutely hammering your system with
small sync writes (perhaps you run a transactional database or something.)
This is a worst case. Then the most you could possibly ever write to your
log before TXG flush would be 180Gbit = 22 GB. In practice, this is
probably unattainable, and certainly unusual if it's even possible. In
practice, 8G is more than enough for 99% of what people care about. In
fact, the DDRDrive is designed specifically for this purpose, and it's only
4G. Even 4G is sufficient for 99%
Doug Hughes
2011-09-19 13:03:02 UTC
Permalink
Post by Luke S. Crawford
Post by Luke S. Crawford
Post by Anton Cohen
To sum it up, I'd trust consumer SSDs in my laptop, I'd trust them for
L2ARC, I wouldn't trust them as primary storage for critical data like
an Oracle database.
But not the ZIL, right? or is there a way to mirror SSDs for a ZIL?
has been since update 6. zpool add zpool1 log mirror<dev1> <dev2>
I typically carve off a slice on my SSD for ZIL that's about 8G each and
then let the rest be used for OS.
Sweet! thanks for the info. Why only 8G? I mean, my systems are pretty
much being constantly written to in a very random manner; unrolling that
into sequential writes, which I believe is what a ZIL does, would be
a huge win.
_______________________________________________
The ZIL rarely uses even 2G even under very heavy load. 8G is about 4X
what I calculated would be our maximum usage. There are a bunch of
documents out there on sizing the ZIL, including this one:
http://www.nexenta.com/corp/content/view/274/119/

"* The maximum size of a log device should be approximately 1/2 the size
of physical memory because that is the maximum amount of potential
in-play data that can be stored. For example, if a system has 16 Gbytes
of physical memory, consider a maximum log device size of 8 Gbytes.

* For a target throughput of X MB/sec and given that ZFS pushes
transaction groups every 5 seconds (and have 2 outstanding), we also
expect the ZIL to not grow beyond X MB/sec * 10 sec. So to service
100MB/sec of synchronous writes, 1 GBytes of log device should be
sufficient. "

The biggest WIN for ZIL is NFS workloads with lots of small files and
lots of metadata operations (think rm -rf a directory tree with
thousands of files). NFS consistency semantics (close on write) demand
that ZFS not wait for it's 5 second transaction group window and flush
right away. So, for a big rm -rf it's doing a flush on every file that
is removed (directory metadata). That's painful. Build workloads,
compiles, fast temporary datasets and the like are where the ZIL on
flash is a good choice.

One final thought, if you are doing large sequential accesses you are
better off not having a dedicated ZIL. The channel to the 1 or 2 drives
will be your bottleneck and having ZIL striped over your pool of disks
is better. The 5 second flush will also be your friend. You should tune
your filesystem latency to throughput mode using the log_bias tunable.
Edward Ned Harvey
2011-09-19 13:40:22 UTC
Permalink
On Behalf Of Doug Hughes
The biggest WIN for ZIL is NFS workloads with lots of small files and
This is also the biggest win, for disabling the ZIL completely. ;-)

Also, on your NFS clients, add the "async" NFS mount option.

Very few NFS servers are servicing clients which cannot accept the risk
introduced by disabling ZIL. If your server crashes, some stale NFS file
handles linger on some clients. I've never seen them cause harm (beyond
whatever you normally expect whenever your NFS server crashes) but
eventually you'll reboot your client or dismount/remount, and at that time,
they will disappear.
Doug Hughes
2011-09-19 15:17:02 UTC
Permalink
Post by Edward Ned Harvey
On Behalf Of Doug Hughes
The biggest WIN for ZIL is NFS workloads with lots of small files and
This is also the biggest win, for disabling the ZIL completely. ;-)
Also, on your NFS clients, add the "async" NFS mount option.
Very few NFS servers are servicing clients which cannot accept the risk
introduced by disabling ZIL. If your server crashes, some stale NFS file
handles linger on some clients. I've never seen them cause harm (beyond
whatever you normally expect whenever your NFS server crashes) but
eventually you'll reboot your client or dismount/remount, and at that time,
they will disappear.
I have tended to subscribe to the more cautionary advice still
espoused by the ZFS Evil Tuning guide:
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Disabling_the_ZIL_.28Don.27t.29

In particular:
"Caution: Disabling the ZIL on an NFS server can lead to client side
corruption. the ZFS pool integrity itself is not compromised by this
tuning."
Edward Ned Harvey
2011-09-20 03:00:43 UTC
Permalink
Post by Doug Hughes
I have tended to subscribe to the more cautionary advice still
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Di
sabling_the_ZIL_.28Don.27t.29
"Caution: Disabling the ZIL on an NFS server can lead to client side
corruption. the ZFS pool integrity itself is not compromised by this
tuning."
It's true. In fact, I wrote a significant chunk of that guide. ;-) But
I'm always a fan of understanding the limits of the system, and choosing
your configurations and settings based on actual limitations and knowledge,
rather than following a rule of thumb. A rule of thumb is always the "safe"
option you tell people who don't want to consider any alternative but the
conservative "Nobody ever got fired for choosing ______"

It goes like this: At all times, async writes are being buffered in ram, no
matter what. It is only the sync writes that hit the ZIL. So in the event
of a system ungraceful crash, while writes are in progress, no matter what
you are going to lose your async writes. Take it for granted, if you're in
the middle of writes when the server crashes, you're going to end up with
some data loss... (Unless you make 100% of your writes sync, which is not
typical for a NFS server.)

No matter what, you will need to remount your clients in order to expect
they're in a consistent state with the server. And no matter what, your
clients will be consistent after remounting.

If the data on your NFS server represents peoples' work... In every place
I've worked, I feel confident saying that users can accept something like a
15 second loss of their work in the event of a dramatic server crash.
Because the server crashes so rarely. Also acknowledged, something like 15
sec of async data will be lost unconditionally... The only protection you
can possibly offer is something like 15 sec of sync writes too.

In the above scenario, if you were willing to accept 15 sec of async data
loss, and you don't even know which applications are performing async vs
sync operations... Then you're also willing to accept the 15 sec of sync
mode data loss. You can gain performance year round by disabling the ZIL.

So why does the caution comment exist? Because inevitably you're going to
have somebody using something like a database running over a NFS link. Or
let's imagine a mail server spooling to the NFS server. Somebody is
processing credit card transactions. You don't want to acknowledge the
transaction has been finalized, and 15 seconds later forget about it.
Anything where the NFS server is being used as the backend data store for
some transactional service which cannot be interrupted and relies on sync
mode writes for consistency. Any application which cannot itself survive an
ungraceful crash. Then you need to honor the sync. These are not the types
of data that are typically stored on NFS servers. But could be, and surely
is somewhere. So the cautionary comment remains intact...

So you always have to weigh your own options and make your own decisions...
On the one hand, increase the risk of data loss from 15sec async to 15sec
async&sync... Weigh that risk against the potential gain on the other
hand... Something like twice the performance 364.5 days a year, depending
on how you measure your performance... Apply your own personal beliefs in
probabilities and expectations and value of your work load. Everyone has
their own opinions and philosophies. There is no single right answer.

Ultimately, the safer option to tell people who are uncomfortable making
that judgement is... Don't disable the ZIL. But I personally do almost
systematically, disable the ZIL. Because I like having optimum performance
and minimal cost, and in essentially all situations that I encounter, the
performance benefit outweighs the actual delta of risk.
Anton Cohen
2011-09-19 02:55:19 UTC
Permalink
But not the ZIL, right?   or is there a way to mirror SSDs for a ZIL?
You can mirror the ZIL, you actually can't mirror the L2ARC.

The reason I said L2ARC and not ZIL for MLC consumer SSDs has to do
with size, performance, and price.

The maximum size for the ZIL is about 1/2 RAM, effective size is even
smaller, it needs enough to store about 10 seconds worth of writes. It
also needs to have very fast writes. Something like the STEC ZeusRAM
(a flash-backed RAM drive) or a small SLC SSD would be ideal.

The L2ARC is an extension of RAM for read cache. Spending $3000 on a
100GB SLC SSD isn't a great deal, when you can spend $3000 on 128GB of
RAM that would be faster and do the same thing. On the other hand
spending $500 on 300GB of MLC SSD sounds pretty good.

-Anton
Luke S. Crawford
2011-09-19 07:33:03 UTC
Permalink
Post by Anton Cohen
The reason I said L2ARC and not ZIL for MLC consumer SSDs has to do
with size, performance, and price.
The maximum size for the ZIL is about 1/2 RAM, effective size is even
smaller, it needs enough to store about 10 seconds worth of writes. It
also needs to have very fast writes. Something like the STEC ZeusRAM
(a flash-backed RAM drive) or a small SLC SSD would be ideal.
Interesting product; probably out of my price range, though. It used
to be they would sell products that took normal DIMMS, battery back them,
and would output a normal sata connector. The last time I saw such a
device it only supported non-ecc ddr2, and thus was completely useless
to me.
Post by Anton Cohen
The L2ARC is an extension of RAM for read cache. Spending $3000 on a
100GB SLC SSD isn't a great deal, when you can spend $3000 on 128GB of
RAM that would be faster and do the same thing. On the other hand
spending $500 on 300GB of MLC SSD sounds pretty good.
yeah, makes sense.
Edward Ned Harvey
2011-09-19 11:25:42 UTC
Permalink
On Behalf Of Luke S. Crawford
It used
to be they would sell products that took normal DIMMS, battery back them,
and would output a normal sata connector. The last time I saw such a
device it only supported non-ecc ddr2, and thus was completely useless
to me.
Bear in mind, the magnetic surface of a disk platter doesn't do ECC either.
But in response to this, they use FEC chips on the circuit board of the hard
drive, and encode more bits onto the magnetic surface. Whenever a checksum
error occurs, the disk controller will silently retry (indicates a soft
error, a 1-rotation performance hit) but as long as there's no error on the
2nd or 3rd or 4th attempt, the hardware silently hides this condition from
the OS. You might get SMART indicating failure predicted.

So what if your device only takes non-ECC ram? Does it have a FEC chip on
the controller board? Does your OS/FS do any checksumming? Have any
redundant copies with which to restore/recreate data after a checksum error
occurs?

There are so many levels of checksumming and error detection/correction. I
agree zero isn't enough, but ... Is it really zero in the case mentioned?
'Luke S. Crawford'
2011-09-20 03:18:13 UTC
Permalink
Post by Edward Ned Harvey
Bear in mind, the magnetic surface of a disk platter doesn't do ECC either.
But in response to this, they use FEC chips on the circuit board of the hard
drive, and encode more bits onto the magnetic surface. Whenever a checksum
error occurs, the disk controller will silently retry (indicates a soft
error, a 1-rotation performance hit) but as long as there's no error on the
2nd or 3rd or 4th attempt, the hardware silently hides this condition from
the OS. You might get SMART indicating failure predicted.
I still don't trust a single drive. Mirror them.

Of course, if bits on a hard drive flipped as easily as they do in ram
(and bits on a hard drive do occasionally silently flip, just not nearly
as often as ram) then mirroring would not be enough; I'd need something
like zfs to checksum and mirror.
Post by Edward Ned Harvey
So what if your device only takes non-ECC ram? Does it have a FEC chip on
the controller board? Does your OS/FS do any checksumming? Have any
redundant copies with which to restore/recreate data after a checksum error
occurs?
So you are suggesting that maybe the device does the sort of error
correction that hard drives do on their platters on non-ECC ram?

I soppose that is possible... but I find it fairly unlikely. this was
not an 'Enterprise' product, and really, I don't know of any motherboards
that do that sort of error correction on non-ecc ram, so yeah, I'd bet
money that it had no protection against ram errors. .
Post by Edward Ned Harvey
There are so many levels of checksumming and error detection/correction. I
agree zero isn't enough, but ... Is it really zero in the case mentioned?
Eh, I wouldn't trust it. bit flips in ram happen; go poke through
your EDAC logs on any sufficently large population of servers and you will
see a fairly large number of detected/corrected errors. Some servers only
have a few, and maybe are ignorable? (I am on the 'replace after the first
error' side here, but if there was only one or two, I can respect those
on the other side) but most servers will have a lot of corrected errors.

I mean, yeah, I soppose you could implement some sort of error correction
outside of the dimm? but why would you? I think you'd have a difficult
time doing it both safely and more efficently than commodity ECC ram.
--
Luke S. Crawford
http://prgmr.com/xen/ - Hosting for the technically adept
http://nostarch.com/xen.htm - We don't assume you are stupid.
Edward Ned Harvey
2011-09-20 12:00:53 UTC
Permalink
On Behalf Of 'Luke S. Crawford'
Post by Edward Ned Harvey
Bear in mind, the magnetic surface of a disk platter doesn't do ECC either.
But in response to this, they use FEC chips on the circuit board of the hard
drive, and encode more bits onto the magnetic surface. Whenever a
checksum
Post by Edward Ned Harvey
error occurs, the disk controller will silently retry (indicates a soft
error, a 1-rotation performance hit) but as long as there's no error on the
2nd or 3rd or 4th attempt, the hardware silently hides this condition from
the OS. You might get SMART indicating failure predicted.
I still don't trust a single drive. Mirror them.
I don't quite get where you're coming from here. There are two separate
issues - mirror-vs-not-mirror of the ZIL, which isn't mentioned above. And
somebody said the lack of ECC in the DRAM-based sata devices made them not
an option, which is what I'm discussing above.

As for mirroring the ZIL: Distrust for a single drive has some truth in it.
If you have a disk failure (including data error) on your unmirrored ZIL
device, which coincides with a system ungraceful crash, then the data on
that device would be lost. The assumption, if you don't mirror your ZIL, is
that the probability of these multiple failures coinciding is small enough
to be comparable to the probability of multiple disk failures coinciding.
So you are suggesting that maybe the device does the sort of error
correction that hard drives do on their platters on non-ECC ram?
Just suggesting a possibility. We know this is the case for HDD's and
SSD's. Why not also DRAM based drives?
I soppose that is possible... but I find it fairly unlikely. this was
not an 'Enterprise' product,
I mean all HDD's and SSD's. Not just enterprise ones. So this DRAM device
not being enterprise level... Maybe significant, maybe not.
I mean, yeah, I soppose you could implement some sort of error correction
outside of the dimm? but why would you? I think you'd have a difficult
time doing it both safely and more efficently than commodity ECC ram.
Take it for granted, because of HDD/SSD, yes it's definitely possible, and
common, for error detection/correction to happen on-chip, outside of the
storage media, very close to the storage media. Now you raise an excellent
question: In the DRAM SATA device, which design would be more attractive to
the manufacturer? use ECC ram, or use FEC outside of the ram, as they do
for other types of devices (HDD/SSD)?

I can say this: ECC ram uses 9 bits instead of 8. This is not a simple
parity bit (because parity is only useful for detecting, not correcting
errors). But the payload is 8/9. Also, the actual error detection happens
off-chip, not inside the DIMM. That's why your motherboard needs to have
support for ECC ram in order to use it, and ECC ram is slightly slower than
non-ECC. Also, the volume of sales for non-ECC ram is much higher, so
non-ECC ram is significantly cheaper (not just a ratio of 8:9).

So take it for granted, the non-ECC ram is significantly cheaper, and even
if you're using ECC, then the error detection is going to happen outside the
DIMM anyway.

In the case of ECC for your system memory, you need to operate on 32bits or
64bits depending on your architecture. But in the case of your DRAM SATA
device, it's either 512 bytes, or 4K bytes (4096 or 32768 bits). Basically
1000 times larger word. This allows you to use a standard SATA FEC chip,
which has a much better payload than 8/9. Say, for example, the FEC is
using LDPC, which operates at or near the theoretical limit of the channel,
it means you're (a) operating at optimal speed, (b) operating at minimal
cost, (c) operating at maximum reliability.

So yes, there is motivation to do the error detection outside of ECC, using
FEC on non-ECC ram on the DRAM SATA device. I cannot say, of course,
whether or not they're doing any of this. I can only say that yes, it's
reasonable, yes it's common in other products, and yes there is motivation
to do so. Don't make any assumptions about it not being done at all just
because it's non-ECC ram.
Doug Hughes
2011-09-20 13:17:02 UTC
Permalink
Post by Edward Ned Harvey
As for mirroring the ZIL: Distrust for a single drive has some truth in it.
If you have a disk failure (including data error) on your unmirrored ZIL
device, which coincides with a system ungraceful crash, then the data on
that device would be lost. The assumption, if you don't mirror your ZIL, is
that the probability of these multiple failures coinciding is small enough
to be comparable to the probability of multiple disk failures coinciding.
It used to be that if you didn't mirror the ZIL and had it go bad that you
had a major failure condition and you would have to recreate the spool from
scratch. Granted, this was a pretty silly limitation and was fixed in 2010.
Now, if you lose the zip it will fail back to the spool disks. Still, since
I'm already mirroring the OS, I get ZIL mirroring essentially free. The ZIL
is only used for writing unless there is a crash.
'Luke S. Crawford'
2011-09-20 19:02:26 UTC
Permalink
Post by Edward Ned Harvey
I can say this: ECC ram uses 9 bits instead of 8. This is not a simple
parity bit (because parity is only useful for detecting, not correcting
errors). But the payload is 8/9. Also, the actual error detection happens
off-chip, not inside the DIMM. That's why your motherboard needs to have
support for ECC ram in order to use it, and ECC ram is slightly slower than
non-ECC. Also, the volume of sales for non-ECC ram is much higher, so
non-ECC ram is significantly cheaper (not just a ratio of 8:9).
So take it for granted, the non-ECC ram is significantly cheaper, and even
if you're using ECC, then the error detection is going to happen outside the
DIMM anyway.
This varies a /lot/ by density. I can get an 8GiB stick of Reg. ecc ddr3 for
$96. for unbuffered ecc? (and I think the speed hit is on the buffering,
not the ecc. unbuffered ecc, I believe, is about as fast as non-ecc) you
are talking 4x that much.

I don't know anyone that sells 8GiB non-ecc ddr3 at any price..

Of course, if 4GiB modules are enough for what you are trying to do,
those are all great points. Just saying, I'm currently in a situation
where buying registered ram is cheaper, not more expensive than the
unbuffered stuff, and non-ecc isn't even possible.

(I would be happy to be wrong; the LGA1155 xeons look really nice;
the only problem is that they only support unbuffered ram, which means
getting the 32GiB ram in there I need would require the ruinously expensive
8gib unbuffered modules)
Steve Drees
2011-09-20 21:17:36 UTC
Permalink
Post by Edward Ned Harvey
On Behalf Of Luke S. Crawford
It used
to be they would sell products that took normal DIMMS, battery back them,
and would output a normal sata connector. The last time I saw such a
device it only supported non-ecc ddr2, and thus was completely useless
to me.
Bear in mind, the magnetic surface of a disk platter doesn't do ECC either.
I think the OP was concerned because he likely had a drawer full of lefotver
ECC DIMMs from system pulls.
Doug Hughes
2011-09-19 12:51:05 UTC
Permalink
Post by Luke S. Crawford
Post by Anton Cohen
The reason I said L2ARC and not ZIL for MLC consumer SSDs has to do
with size, performance, and price.
The maximum size for the ZIL is about 1/2 RAM, effective size is even
smaller, it needs enough to store about 10 seconds worth of writes. It
also needs to have very fast writes. Something like the STEC ZeusRAM
(a flash-backed RAM drive) or a small SLC SSD would be ideal.
Interesting product; probably out of my price range, though. It used
to be they would sell products that took normal DIMMS, battery back them,
and would output a normal sata connector. The last time I saw such a
device it only supported non-ecc ddr2, and thus was completely useless
to me.
I'll point out again, that if you just want to put ZIL on fast devices
to, say, make NFS accesses for small files and meta-data fast, the Intel
320 Series SSD is a good choice. In our benchmarks, it is as fast as the
older, more expensive X-25E SLC drive, plus it has the capacitor on
board to guarantee data integrity in power failure. I doubt you'd see
more than incremental improvements with the STEC at a very steep price
premium.
Zack Williams
2011-09-19 23:23:28 UTC
Permalink
I'll point out again, that if you just want to put ZIL on fast devices to, say, make NFS accesses for small files and meta-data fast, the Intel 320 Series SSD is a good choice. In our benchmarks, it is as fast as the older, more expensive X-25E SLC drive, plus it has the capacitor on board to guarantee data integrity in power failure. I doubt you'd see more than incremental improvements with the STEC at a very steep price premium.
Intel sells a 20GB SLC flash drive, the 311, which goes for around $115 online:

http://www.intel.com/content/www/us/en/solid-state-drives/solid-state-drives-311-series.html.html

Haven't tried one yet, but the SLC flash would likely be ideal in terms of longevity with a write-heavy ZIL workload.

- Zack
--
Zack Williams - Artisan Computer Services - 520.867.8701
***@artisancomputer.com http://www.artisancomputer.com
Apple Certified System Administrator, Apple Consultants Network
Microsoft Certified Professional, Small Business Specialist
Sun Certified System Administrator
Linux Professional Institute LPIC-1
Dan Schlitt
2012-01-23 16:51:36 UTC
Permalink
A suspicious file has appeared on my Ubuntu linux box. It is in a strage
place for a file that is written to - /usr/include/openssl/aes1.h. It
contains plain text information that shouldn't be kept.

I have looked diligently to find where it is coming from without finding
anything.

It is definitely connected in some way to ssh (which I have removed and
reinstalled to no effect.) If the file is not world writable ssh crashes
after connecting and logging in to the remote end. It doesn't mind the
read permissions being removed.

Does anyone recognize the malware or configuration that this belongs to.

Any help would be appreciated.

/dan

--
Dan Schlitt
***@theworld.com
Theo Van Dinter
2012-01-23 17:07:11 UTC
Permalink
My first thought would be a hardlink or symlink, not necessarily malware.
If you search either for another file w/ the same inode # or a symlink that
points at that path (in any of the variety of ways to do so), do you find
anything?
Alternatively, you could do an strace on ssh and see what files it's
accessing. It sounds like it'd be something opened w/ read-write or
write-only privs, but if the process is crashing, it should be pretty
obvious.
Post by Dan Schlitt
A suspicious file has appeared on my Ubuntu linux box. It is in a strage
place for a file that is written to - /usr/include/openssl/aes1.h. It
contains plain text information that shouldn't be kept.
I have looked diligently to find where it is coming from without finding
anything.
It is definitely connected in some way to ssh (which I have removed and
reinstalled to no effect.) If the file is not world writable ssh crashes
after connecting and logging in to the remote end. It doesn't mind the
read permissions being removed.
Does anyone recognize the malware or configuration that this belongs to.
Any help would be appreciated.
Yves Dorfsman
2012-01-23 17:19:33 UTC
Permalink
Post by Dan Schlitt
A suspicious file has appeared on my Ubuntu linux box. It is in a strage
place for a file that is written to - /usr/include/openssl/aes1.h. It
contains plain text information that shouldn't be kept.
Have you done a recent update on that box?
Post by Dan Schlitt
It is definitely connected in some way to ssh (which I have removed and
reinstalled to no effect.) If the file is not world writable ssh crashes
after connecting and logging in to the remote end. It doesn't mind the
read permissions being removed.
That file is definitely on my current Ubuntu box, but it is only writable by
root. I can't think of any reason why access to an include file would be
necessary for a binary. It definitely does sound like the version of ssh you
are running has been compromised, and that they are just using an existing
file for a different purpose in order not to raise any suspicion.
--
Yves. http://www.SollerS.ca/
http://ipv6.SollerS.ca
http://blog.zioup.org/
Chris Palmer
2012-01-23 17:43:15 UTC
Permalink
That file is definitely on my current Ubuntu box, but it is only writable by root. I can't think of any reason why access to an include file would be necessary for a binary. It definitely does sound like the version of ssh you are running has been compromised, and that they are just using an existing file for a different purpose in order not to raise any suspicion.
Yeah I've seen hacks like that before, though long ago.

the standard SSH rootkits I had fun cleaning up usually also compromised various utilities that would let you otherwise see them in action, like lsof and ps and whatnot. Check those to see if their change times are different, and all the same as each other, and try reinstalling them.


---
Asst Coach, Lexington Debate: http://www.lexdebate.org
Tournament Tab Software: http://www.tabroom.com
Brad Hudson
2012-01-23 17:44:21 UTC
Permalink
And then he noticed the note about ssh dying if the file is not
writable ...

Considering the ssh crash I would agree that ssh could be compromised.
The best thing to do would be to re-install all ssh/ssl related
packages. Before doing this make sure you clear your cache, validate
your apt sources (to make sure they are the dist sources) and force
apt to re-download/reinstall.

After the re-install it would be a good time to change all passwords,
just in case.

Brad
Dan;
It is most likely from a dev package. I have an aes.h on my
system that comes from libssl-dev. I have no aes1.h.
/usr/include/openssl/aes.h
Is the file an actual header file? If so it should start with
something like the following, with a lot of defines and includes
in the actual code.
/* crypto/aes/aes.h -*- mode:C; c-file-style: "eay" -*- */ /*
====================================================================
* Copyright (c) 1998-2002 The OpenSSL Project. All rights reserved.
... #ifndef HEADER_AES_H #define HEADER_AES_H
#include <openssl/opensslconf.h>
#ifdef OPENSSL_NO_AES #error AES is disabled. #endif
What version of Ubuntu/openssl are you currently running? The .h
files would only be used at compile time, if you are worried about
it there is no reason you could not either remove the file or the
-dev package it belongs to (unless you want to compile something
with ssl support).
Brad
Post by Dan Schlitt
A suspicious file has appeared on my Ubuntu linux box. It is in
a strage place for a file that is written to -
/usr/include/openssl/aes1.h. It contains plain text information
that shouldn't be kept.
I have looked diligently to find where it is coming from without
finding anything.
It is definitely connected in some way to ssh (which I have
removed and reinstalled to no effect.) If the file is not world
writable ssh crashes after connecting and logging in to the
remote end. It doesn't mind the read permissions being removed.
Does anyone recognize the malware or configuration that this
belongs to.
Any help would be appreciated.
/dan
_______________________________________________ Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech This list
provided by the League of Professional System Administrators
http://lopsa.org/
- --
Brad Hudson
SA Team Lead
The Pythian Group - love your data
Desk: 613-565-8696 x202
IM: pythianhudson
Dan Schlitt
2012-01-23 18:18:12 UTC
Permalink
It is definitely not a header file.

I did reinstall the ssh but the number of files that were removed when
removing openssl was a bit daunting so I didn't do it.

I don't intentionally have the openssl development package installed

The installed versions are openssl 0.9.8k-7ubuntu8.6 and ssh
1:5.3p1-3ubuntu7 if that is useful information.

I jsut removed an ssl development package and all the .h files in that
directory went away but the file in question remained. I just checked
again and the file must be world writeable of the ssh client crashes.

It certainly looks like an attempt to hide that file.

/dan

--
Dan Schlitt
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
And then he noticed the note about ssh dying if the file is not
writable ...
Considering the ssh crash I would agree that ssh could be compromised.
The best thing to do would be to re-install all ssh/ssl related
packages. Before doing this make sure you clear your cache, validate
your apt sources (to make sure they are the dist sources) and force
apt to re-download/reinstall.
After the re-install it would be a good time to change all passwords,
just in case.
Brad
Dan;
It is most likely from a dev package. I have an aes.h on my
system that comes from libssl-dev. I have no aes1.h.
/usr/include/openssl/aes.h
Is the file an actual header file? If so it should start with
something like the following, with a lot of defines and includes
in the actual code.
/* crypto/aes/aes.h -*- mode:C; c-file-style: "eay" -*- */ /*
====================================================================
* Copyright (c) 1998-2002 The OpenSSL Project. All rights reserved.
... #ifndef HEADER_AES_H #define HEADER_AES_H
#include <openssl/opensslconf.h>
#ifdef OPENSSL_NO_AES #error AES is disabled. #endif
What version of Ubuntu/openssl are you currently running? The .h
files would only be used at compile time, if you are worried about
it there is no reason you could not either remove the file or the
-dev package it belongs to (unless you want to compile something
with ssl support).
Brad
Post by Dan Schlitt
A suspicious file has appeared on my Ubuntu linux box. It is in
a strage place for a file that is written to -
/usr/include/openssl/aes1.h. It contains plain text information
that shouldn't be kept.
I have looked diligently to find where it is coming from without
finding anything.
It is definitely connected in some way to ssh (which I have
removed and reinstalled to no effect.) If the file is not world
writable ssh crashes after connecting and logging in to the
remote end. It doesn't mind the read permissions being removed.
Does anyone recognize the malware or configuration that this belongs to.
Any help would be appreciated.
/dan
_______________________________________________ Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech This list
provided by the League of Professional System Administrators
http://lopsa.org/
- --
Brad Hudson
SA Team Lead
The Pythian Group - love your data
Desk: 613-565-8696 x202
IM: pythianhudson
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk8dnHUACgkQQ6JZA6y/BxmgnwCfbKMzuCRiYMppev0BeDnIeNDp
NQQAmwXPJ7+WlOCbD1W2lw7mcDcSD0q8
=BITl
-----END PGP SIGNATURE-----
Steven Kurylo
2012-01-23 18:28:05 UTC
Permalink
Post by Dan Schlitt
It is definitely not a header file.
I did reinstall the ssh but the number of files that were removed when
removing openssl was a bit daunting so I didn't do it.
You'll want to skip dependency checking

sudo dpkg --force-all --remove openssl
sudo apt-get install openssl
Post by Dan Schlitt
It certainly looks like an attempt to hide that file.
A reinstall of the entire OS is best. Though it might be good to keep
an image of the hard drive around, so you can dissect it later.
Dan Schlitt
2012-01-24 20:08:43 UTC
Permalink
Thanks for all the suggestions. Reinstalling didn't seem to change
anything, To take care of the file I just pointed it to /devnull.

Being rusty I forgot one thing that would provide information. I did
strings on the executables. Sure enought there was the file name in both
ssh and sshd. I think that I got a copy of the current source. Couldn't
find the file in the source code. It might be in some header file that is
in ssl. I may look at that when I find time.

/dan

--
Dan Schlitt
***@theworld.com
Tom Perrine
2012-01-25 19:07:05 UTC
Permalink
Post by Dan Schlitt
Thanks for all the suggestions. Reinstalling didn't seem to change
anything, To take care of the file I just pointed it to /devnull.
Did you do a full system re-install, or just the ssh package?

If you haven't done the full system re-install, you really need to go
that route.

--tep
Dan Foster
2012-01-26 10:29:43 UTC
Permalink
Post by Tom Perrine
Post by Dan Schlitt
Thanks for all the suggestions. Reinstalling didn't seem to change
anything, To take care of the file I just pointed it to /devnull.
Did you do a full system re-install, or just the ssh package?
If you haven't done the full system re-install, you really need to go
that route.
Hey Dan,

Tom speaks the absolute truth.
Lynda
2012-01-26 13:37:18 UTC
Permalink
Post by Dan Foster
Post by Tom Perrine
Post by Dan Schlitt
Thanks for all the suggestions. Reinstalling didn't seem to change
anything, To take care of the file I just pointed it to /devnull.
Did you do a full system re-install, or just the ssh package?
If you haven't done the full system re-install, you really need to go
that route.
Hey Dan,
Tom speaks the absolute truth.
Brandon Allbery
2012-01-26 20:27:03 UTC
Permalink
Post by Tom Perrine
Post by Dan Schlitt
Thanks for all the suggestions. Reinstalling didn't seem to change
Post by Dan Schlitt
anything, To take care of the file I just pointed it to /devnull.
If you haven't done the full system re-install, you really need to go
Post by Dan Schlitt
that route.
d***@lang.hm
2012-01-26 21:02:12 UTC
Permalink
Post by Tom Perrine
Post by Dan Schlitt
Thanks for all the suggestions. Reinstalling didn't seem to change
Post by Dan Schlitt
anything, To take care of the file I just pointed it to /devnull.
If you haven't done the full system re-install, you really need to go
Post by Dan Schlitt
that route.
Simon Lyall
2012-01-23 23:04:23 UTC
Permalink
The "debsums" program while check the checksums of packages with what they
are supposed to be.

To be safest you should boot to something trusted rather than the main
hard-drive since the rootkit could mask the changed files.

http://manpages.ubuntu.com/manpages/lucid/man1/debsums.1.html
Post by Dan Schlitt
It is definitely not a header file.
I did reinstall the ssh but the number of files that were removed when
removing openssl was a bit daunting so I didn't do it.
I don't intentionally have the openssl development package installed
The installed versions are openssl 0.9.8k-7ubuntu8.6 and ssh
1:5.3p1-3ubuntu7 if that is useful information.
I jsut removed an ssl development package and all the .h files in that
directory went away but the file in question remained. I just checked
again and the file must be world writeable of the ssh client crashes.
It certainly looks like an attempt to hide that file.
/dan
--
Dan Schlitt
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
And then he noticed the note about ssh dying if the file is not
writable ...
Considering the ssh crash I would agree that ssh could be compromised.
The best thing to do would be to re-install all ssh/ssl related
packages. Before doing this make sure you clear your cache, validate
your apt sources (to make sure they are the dist sources) and force
apt to re-download/reinstall.
After the re-install it would be a good time to change all passwords,
just in case.
Brad
Dan;
It is most likely from a dev package. I have an aes.h on my
system that comes from libssl-dev. I have no aes1.h.
/usr/include/openssl/aes.h
Is the file an actual header file? If so it should start with
something like the following, with a lot of defines and includes
in the actual code.
/* crypto/aes/aes.h -*- mode:C; c-file-style: "eay" -*- */ /*
====================================================================
* Copyright (c) 1998-2002 The OpenSSL Project. All rights reserved.
... #ifndef HEADER_AES_H #define HEADER_AES_H
#include <openssl/opensslconf.h>
#ifdef OPENSSL_NO_AES #error AES is disabled. #endif
What version of Ubuntu/openssl are you currently running? The .h
files would only be used at compile time, if you are worried about
it there is no reason you could not either remove the file or the
-dev package it belongs to (unless you want to compile something
with ssl support).
Brad
Post by Dan Schlitt
A suspicious file has appeared on my Ubuntu linux box. It is in
a strage place for a file that is written to -
/usr/include/openssl/aes1.h. It contains plain text information
that shouldn't be kept.
I have looked diligently to find where it is coming from without
finding anything.
It is definitely connected in some way to ssh (which I have
removed and reinstalled to no effect.) If the file is not world
writable ssh crashes after connecting and logging in to the
remote end. It doesn't mind the read permissions being removed.
Does anyone recognize the malware or configuration that this belongs to.
Any help would be appreciated.
/dan
_______________________________________________ Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech This list
provided by the League of Professional System Administrators
http://lopsa.org/
- --
Brad Hudson
SA Team Lead
The Pythian Group - love your data
Desk: 613-565-8696 x202
IM: pythianhudson
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk8dnHUACgkQQ6JZA6y/BxmgnwCfbKMzuCRiYMppev0BeDnIeNDp
NQQAmwXPJ7+WlOCbD1W2lw7mcDcSD0q8
=BITl
-----END PGP SIGNATURE-----
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
--
Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/
"To stay awake all night adds a day to your life" - Stilgar | eMT.
Paul Heinlein
2012-01-23 17:48:59 UTC
Permalink
Post by Brad Hudson
And then he noticed the note about ssh dying if the file is not
writable ...
Considering the ssh crash I would agree that ssh could be
compromised. The best thing to do would be to re-install all ssh/ssl
related packages.
I suspect the best thing would be to assume the worst: all passwords
and private keys on that box are compromised.

I'd get it off-line, clone it for forensics, and repave it as soon as
possible. Then talk to anyone with user credentials on that account
and make provision for changing passwords and re-generating any
private keys.
--
Paul Heinlein <> ***@madboa.com <> http://www.madboa.com/
Brad Hudson
2012-01-23 17:35:46 UTC
Permalink
Dan;

It is most likely from a dev package. I have an aes.h on my system
that comes from libssl-dev. I have no aes1.h.

$ dpkg-query -S /usr/include/openssl/aes.h
libssl-dev: /usr/include/openssl/aes.h

Is the file an actual header file? If so it should start with
something like the following, with a lot of defines and includes in
the actual code.

/* crypto/aes/aes.h -*- mode:C; c-file-style: "eay" -*- */
/* ====================================================================
* Copyright (c) 1998-2002 The OpenSSL Project. All rights reserved.
...
#ifndef HEADER_AES_H
#define HEADER_AES_H

#include <openssl/opensslconf.h>

#ifdef OPENSSL_NO_AES
#error AES is disabled.
#endif

What version of Ubuntu/openssl are you currently running? The .h
files would only be used at compile time, if you are worried about it
there is no reason you could not either remove the file or the -dev
package it belongs to (unless you want to compile something with ssl
support).

Brad
Post by Dan Schlitt
A suspicious file has appeared on my Ubuntu linux box. It is in a
strage place for a file that is written to -
/usr/include/openssl/aes1.h. It contains plain text information
that shouldn't be kept.
I have looked diligently to find where it is coming from without
finding anything.
It is definitely connected in some way to ssh (which I have removed
and reinstalled to no effect.) If the file is not world writable
ssh crashes after connecting and logging in to the remote end. It
doesn't mind the read permissions being removed.
Does anyone recognize the malware or configuration that this
belongs to.
Any help would be appreciated.
/dan
_______________________________________________ Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech This list
provided by the League of Professional System Administrators
http://lopsa.org/
- --
Brad Hudson
SA Team Lead
The Pythian Group - love your data
Desk: 613-565-8696 x202
IM: pythianhudson
David E. Smith
2011-09-16 14:37:46 UTC
Permalink
Were it not for the DD-WRT/OpenWRT requirement, my usual go-to
suggestion for "cheap home router" is a Mikrotik Routerboard 750GL.
Five gigabit ports, sixty dollars, and I think it meets all your other
requirements. It runs "RouterOS," which is basically Linux with a
proprietary GUI.

(It's a MIPS big-endian processor, so you might even be able to flash
DD-WRT onto it; I know they support a lot of hardware, but I've never
looked into that particular combination.)

David Smith
MVN.net
unix_fan
2011-09-16 16:47:10 UTC
Permalink
My ancient SMC7008VBR 8-port wired (not wireless) home router died as a result of hurricane Irene. 
[snip]
I'd like to find something that can run DD-WRT or OpenWRT.
    It needs to be low power (run for a couple of hours on a dedicated so/ho UPS);
    I don't need or want Wi-Fi in this application;
    I'd like it to support 2 or more LANs, jointly NATed onto one WAN port;
    I'd like it to be inexpensive (about $100 or less, but I'd go as high as $200 if necessary);
    and I'd like to be able to support IPv4-NAT/NPT, IPv6, and simple firewall rules.
There's an easy way to turn any of the wi-fi models into a wired only model: disable the wi-fi

At that point a standard wrt54gl should meet your needs, no?
if 8 ports was part of the unwritten nice to haves, dd-wrt works on this dlink
http://www.amazon.com/D-Link-DIR-632-Wireless-N-8-Port-Router/dp/B003XMAD22
d***@lang.hm
2011-09-16 18:04:39 UTC
Permalink
the linksys 3700 has two ports on the main processor, but one of those
ports is connected to a 4-port switch that is able to do vlans so you
could partition it up and use all 5 ports independantly. It does have both
2.4 and 5G wireless, but you can turn those off.

David Lang

On Fri, 16 Sep
My ancient SMC7008VBR 8-port wired (not wireless) home router died as a
result of hurricane Irene. I've temporarily replaced it with it's
predecessor, a 4-port SMC7004VBR, but that was underpowered and on it's last
legs 8 years ago when I replaced it. So I'm in the market for a new wired
(not wireless) router.
I'd like to find something that can run DD-WRT or OpenWRT.
It needs to be low power (run for a couple of hours on a dedicated
so/ho UPS);
I don't need or want Wi-Fi in this application;
I'd like it to support 2 or more LANs, jointly NATed onto one WAN port;
I'd like it to be inexpensive (about $100 or less, but I'd go as high
as $200 if necessary);
and I'd like to be able to support IPv4-NAT/NPT, IPv6, and simple
firewall rules.
I've googled "wired router" and searched the OpenWRT and DD-WRT websites for
hardware recommendations, but everything I find is oriented to WiFi, and has
at most 2 ethernet ports (one WAN and one LAN, plus lots of groovy WiFi that
I don't want or need)
As a last resort, I'm willing to build my own from parts (e.g. a
routerstation Pro or ALIX board and OpenWRT) but a commercial solution would
be preferable.
Anybody got a suggestion?
Thanks!
Rick
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Rick Thomas
2011-09-16 20:17:57 UTC
Permalink
Hi David,

That's interesting and helpful. Do you have a URL for technical details on the 3700?

I'm not familiar with vlans (I know what they are and I've read the IEEE specs on how they are implemented, but I don't have any experience with the nitty-gritty of actually using them.) Please correct me if I'm getting this all wrong...

I'd configure the one on-chip port to have all 4 vlans. The on-chip port is connected to the uplink of the 4-port switch and delivers all traffic to it. The switch would then be configured to send each vlan in/out a separate port, in the process stripping out the vlan protocol overhead bits.

That would seem to give 4 ports, not 5. But that's just a nit -- or it could be evidence of a basic misunderstanding on my part; correction are welcome. Have I got the rest of it right?

Thanks!

Rick
the linksys 3700 has two ports on the main processor, but one of those ports is connected to a 4-port switch that is able to do vlans so you could partition it up and use all 5 ports independantly. It does have both 2.4 and 5G wireless, but you can turn those off.
David Lang
d***@lang.hm
2011-09-16 22:16:59 UTC
Permalink
I haven't actually used the different ports yet, but here is my
understanding of it

all of the interfaces are gig

you have eth1 that is the port labeled 'wan'

you then have eth0 that is connected to the 4-port switch, you can
configure the switch so that port 1 is eth0.3 port 2 is eth0.2, etc. If
you look at stats for eth1 on the system you will see the combined
traffic, but after doing the initial setup, you can use the eth1.1
interfaces exactly the same way you would use the eth1, eth2, etc
interfaces on a normal system.

the switch is only able to use vlans 0-15 (4 bits of vlan addressing)

sorry, this is a netgear, not a linksys

http://wiki.openwrt.org/toh/netgear/wndr3700

they run ~$130 street price

the bufferbloat project picked this model to use for their testing and
they have a openwrt derivitive tweaked for it called cerowrt

http://www.bufferbloat.net/projects/cerowrt

David Lang
Post by Rick Thomas
Hi David,
That's interesting and helpful. Do you have a URL for technical details on the 3700?
I'm not familiar with vlans (I know what they are and I've read the IEEE specs on how they are implemented, but I don't have any experience with the nitty-gritty of actually using them.) Please correct me if I'm getting this all wrong...
I'd configure the one on-chip port to have all 4 vlans. The on-chip port is connected to the uplink of the 4-port switch and delivers all traffic to it. The switch would then be configured to send each vlan in/out a separate port, in the process stripping out the vlan protocol overhead bits.
That would seem to give 4 ports, not 5. But that's just a nit -- or it could be evidence of a basic misunderstanding on my part; correction are welcome. Have I got the rest of it right?
Thanks!
Rick
the linksys 3700 has two ports on the main processor, but one of those ports is connected to a 4-port switch that is able to do vlans so you could partition it up and use all 5 ports independantly. It does have both 2.4 and 5G wireless, but you can turn those off.
David Lang
Rick Thomas
2011-09-16 19:20:48 UTC
Permalink
Hi Chris,

Two questions...

*) Are the ethernet ports separately addressable? Or are they just (internally) one port and a 4-port switch? I.e. can I (using dd-wrt) assign separate IP addresses to each one?

*) Suppose I decide to go the whole hog and re-do my wireless configuration (reluctance to do this is the main reason behind my request for a wired router). I see that there is also a "Wireless-N" version, called "WRT160NL", that is also "Linux compatible". Does that mean it can also run dd-wrt?

Thanks!

Rick
I've got one of these running dd-wrt. Yes, it's wireless, but there
are 4 ethernet ports. If I need more, I just use a switch.
http://www.amazon.com/Cisco-Linksys-WRT54GL-Wireless-G-Broadband-Compatible/dp/B000BTL0OA
I don't know about ipv6, but if you have the right version of dd-wrt,
it's supposed to work.
My ancient SMC7008VBR 8-port wired (not wireless) home router died as a
result of hurricane Irene. I've temporarily replaced it with it's
predecessor, a 4-port SMC7004VBR, but that was underpowered and on it's last
legs 8 years ago when I replaced it. So I'm in the market for a new wired
(not wireless) router.
I'd like to find something that can run DD-WRT or OpenWRT.
It needs to be low power (run for a couple of hours on a dedicated
so/ho UPS);
I don't need or want Wi-Fi in this application;
I'd like it to support 2 or more LANs, jointly NATed onto one WAN port;
I'd like it to be inexpensive (about $100 or less, but I'd go as high
as $200 if necessary);
and I'd like to be able to support IPv4-NAT/NPT, IPv6, and simple
firewall rules.
I've googled "wired router" and searched the OpenWRT and DD-WRT websites for
hardware recommendations, but everything I find is oriented to WiFi, and has
at most 2 ethernet ports (one WAN and one LAN, plus lots of groovy WiFi that
I don't want or need)
As a last resort, I'm willing to build my own from parts (e.g. a
routerstation Pro or ALIX board and OpenWRT) but a commercial solution would
be preferable.
Anybody got a suggestion?
Thanks!
Rick
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Brandon Allbery
2011-09-16 22:26:50 UTC
Permalink
Post by Rick Thomas
*) Suppose I decide to go the whole hog and re-do my wireless configuration
(reluctance to do this is the main reason behind my request for a wired
router). I see that there is also a "Wireless-N" version, called
"WRT160NL", that is also "Linux compatible". Does that mean it can also run
dd-wrt?
Chances are both of them will run some form of DD-WRT; the "Linux
compatible" models have additional memory, more or less specifically to
enable them to run *full* DD-WRT instead of the cut-down version that will
fit in the memory of the VxWorks-based models.
--
brandon s allbery ***@gmail.com
wandering unix systems administrator (available) (412) 475-9364 vm/sms
Chris Palmer
2011-09-16 22:55:33 UTC
Permalink
Post by Rick Thomas
*) Are the ethernet ports separately addressable? Or are they just
(internally) one port and a 4-port switch? I.e. can I (using dd-wrt) assign
separate IP addresses to each one?
Yes, dd-wrt lets you do this on most routers; I had three external IPs coming into one
box at one point.
Post by Rick Thomas
*) Suppose I decide to go the whole hog and re-do my wireless configuration
(reluctance to do this is the main reason behind my request for a wired
router). I see that there is also a "Wireless-N" version, called "WRT160NL",
that is also "Linux compatible". Does that mean it can also run dd-wrt?
This can be a challenge with Linksys routers; they will sometimes change the
chipset and will not change the model number; there'll be a "version number"
that bumps but isn't discernable on the outside of the box. Ordering online
can be a real roulette wheel.

Asus makes routers that I use at work that are certified to support dd-wrt;
they're reasonably priced and perform all right, though they are a touch less
power efficient and don't last as long.

-C
Rick Thomas
2011-09-17 04:53:19 UTC
Permalink
Post by Chris Palmer
Asus makes routers that I use at work that are certified to support dd-wrt;
they're reasonably priced and perform all right, though they are a touch less
power efficient and don't last as long.
Hmmm... Interesting! Do you have a model number? Or better yet, a
URL for the tech specs and a price?

Thanks!

Rick
Anton Cohen
2011-09-17 06:22:05 UTC
Permalink
Hmmm... Interesting!  Do you have a model number?  Or better yet, a URL for
the tech specs and a price?
http://www.newegg.com/Product/Product.aspx?Item=N82E16833320038

Search Newegg for "dd-wrt" for a few others.

Most, if not all of Buffalo's routers ship with DD-WRT by default:
http://www.buffalotech.com/products/wireless/

-Anton
Chris Palmer
2011-09-17 17:14:24 UTC
Permalink
Post by Chris Palmer
Asus makes routers that I use at work that are certified to support dd-wrt;
they're reasonably priced and perform all right, though they are a touch less
power efficient and don't last as long.
Hmmm... Interesting! Do you have a model number? Or better yet, a URL for the tech specs and a price?
Yeah, I've bought this one for simple lightweight home use:

http://www.newegg.com/Product/Product.aspx?Item=N82E16833320023

and this overpowered one for the office to run some OpenVPN end to end links with:

http://www.newegg.com/Product/Product.aspx?Item=N82E16833320038

The failure rates on the cheaper one have been a tad high, but we have more of them; the more expensive ones haven't failed yet (3+ years running).

-Chris
Chris Reisor
2011-09-19 19:15:20 UTC
Permalink
Post by Rick Thomas
Hi Chris,
Two questions...
*) Are the ethernet ports separately addressable?  Or are they just (internally) one port and a 4-port switch?  I.e. can I (using dd-wrt) assign separate IP addresses to each one?
I'm not sure. I've always used static DHCP (with dnsmasq) myself.
Post by Rick Thomas
*) Suppose I decide to go the whole hog and re-do my wireless configuration (reluctance to do this is the main reason behind my request for a wired router).  I see that there is also a "Wireless-N" version, called "WRT160NL", that is also "Linux compatible". Does that mean it can also run dd-wrt?
According to their Supported Devices page, yes:
http://www.dd-wrt.com/wiki/index.php/Supported_Devices
Post by Rick Thomas
Thanks!
Rick
I've got one of these running dd-wrt.  Yes, it's wireless, but there
are 4 ethernet ports.  If I need more, I just use a switch.
http://www.amazon.com/Cisco-Linksys-WRT54GL-Wireless-G-Broadband-Compatible/dp/B000BTL0OA
I don't know about ipv6, but if you have the right version of dd-wrt,
it's supposed to work.
My ancient SMC7008VBR 8-port wired (not wireless) home router died as a
result of hurricane Irene.  I've temporarily replaced it with it's
predecessor, a 4-port SMC7004VBR, but that was underpowered and on it's last
legs 8 years ago when I replaced it. So I'm in the market for a new wired
(not wireless) router.
I'd like to find something that can run DD-WRT or OpenWRT.
       It needs to be low power (run for a couple of hours on a dedicated
so/ho UPS);
       I don't need or want Wi-Fi in this application;
       I'd like it to support 2 or more LANs, jointly NATed onto one WAN
port;
       I'd like it to be inexpensive (about $100 or less, but I'd go as high
as $200 if necessary);
       and I'd like to be able to support IPv4-NAT/NPT, IPv6, and simple
firewall rules.
I've googled "wired router" and searched the OpenWRT and DD-WRT websites for
hardware recommendations, but everything I find is oriented to WiFi, and has
at most 2 ethernet ports (one WAN and one LAN, plus lots of groovy WiFi that
I don't want or need)
As a last resort, I'm willing to build my own from parts (e.g. a
routerstation Pro or ALIX board and OpenWRT) but a commercial solution would
be preferable.
Anybody got a suggestion?
Thanks!
Rick
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Rick Thomas
2011-09-16 20:02:05 UTC
Permalink
You make a good point, Duncan. It's probably the simplest and cheapest solution, all things considered.

My resistance to replacing a wired router with a wireless router is just me being a careful system administrator. I have a working system that uses a wired router and a couple of access-points for Wi-Fi. The space I'm in has foil insulated walls and floors that make it impossible to reach all rooms with one radio. If I put in a wireless router, I may get unexpected interactions with the APs that will make the whole thing harder to diagnose and configure.

If I were starting from scratch, I would certainly go with a modern commercial wireless router, add APs as necessary, slave them to the router, and be done with it. I may wind up going ahead and doing just that (using the existing APs as much as I can) but I'm trying to avoid a long period of down-time while I get it configured.

Thanks for the information about the buffalo wzr-g300-nh. That will be helpful if I go this route. Do you have a URL for technical details on that one?

Rick
My ancient SMC7008VBR 8-port wired (not wireless) home router died as a
result of hurricane Irene. I've temporarily replaced it with it's
predecessor, a 4-port SMC7004VBR, but that was underpowered and on it's
last legs 8 years ago when I replaced it. So I'm in the market for a new
wired (not wireless) router.
I hear that you don't want wireless, but I wonder about your objection?
Wouldn't it be easier/cheaper to just get a well supported modern
device, put openwrt on it and turn off the radio?
I'm happy with my buffalo wzr-g300-nh. more memory and flash than most.
--
Duncan Hutty
Michael Neuffer
2011-09-17 07:29:31 UTC
Permalink
Post by d***@lang.hm
all of the chip based memory devices that are able to maintain the data
without power have a limited number of write cycles.
In the case of modern flash drives this is in the millions of cycles on
an individual block.
older flash drives wear out faster than newer ones, MLC flash drives
wear out faster than SLC drives (but MLC stores two bits per cell nstead
of one and has further price advantages due to volume production so it's
_far_ cheaper)
[...]
erasing data on a flash drive is very slow, SSDs get around this by
writing a new copy of the data to a new location and erasing the old
copy sometime later. All modern drives contain more flash than what you
pay for so that they have a large pool of 'blank' space that they can
write to (this also helps spread out the writes across more flash so
that you are less likely to hit the 1m write limit on any one spot). If
Sorry to barge in on this, but the write limit on modern MLC cells is
much lower. For modern flash chips the MTTF is somewhere in the range of
5000 - 25000 write cycles on a cell. The rest is owed to intelligent
write allocation/distribution which spreads the writes out over all the
unallocated area of the disk. This is also the reason why it is so
important to use the TRIM function which informs the disk about unused
areas. Allowing the disk to "pre-erase" the space is just a welcome side
effect.

Older flash chips with larger structures have a much better MTTF.


Cheers
Mike
Loading...