Discussion:
[lopsa-tech] tools for managing and composing shell scripts?
(too old to reply)
Miles Fidelman
2012-03-20 01:28:51 UTC
Permalink
Hi Folks,

New list member here, seems like this might be the place to pose this
question:

I'm getting ready to rebuild a small cluster - mostly used for
development, but likely to move into a production role, and thinking
about being prepared if we need to scale. As a result, I'm looking at
way to move from a rather ad hoc management approach - configuration
notes and task checklists in word documents, spreadsheet of IP addresses
and DNS records, bunches of shell scripts and cron jobs - to something a
bit more organized and scaleable.

My first thought is to start with a simple database for config. info,
and moving scripts under configuration control; maybe adding an
orchestration tool like rundeck (though call me old fashioned enough to
think about writing some wrapper code in tcl).

Anyway, when I asked for tool suggestions on one of the devops lists,
where I kind of figured was the place to ask about tools, about all I
got were religious pontifications about puppet or chef being the "one
true way" - which seems a bit out of line with my experience of
real-world operations, both in the small, and in the large (been around
both, though more as an architect for large systems than as an operator).

Which leads me to pose two questions:

1. What is the state of the practice right now? How much is sys
administration a world of traditional approaches, vs. how much have the
new generation of devops tools caught on outside the core of folks
who've drunk the koolaid? (I'm really trying to get calibrated in
reality here.)

2. My more specific question re. what are people using to manage
accumulated scripts and semi-manual procedures?

I'm looking for something along the following lines - an orchestration
tool that would:

- provide for editing/managing/versioning scripts (script = anything
that can be invoked at the command line)
- a library of control functions for use within scripts
- invoking scripts, combinations of scripts, pipelines of scripts (in
the Unix sense of pipes) - locally, remotely, across multiple machines
- remote script execution via ssh, rather than some kind of agent
- providing a simple database for keeping track of variables used by
scripts (e.g., IP addresses, DNS records for use by a provisioning
script) - that can be accessed from scripts
- accessing the above via cli, RESTful API, GUI
- cross-platform
- (nice-to-have) minimal environmental requirements (i.e., a step above
the unix shell, say the gnu buildtool suite rather than the JVM + a mass
of libraries, or the ruby ecosystem -- a self-configuring basic
environment would be nice, like perl+cpan)

I'm sure we've each cobbled something together along these lines - be it
through manual processes, standard installs, or something fancier. I'm
kind of wondering what different people have done, what pieces fit
together better than others, what's available that's more integrated but
that doesn't create a monolithic environment.

Rundeck seems to do most (all?) of this, running on Java. It looks
nice, but it's also really new.

tcl/tk provides a lot of this - but without the organizing functions
(tcl/tk + git, or maybe a tcl/tk IDE perhaps?)

What have people found that works?

Thanks very much,

Miles Fidelman
--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra
d***@lang.hm
2012-03-20 01:56:52 UTC
Permalink
Post by Miles Fidelman
New list member here, seems like this might be the place to pose this
I'm getting ready to rebuild a small cluster - mostly used for development,
but likely to move into a production role, and thinking about being prepared
if we need to scale. As a result, I'm looking at way to move from a rather
ad hoc management approach - configuration notes and task checklists in word
documents, spreadsheet of IP addresses and DNS records, bunches of shell
scripts and cron jobs - to something a bit more organized and scaleable.
My first thought is to start with a simple database for config. info, and
moving scripts under configuration control; maybe adding an orchestration
tool like rundeck (though call me old fashioned enough to think about writing
some wrapper code in tcl).
Anyway, when I asked for tool suggestions on one of the devops lists, where I
kind of figured was the place to ask about tools, about all I got were
religious pontifications about puppet or chef being the "one true way" -
which seems a bit out of line with my experience of real-world operations,
both in the small, and in the large (been around both, though more as an
architect for large systems than as an operator).
1. What is the state of the practice right now? How much is sys
administration a world of traditional approaches, vs. how much have the new
generation of devops tools caught on outside the core of folks who've drunk
the koolaid? (I'm really trying to get calibrated in reality here.)
2. My more specific question re. what are people using to manage
accumulated scripts and semi-manual procedures?
You name it, it's probably in use in a very large environment.

at $work we have hundreds of production systems serving over ten million
users (real users, not freebe logins) where the majority of the real
configuation is logging into the server and running vi on a file. A couple
of years ago I got a report from 302 production systems as to what
packages were installed, I did a count of the number of packages on each
system and came up with 149 different package counts among the systems
(and this was with ~100 of the systems being identical)

we have other places where there are hundreds of systems configured
entirely through automated tools where we are utterly confident that all
of the systems are running identical software, even though these are
organized into >100 different farms, each with different configurations,
connected to different network (in-house tools, dating back 10+ years in
an environment that cfengine was not a valid choice for), and we have
other areas where the company has spent millions on commercial automation
tools.

And this is all in one company.

I think the one constant is that nobody is completely satisfied with what
they have and knows things that they would like to do to improve things
(either that, or they are completely ignorant of automation and are
wishing that things were easier to configure, which boils down to the same
thing)

I would say that the key is not to try and do everything at once. I would
go through a progression along the lines of the following.

1. start off by managing installed software and patches

2. move from there to making sure configs were synced between appropriate
boxes (especially primary and backup where you have them)

3. then get user management under control

4. then work to publish/track your config files from a central master
(even if they are manually edited on that central master)

5. then work to eliminate the manual editing.

David Lang
Miles Fidelman
2012-03-20 13:09:14 UTC
Permalink
Thanks to all who responded. A few follow-up questions and comments if
you don't mind. For brevity's sake, I've consolidated stuff from
Post by d***@lang.hm
You name it, it's probably in use in a very large environment.
at $work we have hundreds of production systems serving over ten
million users (real users, not freebe logins) where the majority of
the real configuation is logging into the server and running vi on a
file. A couple of years ago I got a report from 302 production systems
as to what packages were installed, I did a count of the number of
packages on each system and came up with 149 different package counts
among the systems (and this was with ~100 of the systems being identical)
we have other places where there are hundreds of systems configured
entirely through automated tools where we are utterly confident that
all of the systems are running identical software, even though these
are organized into >100 different farms, each with different
configurations, connected to different network (in-house tools, dating
back 10+ years in an environment that cfengine was not a valid choice
for), and we have other areas where the company has spent millions on
commercial automation tools.
Sort of what I expect is typical.
Post by d***@lang.hm
I think the one constant is that nobody is completely satisfied with
what they have and knows things that they would like to do to improve
things (either that, or they are completely ignorant of automation and
are wishing that things were easier to configure, which boils down to
the same thing)
:-)
Post by d***@lang.hm
I would say that the key is not to try and do everything at once. I
would go through a progression along the lines of the following.
1. start off by managing installed software and patches
2. move from there to making sure configs were synced between
appropriate boxes (especially primary and backup where you have them)
3. then get user management under control
4. then work to publish/track your config files from a central master
(even if they are manually edited on that central master)
5. then work to eliminate the manual editing.
Pretty much ok on 1-3, 4 and 5 are what I'm focusing on right now.
Right now, most of my build and config takes the form of fairly simple
checklists, containing steps like:
- apt-get install <foo> (or: download; untar; ./configure, ./make,
./make test, ./make install)
- look up some config info (e.g., pull an unused IP address off a checklist)
- add a domain record to a zone file
- edit one or more config files
- ,/init.d/<foo> start

other than editing config config files, pretty much everything consists
of one-line shell commands - easy enough to stick into a bash script;
and I guess a lot of the configuration could be done by adding sed
commands to the script

thinking more and more that rundeck (or something like it) would make it
easy to manage and execute those scripts (the other thing that looks
pretty interesting is a little project called sm-framework
(https://sm.beginrescueend.com/presentations/SM-Framework-Presentation.pdf)
- essentially adds some glue to the zsh and treats scripts like plug-ins
Post by d***@lang.hm
I've used both cfengine and Puppet over the years, and have met people
who have been successful with Chef. What has worked for others in
their specific environment may (or may not) work well for yours.
What I keep coming back to is that they seem like overkill for what I'm
doing right now - and carry with them the need to:
- install a layer of infrastructure
- work in yet another language (bash and sed do pretty much all I need,
why have to wade through recipes written in Ruby or Python or whatever)?
- do I really need to deal with lots of recipes to ultimately execute a
few install and config. statements (who really needs the overhead to
execute "apt-get install" or "./configure; ./make; ./make test, ./make
install") - at least for the stuff I'm managing now, the details are in
site-specific configurations, and it seems like a lot of getting that
right with Chef (or whatever) involves wading into and understanding how
the individual recipes work (or writing them)

Right now, I'm managing a small high-availability cluster (essentially a
mini-cloud) with one VM for email and list handling, another for
managing backups, and several for development. Where I expect that
something like chef, or puppet, or whatever will be helpful is in about
6 months when we might be worrying about putting some code into
production, and (fingers crossed) scaling services across multiple VMs,
some of which might be running on AWS or elsewhere in the cloud. Right
now, though, it's more about documenting and streamlining management of
the small cluster - partially to simplify handoff to someone else to
administer, partially to simplify disaster recovery or hardware migration.
Post by d***@lang.hm
That's what offers the excuse for the "one true way" arguments.
(http://thuktun.wordpress.com/toms-rules-for-happy-sysadmins-and-users/)
:-)
Post by d***@lang.hm
I'd start with this list
(http://en.wikipedia.org/wiki/Comparison_of_open_source_configuration_management_software
Thanks for the pointer. The list seems to have grown a lot, since I
last looked (maybe 2 years ago) - looks like a pretty good starting point.
Post by d***@lang.hm
1. Support for the way you manage your site today. Think about whether
or not you have a CMDB or inventory system, or you want your CM tool
to provide it. Think about what OSes you need to support: If you need
Windows (and Mac, and UNIX) support your choices may be more limited.
Think about the things you like about the way you manage now (and want
to keep) and the things you want to change. See which tools are more
like the way you want things to be in the future, not which ones
exactly match what you're doing now.
This is actually what I see as most useful. I find that the most time
consuming part of setting up a VM involves creating logical volumes,
assigning IP addresses (both for external access and for pairing Disk
Mirrors and failover instances - DRBD and pacemaker are a pain in this
regard), generating DNS records, and such. Right now this is all
manual - a spreadsheet and editing config files. A simple CMDB, and
hooks that can be accessed from shell scripts would go a LONG way to
simplify my life.

Suggestions along these lines would be most welcomed.

The rest of your points are well taken - thanks!
Post by d***@lang.hm
Excessive Kool-Aid is most likely clouding my vision, but the way I
see it everyone is a) drinking the Kool-Aid, b) wishing they had some
tasty Kool-Aid to drink, or c) too drunk on stale beer to know
Kool-Aid tastes way better.
Personally, I prefer the electric variety :-)
Post by d***@lang.hm
To sum up your requirements: You want a cross-platform tool with
a library of functions, that executes scripts on remote hosts over a
secure protocol, and has access to variables in a database. You want
to access the tool from CLI, GUI, and REST API. You want the scripts
to be version controlled.
What you described /is/ Configuration Management, like Puppet and
Chef. Really, what you asked for exists, except they communicate over
HTTPS instead of SSH, and are written in Ruby instead of Tcl.
The dependency requirements aren't that bad, at least for Puppet and
CFEngine. They can be satisfied from the standard repos on
Debian-based hosts and standard repos + EPEL on Red Hat-based hosts.
Well, I'm more specifically interested in getting bash scripts under
control, not scripts written in Ruby or Python. Not really looking to
either rewrite existing setup stuff, or to buy off on having to wade
through chef cookbooks (or the puppet or whatever equivalent) to find,
understand, and configure scripts to suit my environment.

Hence, I've been zeroing in more on the combination of an orchestration
tool (e.g., rundeck), a version control tool (probably git), and some
kind of simple CMDB to track IP addresses, DNS records, and such ---
been doing a lot of my own evaluation, but wondering what others have
done in this regard, and/or suggestions for specific combinations of
orchestration+version_control+cmdb tools that are easy to wire together.
(Easy integration w/ FAI, for initial system build would be a plus).
Post by d***@lang.hm
(Wait, I just re-read your email... did you say Perl + CPAN is nice?
That's like saying getting punched repeatedly in the face, while your
hands are tied behind your back is nice.)
My major production system right now is Sympa (a really nice open source
email list manager, built by a consortium of French universities - sort
of the step up from Mailman if you manage a lot of lists with common
users). It's written in perl (say what you will about perl, but it's a
pretty effective language for text processing, a respectable design
choice, particularly for the time, and it's very well written perl).
The thing that impresses me is how nicely the package builds from source
(I've yet to find the Debian packaging to be either up-to-date or
stable). Build is a simple, ./configure; make; make install - and in
the process it uses CPAN to install/update all kinds of perl modules -
all nice, clean, and automatic - and it all fits in the context of the
gnu build tools.

So, yes, in line with automating installations and configuration -
Perl+CPAN is a pretty good example of something that "just works."

In the case of Sympa, the install automation also creates a user account
for sympa to run under, creates and populates a mysql database, installs
mail aliases, creates/updates configuration files, populates rc.d, and
starts the sevice - pretty much everything on autopilot. All with
"./configure; make; make install" - there are something like three
questions to answer during the install, notably an admin
username/password - and those can be pre-seeded, or provided on the
command line.

All in all, a very impressive example of automatic installation and
configuration. Possibly the best I've seen.

Thanks Again to All,

Miles
--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra
d***@lang.hm
2012-03-20 13:21:55 UTC
Permalink
Post by Tom Perrine
I've used both cfengine and Puppet over the years, and have met people
who have been successful with Chef. What has worked for others in
their specific environment may (or may not) work well for yours.
What I keep coming back to is that they seem like overkill for what I'm doing
- install a layer of infrastructure
- work in yet another language (bash and sed do pretty much all I need, why
have to wade through recipes written in Ruby or Python or whatever)?
- do I really need to deal with lots of recipes to ultimately execute a few
install and config. statements (who really needs the overhead to execute
"apt-get install" or "./configure; ./make; ./make test, ./make install") - at
least for the stuff I'm managing now, the details are in site-specific
configurations, and it seems like a lot of getting that right with Chef (or
whatever) involves wading into and understanding how the individual recipes
work (or writing them)
Right now, I'm managing a small high-availability cluster (essentially a
mini-cloud) with one VM for email and list handling, another for managing
backups, and several for development. Where I expect that something like
chef, or puppet, or whatever will be helpful is in about 6 months when we
might be worrying about putting some code into production, and (fingers
crossed) scaling services across multiple VMs, some of which might be running
on AWS or elsewhere in the cloud. Right now, though, it's more about
documenting and streamlining management of the small cluster - partially to
simplify handoff to someone else to administer, partially to simplify
disaster recovery or hardware migration.
If you know you wnt to automate a bigger cluster later, take the time in
development with your small cluster to learn the tool and work through the
configuration.

David Lang
Miles Fidelman
2012-03-20 13:36:38 UTC
Permalink
Post by d***@lang.hm
Post by Miles Fidelman
Post by Tom Perrine
I've used both cfengine and Puppet over the years, and have met people
who have been successful with Chef. What has worked for others in
their specific environment may (or may not) work well for yours.
What I keep coming back to is that they seem like overkill for what
- install a layer of infrastructure
- work in yet another language (bash and sed do pretty much all I
need, why have to wade through recipes written in Ruby or Python or
whatever)?
- do I really need to deal with lots of recipes to ultimately execute
a few install and config. statements (who really needs the overhead
to execute "apt-get install" or "./configure; ./make; ./make test,
./make install") - at least for the stuff I'm managing now, the
details are in site-specific configurations, and it seems like a lot
of getting that right with Chef (or whatever) involves wading into
and understanding how the individual recipes work (or writing them)
Right now, I'm managing a small high-availability cluster
(essentially a mini-cloud) with one VM for email and list handling,
another for managing backups, and several for development. Where I
expect that something like chef, or puppet, or whatever will be
helpful is in about 6 months when we might be worrying about putting
some code into production, and (fingers crossed) scaling services
across multiple VMs, some of which might be running on AWS or
elsewhere in the cloud. Right now, though, it's more about
documenting and streamlining management of the small cluster -
partially to simplify handoff to someone else to administer,
partially to simplify disaster recovery or hardware migration.
If you know you wnt to automate a bigger cluster later, take the time
in development with your small cluster to learn the tool and work
through the configuration.
Well sure, but I see these as different use cases, with different
problems, and probably different tools.

I NEVER expect to replicate our development or administrative
environments to any serious scale - these are going to remain fairly
customized, and in the case of development there are likely to be lots
of "throw away" VMs as we experiment with different plumbing and
experimentation. Cleaning up processes yes (to simplify disaster
recovery), lots of automation doesn't seem to apply. If I were starting
from scratch, I'd probably start with Ganetti (a really nice tool from
Google that they use to manage small, internal clusters, used for
administrative applications - though it has some subtleties that get in
the way of doing a completely automated failover). As I'm not starting
from scratch, I just want to get existing shell scripts under
configuration control, and automate manual steps where that's easy
(mostly re. IP address and DNS record managment when creating new
virtual machines).

For production, on the other hand, if we're fortunate, the focus will be
on deploying lots of cookie cutter nodes - each with a fairly
constrained environment - and that's where the combination of an
orchestration tool + a configuration engine seems to offer a lot of
leverage. And, yes, I expect we'll want to wire together a small
cluster first, and get the tooling right before scaling out. It's just
that I don't expect this cluster, or associated procedures, to look
anything like our inward facing stuff.

Thanks,

Miles
--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra
d***@lang.hm
2012-03-20 23:55:27 UTC
Permalink
Post by d***@lang.hm
Post by Miles Fidelman
Post by Tom Perrine
I've used both cfengine and Puppet over the years, and have met people
who have been successful with Chef. What has worked for others in
their specific environment may (or may not) work well for yours.
What I keep coming back to is that they seem like overkill for what I'm
- install a layer of infrastructure
- work in yet another language (bash and sed do pretty much all I need,
why have to wade through recipes written in Ruby or Python or whatever)?
- do I really need to deal with lots of recipes to ultimately execute a
few install and config. statements (who really needs the overhead to
execute "apt-get install" or "./configure; ./make; ./make test, ./make
install") - at least for the stuff I'm managing now, the details are in
site-specific configurations, and it seems like a lot of getting that
right with Chef (or whatever) involves wading into and understanding how
the individual recipes work (or writing them)
Right now, I'm managing a small high-availability cluster (essentially a
mini-cloud) with one VM for email and list handling, another for managing
backups, and several for development. Where I expect that something like
chef, or puppet, or whatever will be helpful is in about 6 months when we
might be worrying about putting some code into production, and (fingers
crossed) scaling services across multiple VMs, some of which might be
running on AWS or elsewhere in the cloud. Right now, though, it's more
about documenting and streamlining management of the small cluster -
partially to simplify handoff to someone else to administer, partially to
simplify disaster recovery or hardware migration.
If you know you wnt to automate a bigger cluster later, take the time in
development with your small cluster to learn the tool and work through the
configuration.
Well sure, but I see these as different use cases, with different problems,
and probably different tools.
I NEVER expect to replicate our development or administrative environments to
any serious scale - these are going to remain fairly customized, and in the
case of development there are likely to be lots of "throw away" VMs as we
experiment with different plumbing and experimentation. Cleaning up
processes yes (to simplify disaster recovery), lots of automation doesn't
seem to apply. If I were starting from scratch, I'd probably start with
Ganetti (a really nice tool from Google that they use to manage small,
internal clusters, used for administrative applications - though it has some
subtleties that get in the way of doing a completely automated failover). As
I'm not starting from scratch, I just want to get existing shell scripts
under configuration control, and automate manual steps where that's easy
(mostly re. IP address and DNS record managment when creating new virtual
machines).
For production, on the other hand, if we're fortunate, the focus will be on
deploying lots of cookie cutter nodes - each with a fairly constrained
environment - and that's where the combination of an orchestration tool + a
configuration engine seems to offer a lot of leverage. And, yes, I expect
we'll want to wire together a small cluster first, and get the tooling right
before scaling out. It's just that I don't expect this cluster, or
associated procedures, to look anything like our inward facing stuff.
Actually, development and QA is where I see some of the most value in the
complete automation. If you can provide a self-service web page where
developers can click a few selections and get a complete machine (and
destroy an old one) in a few seconds, you have a _very_ valuable tool.

by the way, in case it's not clear, the config management tool should be
configuring your VMs, not just your physical systems.

far too frequently dev and QA machines accumulate 'cruft' left over from
prior installs and tests that end up making new tests work accidently,
causing problems when the code gets into production where such cruft does
not exist.

now I don't think that this means that you need to go to the extreme that
some people do of making the recipe for one app/package try to cover every
possible combination of that package and everything else. I try to work at
a much more granular level where a server seldom has as many as a half
dozen 'app bundles' on them and each app bundle may install a bunch of
packages. If you need systems to do a lot of different things, use a bunch
of VMs, each configured to do one thing (or a small number of things,
hopefully related) instead.

David Lang
Brandon Allbery
2012-03-20 13:23:12 UTC
Permalink
Post by Miles Fidelman
What I keep coming back to is that they seem like overkill for what I'm
- install a layer of infrastructure
- work in yet another language (bash and sed do pretty much all I need,
why have to wade through recipes written in Ruby or Python or whatever)?
- do I really need to deal with lots of recipes to ultimately execute a
few install and config. statements (who really needs the overhead to
execute "apt-get install" or "./configure; ./make; ./make test, ./make
install") - at least for the stuff I'm managing now, the details are in
site-specific configurations, and it seems like a lot of getting that right
with Chef (or whatever) involves wading into and understanding how the
individual recipes work (or writing them)
Yes, it's overkill for small environments --- but going back and adding it
after you've grown is a much larger job than setting it up to begin with,
and it's not especially common to be able to reliably say that you are
unlikely to grow into the situations where you need this. It just seems to
be the nature of our profession that small, simple setups grow and become
more complex and you find yourself needing better tools.
--
brandon s allbery ***@gmail.com
wandering unix systems administrator (available) (412) 475-9364 vm/sms
d***@lang.hm
2012-03-20 13:31:25 UTC
Permalink
Pretty much ok on 1-3, 4 and 5 are what I'm focusing on right now. Right
now, most of my build and config takes the form of fairly simple checklists,
- apt-get install <foo> (or: download; untar; ./configure, ./make, ./make
test, ./make install)
- look up some config info (e.g., pull an unused IP address off a checklist)
- add a domain record to a zone file
- edit one or more config files
- ,/init.d/<foo> start
other than editing config config files, pretty much everything consists of
one-line shell commands - easy enough to stick into a bash script; and I
guess a lot of the configuration could be done by adding sed commands to the
script
thinking more and more that rundeck (or something like it) would make it easy
to manage and execute those scripts (the other thing that looks pretty
interesting is a little project called sm-framework
(https://sm.beginrescueend.com/presentations/SM-Framework-Presentation.pdf) -
essentially adds some glue to the zsh and treats scripts like plug-ins
One problem with just automating apt-get install <foo> is that you don't
know what version of <foo> will be installed. It will be the latest one in
the repository you are pointing at (by default the upstream project). In
development this is fine, but in production this can be a problem.

You can also get into a situation where you have tested with library
foo.1, but library foo.2 is now out. you want to have foo.2 installed in
dev/qa for testing, but if a production machine dies and needs to be
re-imaged, you want to install version foo.1

Or one problem I've seen, version foo.2 comes out, you test in QA. Then
maintinance night rolls around and you do apt-get update in production and
in installs version foo.3 that was released 5 minutes ago and has a bug in
it that kills your system.

to avoid these sorts of problems, you really don't want your production
systems updating themselves from the upstream repositories. You need to
have some process to download the packages, store them locally, and
install just the version that you want.

This can be via some tool like puppet/chef/cfengine, or you can run your
own repositories locally (one each for dev/qa/prod) with some sort of
process to graduate packages from one repository to the next as testing
completes.

David Lang
Miles Fidelman
2012-03-20 13:53:27 UTC
Permalink
Post by d***@lang.hm
Post by Miles Fidelman
Pretty much ok on 1-3, 4 and 5 are what I'm focusing on right now.
Right now, most of my build and config takes the form of fairly
- apt-get install <foo> (or: download; untar; ./configure, ./make,
./make test, ./make install)
- look up some config info (e.g., pull an unused IP address off a checklist)
- add a domain record to a zone file
- edit one or more config files
- ,/init.d/<foo> start
other than editing config config files, pretty much everything
consists of one-line shell commands - easy enough to stick into a
bash script; and I guess a lot of the configuration could be done by
adding sed commands to the script
thinking more and more that rundeck (or something like it) would make
it easy to manage and execute those scripts (the other thing that
looks pretty interesting is a little project called sm-framework
(https://sm.beginrescueend.com/presentations/SM-Framework-Presentation.pdf)
- essentially adds some glue to the zsh and treats scripts like plug-ins
One problem with just automating apt-get install <foo> is that you
don't know what version of <foo> will be installed. It will be the
latest one in the repository you are pointing at (by default the
upstream project). In development this is fine, but in production this
can be a problem.
Point taken.

Thanks!

Miles
--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra
d***@gmail.com
2012-03-20 19:22:16 UTC
Permalink
Post by Miles Fidelman
Post by Tom Perrine
I've used both cfengine and Puppet over the years, and have met people
who have been successful with Chef. What has worked for others in
their specific environment may (or may not) work well for yours.
What I keep coming back to is that they seem like overkill for what I'm
- install a layer of infrastructure
- work in yet another language (bash and sed do pretty much all I need,
why have to wade through recipes written in Ruby or Python or whatever)?
- do I really need to deal with lots of recipes to ultimately execute a
few install and config. statements (who really needs the overhead to
execute "apt-get install" or "./configure; ./make; ./make test, ./make
install") - at least for the stuff I'm managing now, the details are in
site-specific configurations, and it seems like a lot of getting that
right with Chef (or whatever) involves wading into and understanding how
the individual recipes work (or writing them)
It's useful to look at the benefits of configuration management tools
beyond automation. As a small example, you can run your provisioning
shell/perl scripts from within a CM framework as shell commands. What
would be the point? Once your hosts are in a CM system you can leverage
the existing tools to get the additional features you want.

1. You get inventory tools such as facter and ohai to report on the
systems configuration in a standardized manner.

2. Most CM tools can store your hosts configuration in a central database
to provide ongoing system inventory information.

3. You get reporting on the state of your hosts (Example:
http://puppetlabs.com/puppet/related-projects/dashboard/). As you add
checks for each script, you will see whether or not your hosts are
compliant, when the script was run, and how long the script took to run.
This information is recorded in a database for historical purposes.

4. There are tools such as foreman (http://theforeman.org/) that integrate
inventory management/provisioning with your CM system. Foreman can control
DNS and DHCP for you when you add a new host to the system. Foreman
doesn't require that you use a CM system, but when you do, there is a
single place to record the state each host should be in.

5. Going forward you may want to take parts of your scripts and move them
into discrete clauses in your CM configuration. That will start to give
you the version control and idempotence benefits that others have
mentioned.

6. You mentioned that you like the power of perl+CPAN for application
installation. One of the big gains in modern CM systems is the
availability of modules or recipes. They provide a bundled set of
functionality that you can then use in your CM code similar to the way
CPAN modules can be used by various perl applications. A side benefit is
that they often provide good documentation on deploying various
applications.

Some of my examples are Puppet specific because that is what I'm familiar
with. Any of the modern CM tools should be able to provide equivalent
functionality. By using a common framework rather than only custom scripts
you get the above functionality for much lower cost than developing it
yourself. You also get to leverage the future work of other
sysadmins/developers who are writing new features within the framework.

Thanks,

Doug
Miles Fidelman
2012-03-20 21:05:34 UTC
Permalink
Post by d***@gmail.com
4. There are tools such as foreman (http://theforeman.org/) that integrate
inventory management/provisioning with your CM system. Foreman can control
DNS and DHCP for you when you add a new host to the system. Foreman
doesn't require that you use a CM system, but when you do, there is a
single place to record the state each host should be in.
Foreman looks pretty interesting, thanks for the pointer.
Post by d***@gmail.com
6. You mentioned that you like the power of perl+CPAN for application
installation. One of the big gains in modern CM systems is the
availability of modules or recipes. They provide a bundled set of
functionality that you can then use in your CM code similar to the way
CPAN modules can be used by various perl applications. A side benefit is
that they often provide good documentation on deploying various
applications.
Actually, those kind of give me nightmares. It's bad enough when .deb
and .rpm packages are out of date, or just plain broken. Now one has to
worry about what a recipe does and whether it works properly? I've
started to read through various chef recipes, to compare what they do to
the more manual ways I've been configuring stuff - there's an awful lot
of complexity and dependencies that worry me a lot.

By and large, for most mature software that we install, I've found that
the upstream tarballs, install instructions, and makefiles "just work."
Editing a config file is simple, and often automatic. Binary packages
are more of a crapshoot. Somehow worrying about chef recipes (or
whatever) scares the heck out of me.


Miles
--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra
Miles Fidelman
2012-03-23 17:16:58 UTC
Permalink
Thanks to all who replied to my original query, looking for an
orchestration tool that would:


- provide for editing/managing/versioning scripts (script = anything
that can be invoked at the command line)
- a library of control functions for use within scripts
- invoking scripts, combinations of scripts, pipelines of scripts (in
the Unix sense of pipes) - locally, remotely, across multiple machines
- remote script execution via ssh, rather than some kind of agent
- providing a simple database for keeping track of variables used by
scripts (e.g., IP addresses, DNS records for use by a provisioning
script) - that can be accessed from scripts
- accessing the above via cli, RESTful API, GUI
- cross-platform
- (nice-to-have) minimal environmental requirements (i.e., a step above
the unix shell, say the gnu buildtool suite rather than the JVM + a mass
of libraries, or the ruby ecosystem -- a self-configuring basic
environment would be nice, like perl+cpan)


A couple of folks on the devops list pointed me at this: https://bdsm.beginrescueend.com/modules/shell

Essentially it's management framework for running shell scripts, written by Wayne Seguin at Engine Yard,
also the author of RVM (Ruby Version Manager). Information on the web site is just a bit disorganized,
but there's a pretty good manual in pdf, an introductory slideshow, and a comprehensive git repo.

Wayne just spent the morning walking me through it - an incredibly powerful tool, which I'm now going to
go off and use as I rebuild a couple of servers.

Miles Fidelman
--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra
Miles Fidelman
2012-03-23 17:21:16 UTC
Permalink
Post by Miles Fidelman
https://bdsm.beginrescueend.com/modules/shell
Essentially it's management framework for running shell scripts,
written by Wayne Seguin at Engine Yard,
also the author of RVM (Ruby Version Manager). Information on the web
site is just a bit disorganized,
but there's a pretty good manual in pdf, an introductory slideshow,
and a comprehensive git repo.
Wayne just spent the morning walking me through it - an incredibly
powerful tool, which I'm now going to
go off and use as I rebuild a couple of servers.
I should add - rundeck and rerun, both from the folks at dtosolutions -
also look like they might be incredibly helpful.
--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra
Yves Dorfsman
2012-03-24 16:51:37 UTC
Permalink
The devops list?
--
Yves. http://www.SollerS.ca/
http://ipv6.SollerS.ca
http://blog.zioup.org/
Miles Fidelman
2012-03-24 22:44:26 UTC
Permalink
Post by Yves Dorfsman
The devops list?
http://groups.google.com/group/devops
and
http://groups.google.com/group/devops-toolchain

Mostly folks who are users, developers, and fans of chef, puppet,
similar tools. A few folks have a broader sense of operations. (Call
me sensitive, but when I queried about management of shell scripts, I
got jumped on by rapid chef and puppet users.)

Miles Fidelman
--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra
Wayne E. Seguin
2012-03-25 23:44:36 UTC
Permalink
On Fri, Mar 23, 2012 at 1:16 PM, Miles Fidelman
Post by Miles Fidelman
Thanks to all who replied to my original query, looking for an
- provide for editing/managing/versioning scripts (script = anything
that can be invoked at the command line)
- a library of control functions for use within scripts
- invoking scripts, combinations of scripts, pipelines of scripts (in
the Unix sense of pipes) - locally, remotely, across multiple machines
- remote script execution via ssh, rather than some kind of agent
- providing a simple database for keeping track of variables used by
scripts (e.g., IP addresses, DNS records for use by a provisioning
script) - that can be accessed from scripts
- accessing the above via cli, RESTful API, GUI
- cross-platform
- (nice-to-have) minimal environmental requirements (i.e., a step above
the unix shell, say the gnu buildtool suite rather than the JVM + a mass
of libraries, or the ruby ecosystem -- a self-configuring basic
environment would be nice, like perl+cpan)
https://bdsm.beginrescueend.**com/modules/shell<https://bdsm.beginrescueend.com/modules/shell>
Essentially it's management framework for running shell scripts, written
by Wayne Seguin at Engine Yard,
also the author of RVM (Ruby Version Manager). Information on the web
site is just a bit disorganized,
but there's a pretty good manual in pdf, an introductory slideshow, and a
comprehensive git repo.
Wayne just spent the morning walking me through it - an incredibly
powerful tool, which I'm now going to
go off and use as I rebuild a couple of servers.
Hello everyone,

The website is very old and outdated, for anyone that wishes to learn more
about the framework please read/do one of the following:

1. Lightning Talk Slides:
https://sm.beginrescueend.com/presentations/SM-Framework-Presentation.pdf
2. Manual (work in progress): https://sm.beginrescueend.com/sm-manual.pdf
3. API Docs (work in progress):
https://github.com/sm/sm_apidoc/blob/master/README.md
4. Contact me (best to start with IRC, details below):

Yours truly,

~Wayne

Wayne E. Seguin
***@gmail.com
wayneeseguin on irc.freenode.net
http://twitter.com/wayneeseguin/
https://github.com/wayneeseguin/

Tom Perrine
2012-03-20 02:38:13 UTC
Permalink
It's too bad that your question sparked so much religious fervor "over
there". For what it's worth, I don't think you can go wrong with any
of those.

I've used both cfengine and Puppet over the years, and have met people
who have been successful with Chef. What has worked for others in
their specific environment may (or may not) work well for yours.

That's what offers the excuse for the "one true way" arguments.

Rule 16 and 36 apply:
(http://thuktun.wordpress.com/toms-rules-for-happy-sysadmins-and-users/)

I'd start with this list
(http://en.wikipedia.org/wiki/Comparison_of_open_source_configuration_management_software
) and evaluate based on a few criteria:

1. Support for the way you manage your site today. Think about whether
or not you have a CMDB or inventory system, or you want your CM tool
to provide it. Think about what OSes you need to support: If you need
Windows (and Mac, and UNIX) support your choices may be more limited.
Think about the things you like about the way you manage now (and want
to keep) and the things you want to change. See which tools are more
like the way you want things to be in the future, not which ones
exactly match what you're doing now.

2. If you think you may need to create extensions, are you comfortable
using the needed programming/extension language? CFEngine is C. Puppet
and Chef are Ruby.

3. Do you want to only manage the operating system and its control
files, or do you want to manage applications, too? (Or vice versa.)

4. For which tools can you "find the answers in the back of the book?"
Do any of the systems already have modules or mechanisms for (any) of
your use cases? If so, build on those.

5. What's the (local) community? Are there people nearby who are using
the systems? Can they provide local support? Can you get some help
for the price of lunch, beer, or dinner :-) How's the online
community?

6. What does the "vendor" support look like? Look at the mailing lists
and bug trackers. Does the "vendor" (or community) seem responsive?
Are there constant complaints that no one is providing (or accepting)
patches?

In other words, decide what criteria are important to you and your
site, and evaluate the choices in light of your requirements, not
mine, or anyone else's.

I can't recommend automated config management enough. No matter which
tool you choose,It will fundamentally change the way you look at
system administration, and (after the painful startup) it will pay you
back in better quality of life, for you and your user community.


Good luck.

And please share your experiences during and after you make the leap...

--tep
Anton Cohen
2012-03-20 04:55:58 UTC
Permalink
On Mon, Mar 19, 2012 at 6:28 PM, Miles Fidelman
Post by Miles Fidelman
1. What is the state of the practice right now? How much is sys
administration a world of traditional approaches, vs. how much have the new
generation of devops tools caught on outside the core of folks who've drunk
the koolaid? (I'm really trying to get calibrated in reality here.)
Excessive Kool-Aid is most likely clouding my vision, but the way I see it
everyone is a) drinking the Kool-Aid, b) wishing they had some tasty
Kool-Aid to drink, or c) too drunk on stale beer to know Kool-Aid tastes
way better.

To sum up your requirements: You want a cross-platform tool with
a library of functions, that executes scripts on remote hosts over a secure
protocol, and has access to variables in a database. You want to access the
tool from CLI, GUI, and REST API. You want the scripts to be version
controlled.

What you described *is* Configuration Management, like Puppet and Chef.
Really, what you asked for exists, except they communicate over HTTPS
instead of SSH, and are written in Ruby instead of Tcl.
The dependency requirements aren't that bad, at least for Puppet and
CFEngine. They can be satisfied from the standard repos on Debian-based
hosts and standard repos + EPEL on Red Hat-based hosts.

Maybe ${tool_name} isn't right for you, but of the many
configuration management systems out there, one of them is probably a good
fit.

-Anton

(Wait, I just re-read your email... did you say Perl + CPAN is nice? That's
like saying getting punched repeatedly in the face, while your hands are
tied behind your back is nice.)
Paul Mooring
2012-03-20 05:38:03 UTC
Permalink
* Disclosure I work for Opscode, the company behind Chef

I just read through this thread and wanted to put in my 2 cents about
what modern config management software does and does not do from your
list of requirements. I should also mention I don't really have any
first hand experience with CFEngine so I could be mistaken on it's
capabilities, that being said here's my thoughts. Config management
is usually not for orchestration Rundeck/MCollective are usually used
with Chef/Puppet, the config management is used to define an
idempotent system or infrastructure and the orchestration tool for
running one off scripts on demand across multiple machines. What you
described seems to incorporate both so I would suggest you considering
using 2 tools.

I think it's also important to note that both of the new popular
config management tools (Puppet and Chef) aren't just written in Ruby
but require at least some Ruby knowledge to even use. They both use
erb templates, Chef allows pure ruby across the board, and Puppet
resources would be written in Ruby, so if you're heading in that
direction you're best off just embracing Ruby from the start.

Also for what it's worth, I really like the Rundeck/Chef combination,
especially for the list of requirements you have.

On Mon, Mar 19, 2012 at 6:28 PM, Miles Fidelman
Post by Miles Fidelman
Hi Folks,
New list member here, seems like this might be the place to pose this
I'm getting ready to rebuild a small cluster - mostly used for development,
but likely to move into a production role, and thinking about being prepared
if we need to scale.  As a result, I'm looking at way to move from a rather
ad hoc management approach - configuration notes and task checklists in word
documents, spreadsheet of IP addresses and DNS records, bunches of shell
scripts and cron jobs - to something a bit more organized and scaleable.
My first thought is to start with a simple database for config. info, and
moving scripts under configuration control; maybe adding an orchestration
tool like rundeck (though call me old fashioned enough to think about
writing some wrapper code in tcl).
Anyway, when I asked for tool suggestions on one of the devops lists, where
I kind of figured was the place to ask about tools, about all I got were
religious pontifications about puppet or chef being the "one true way" -
which seems a bit out of line with my experience of real-world operations,
both in the small, and in the large (been around both, though more as an
architect for large systems than as an operator).
1.   What is the state of the practice right now?  How much is sys
administration a world of traditional approaches, vs. how much have the new
generation of devops tools caught on outside the core of folks who've drunk
the koolaid?  (I'm really trying to get calibrated in reality here.)
2.   My more specific question re. what are people using to manage
accumulated scripts and semi-manual procedures?
I'm looking for something along the following lines - an orchestration tool
- provide for editing/managing/versioning scripts (script = anything
that can be invoked at the command line)
- a library of control functions for use within scripts
- invoking scripts, combinations of scripts, pipelines of scripts (in
the Unix sense of pipes) - locally, remotely, across multiple machines
- remote script execution via ssh, rather than some kind of agent
- providing a simple database for keeping track of variables used by
scripts (e.g., IP addresses, DNS records for use by a provisioning
script) - that can be accessed from scripts
- accessing the above via cli, RESTful API, GUI
- cross-platform
- (nice-to-have) minimal environmental requirements (i.e., a step above
the unix shell, say the gnu buildtool suite rather than the JVM + a mass
of libraries, or the ruby ecosystem -- a self-configuring basic
environment would be nice, like perl+cpan)
I'm sure we've each cobbled something together along these lines - be it
through manual processes, standard installs, or something fancier.  I'm
kind of wondering what different people have done, what pieces fit
together better than others, what's available that's more integrated but
that doesn't create a monolithic environment.
Rundeck seems to do most (all?) of this, running on Java.  It looks
nice, but it's also really new.
tcl/tk provides a lot of this - but without the organizing functions
(tcl/tk + git, or maybe a tcl/tk IDE perhaps?)
What have people found that works?
Thanks very much,
Miles Fidelman
--
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Loading...