Tue Oct 8 03:01:20 2024
EVENTS
 FREE
SOFTWARE
INSTITUTE

POLITICS
JOBS
MEMBERS'
CORNER

MAILING
LIST

NYLXS Mailing Lists and Archives
NYLXS Members have a lot to say and share but we don't keep many secrets. Join the Hangout Mailing List and say your peice.

DATE 2015-02-01

HANGOUT

2024-10-08 | 2024-09-08 | 2024-08-08 | 2024-07-08 | 2024-06-08 | 2024-05-08 | 2024-04-08 | 2024-03-08 | 2024-02-08 | 2024-01-08 | 2023-12-08 | 2023-11-08 | 2023-10-08 | 2023-09-08 | 2023-08-08 | 2023-07-08 | 2023-06-08 | 2023-05-08 | 2023-04-08 | 2023-03-08 | 2023-02-08 | 2023-01-08 | 2022-12-08 | 2022-11-08 | 2022-10-08 | 2022-09-08 | 2022-08-08 | 2022-07-08 | 2022-06-08 | 2022-05-08 | 2022-04-08 | 2022-03-08 | 2022-02-08 | 2022-01-08 | 2021-12-08 | 2021-11-08 | 2021-10-08 | 2021-09-08 | 2021-08-08 | 2021-07-08 | 2021-06-08 | 2021-05-08 | 2021-04-08 | 2021-03-08 | 2021-02-08 | 2021-01-08 | 2020-12-08 | 2020-11-08 | 2020-10-08 | 2020-09-08 | 2020-08-08 | 2020-07-08 | 2020-06-08 | 2020-05-08 | 2020-04-08 | 2020-03-08 | 2020-02-08 | 2020-01-08 | 2019-12-08 | 2019-11-08 | 2019-10-08 | 2019-09-08 | 2019-08-08 | 2019-07-08 | 2019-06-08 | 2019-05-08 | 2019-04-08 | 2019-03-08 | 2019-02-08 | 2019-01-08 | 2018-12-08 | 2018-11-08 | 2018-10-08 | 2018-09-08 | 2018-08-08 | 2018-07-08 | 2018-06-08 | 2018-05-08 | 2018-04-08 | 2018-03-08 | 2018-02-08 | 2018-01-08 | 2017-12-08 | 2017-11-08 | 2017-10-08 | 2017-09-08 | 2017-08-08 | 2017-07-08 | 2017-06-08 | 2017-05-08 | 2017-04-08 | 2017-03-08 | 2017-02-08 | 2017-01-08 | 2016-12-08 | 2016-11-08 | 2016-10-08 | 2016-09-08 | 2016-08-08 | 2016-07-08 | 2016-06-08 | 2016-05-08 | 2016-04-08 | 2016-03-08 | 2016-02-08 | 2016-01-08 | 2015-12-08 | 2015-11-08 | 2015-10-08 | 2015-09-08 | 2015-08-08 | 2015-07-08 | 2015-06-08 | 2015-05-08 | 2015-04-08 | 2015-03-08 | 2015-02-08 | 2015-01-08 | 2014-12-08 | 2014-11-08 | 2014-10-08 | 2014-09-08 | 2014-08-08 | 2014-07-08 | 2014-06-08 | 2014-05-08 | 2014-04-08 | 2014-03-08 | 2014-02-08 | 2014-01-08 | 2013-12-08 | 2013-11-08 | 2013-10-08 | 2013-09-08 | 2013-08-08 | 2013-07-08 | 2013-06-08 | 2013-05-08 | 2013-04-08 | 2013-03-08 | 2013-02-08 | 2013-01-08 | 2012-12-08 | 2012-11-08 | 2012-10-08 | 2012-09-08 | 2012-08-08 | 2012-07-08 | 2012-06-08 | 2012-05-08 | 2012-04-08 | 2012-03-08 | 2012-02-08 | 2012-01-08 | 2011-12-08 | 2011-11-08 | 2011-10-08 | 2011-09-08 | 2011-08-08 | 2011-07-08 | 2011-06-08 | 2011-05-08 | 2011-04-08 | 2011-03-08 | 2011-02-08 | 2011-01-08 | 2010-12-08 | 2010-11-08 | 2010-10-08 | 2010-09-08 | 2010-08-08 | 2010-07-08 | 2010-06-08 | 2010-05-08 | 2010-04-08 | 2010-03-08 | 2010-02-08 | 2010-01-08 | 2009-12-08 | 2009-11-08 | 2009-10-08 | 2009-09-08 | 2009-08-08 | 2009-07-08 | 2009-06-08 | 2009-05-08 | 2009-04-08 | 2009-03-08 | 2009-02-08 | 2009-01-08 | 2008-12-08 | 2008-11-08 | 2008-10-08 | 2008-09-08 | 2008-08-08 | 2008-07-08 | 2008-06-08 | 2008-05-08 | 2008-04-08 | 2008-03-08 | 2008-02-08 | 2008-01-08 | 2007-12-08 | 2007-11-08 | 2007-10-08 | 2007-09-08 | 2007-08-08 | 2007-07-08 | 2007-06-08 | 2007-05-08 | 2007-04-08 | 2007-03-08 | 2007-02-08 | 2007-01-08 | 2006-12-08 | 2006-11-08 | 2006-10-08 | 2006-09-08 | 2006-08-08 | 2006-07-08 | 2006-06-08 | 2006-05-08 | 2006-04-08 | 2006-03-08 | 2006-02-08 | 2006-01-08 | 2005-12-08 | 2005-11-08 | 2005-10-08 | 2005-09-08 | 2005-08-08 | 2005-07-08 | 2005-06-08 | 2005-05-08 | 2005-04-08 | 2005-03-08 | 2005-02-08 | 2005-01-08 | 2004-12-08 | 2004-11-08 | 2004-10-08 | 2004-09-08 | 2004-08-08 | 2004-07-08 | 2004-06-08 | 2004-05-08 | 2004-04-08 | 2004-03-08 | 2004-02-08 | 2004-01-08 | 2003-12-08 | 2003-11-08 | 2003-10-08 | 2003-09-08 | 2003-08-08 | 2003-07-08 | 2003-06-08 | 2003-05-08 | 2003-04-08 | 2003-03-08 | 2003-02-08 | 2003-01-08 | 2002-12-08 | 2002-11-08 | 2002-10-08 | 2002-09-08 | 2002-08-08 | 2002-07-08 | 2002-06-08 | 2002-05-08 | 2002-04-08 | 2002-03-08 | 2002-02-08 | 2002-01-08 | 2001-12-08 | 2001-11-08 | 2001-10-08 | 2001-09-08 | 2001-08-08 | 2001-07-08 | 2001-06-08 | 2001-05-08 | 2001-04-08 | 2001-03-08 | 2001-02-08 | 2001-01-08 | 2000-12-08 | 2000-11-08 | 2000-10-08 | 2000-09-08 | 2000-08-08 | 2000-07-08 | 2000-06-08 | 2000-05-08 | 2000-04-08 | 2000-03-08 | 2000-02-08 | 2000-01-08 | 1999-12-08

Key: Value:

Key: Value:

MESSAGE
DATE 2015-02-17
FROM mrbrklyn@panix.com
SUBJECT Subject: [NYLXS - HANGOUT] systemd and process 1


http://ewontfix.com/14/

Broken by design: systemd
09 Feb 2014 19:56:09 GMT

Recently the topic of systemd has come up quite a bit in various
communities in which I'm involved, including the musl IRC channel and on
the Busybox mailing list.

While the attitude towards systemd in these communities is largely
negative, much of what I've seen has been either dismissable by folks in
different circles as mere conservatism, or tempered by an idea that
despite its flaws, "the design is sound". This latter view comes with
the notion that systemd's flaws are fixable without scrapping it or
otherwise incurring major costs, and therefore not a major obstacle to
adopting systemd.

My view is that this idea is wrong: systemd is broken by design, and
despite offering highly enticing improvements over legacy init systems,
it also brings major regressions in terms of many of the areas Linux is
expected to excel: security, stability, and not having to reboot to
upgrade your system.
The first big problem: PID 1

On unix systems, PID 1 is special. Orphaned processes (including a
special case: daemons which orphan themselves) get reparented to PID 1.
There are also some special signal semantics with respect to PID 1, and
perhaps most importantly, if PID 1 crashes or exits, the whole system
goes down (kernel panic).

Among the reasons systemd wants/needs to run as PID 1 is getting
parenthood of badly-behaved daemons that orphan themselves, preventing
their immediate parent from knowing their PID to signal or wait on them.

Unfortunately, it also gets the other properties, including bringing
down the whole system when it crashes. This matters because systemd is
complex. A lot more complex than traditional init systems. When I say
complex, I don't mean in a lines-of-code sense. I mean in terms of the
possible inputs and code paths that may be activated at runtime. While
legacy init systems basically deal with no inputs except SIGCHLD from
orphaned processes exiting and manual runlevel changes performed by the
administrator, systemd deals with all sorts of inputs, including device
insertion and removal, changes to mount points and watched points in the
filesystem, and even a public DBus-based API. These in turn entail
resource allocation, file parsing, message parsing, string handling, and
so on. This brings us to:
The second big problem: Attack Surface

On a hardened system without systemd, you have at most one
root-privileged process with any exposed surface: sshd. Everything else
is either running as unprivileged users or does not have any channel for
providing it input except local input from root. Using systemd then more
than doubles the attack surface.

This increased and unreasonable risk is not inherent to systemd's goal
of fixing legacy init. However it is inherent to the systemd design
philosophy of putting everything into the init process.
The third big problem: Reboot to Upgrade

Windows Update rebooting

Fundamentally, upgrading should never require rebooting unless the
component being upgraded is the kernel. Even then, for security updates,
it's ideal to have a "hot-patch" that can be applied as a loadable
kernel module to mitigate the security issue until rebooting with the
new kernel is appropriate.

Unfortunately, by moving large amounts of functionality that's likely to
need to be upgraded into PID 1, systemd makes it impossible to upgrade
without rebooting. This leads to "Linux" becoming the laughing stock of
Windows fans, as happened with Ubuntu a long time ago.
Possible counter-arguments

With regards to security, one could ask why can't desktop systems use
systemd, and leave server systems to find something else. But I think
this line of reasoning is flawed in at least three ways:

Many of the selling-point features of systemd are server-oriented.
State-of-the-art transaction-style handling of daemon starting and
stopping is not a feature that's useful on desktop systems. The intended
audience for that sort of thing is clearly servers.

The desktop is quickly becoming irrelevant. The future platform is
going to be mobile and is going to be dealing with the reality of
running untrusted applications. While the desktop made the unix
distinction of local user accounts largely irrelevant, the coming of
mobile app ecosystems full of potentially-malicious apps makes "local
security" more important than ever.

The crowd pushing systemd, possibly including its author, is not
content to have systemd be one choice among many. By providing public
APIs intended to be used by other applications, systemd has set itself
up to be difficult not to use once it achieves a certain adoption
threshold.

With regards to upgrades, systemd's systemctl has a daemon-reexec
command to make systemd serialize its state, re-exec itself, and
continue uninterrupted. This could perhaps be used to switch to a new
version without rebooting. Various programs already use this technique,
such as the IRC client irssi which lets you /upgrade without dropping
any connections. Unfortunately, this brings us back to the issue of PID
1 being special. For normal applications, if re-execing fails, the worst
that happens is the process dies and gets restarted (either manually or
by some monitoring process) if necessary. However for PID 1, if
re-execing itself fails, the whole system goes down (kernel panic).

For common reasons it might fail, the execve syscall returns failure in
the original process image, allowing the program to handle the error.
However, failure of execve is not entirely atomic:

The kernel may fail setting up the VM for the new process image
after the original VM has already been destroyed; the main situation
under which this would happen is resource exhaustion.

Even after the kernel successfully sets up the new VM and transfers
execution to the new process image, it's possible to have failures prior
to the transfer of control to the actual application program. This could
happen in the dynamic linker (resource exhaustion or other transient
failures mapping required libraries or loading configuration files) or
libc startup code. Using musl libc with static linking or even dynamic
linking with no additional libraries eliminates these failure cases, but
systemd is intended to be used with glibc.

In addition, systemd might fail to restore its serialized state due to
resource allocation failures, or if the old and new versions have
diverged sufficiently that the old state is not usable by the new
version.

So if not systemd, what? Debian's discussion of whether to adopt systemd
or not basically devolved into a false dichotomy between systemd and
upstart. And except among grumpy old luddites, keeping legacy sysvinit
is not an attractive option. So despite all its flaws, is systemd still
the best option?

No.

None of the things systemd "does right" are at all revolutionary.
They've been done many times before. DJB's daemontools, runit, and
Supervisor, among others, have solved the "legacy init is broken"
problem over and over again (though each with some of their own flaws).
Their failure to displace legacy sysvinit in major distributions had
nothing to do with whether they solved the problem, and everything to do
with marketing. Said differently, there's nothing great and
revolutionary about systemd. Its popularity is purely the result of an
aggressive, dictatorial marketing strategy including elements such as:

Engulfing other "essential" system components like udev and making
them difficult or impossible to use without systemd (but see eudev).

Setting up for API lock-in (having the DBus interfaces provided by
systemd become a necessary API that user-level programs depend on).

Dictating policy rather than being scoped such that the user,
administrator, or systems integrator (distribution) has to provide glue.
This eliminates bikesheds and thereby fast-tracks adoption at the
expense of flexibility and diversity.

So how should init be done right?

The Unix way: with simple self-contained programs that do one thing and
do it well.

First, get everything out of PID 1:

The systemd way: Take advantage of special properties of pid 1 to the
maximum extent possible. This leads to ever-expanding scope creep and
exacerbates all of the problems described above (and probably many more
yet to be discovered).

The right way: Do away with everything special about pid 1 by making pid
1 do nothing but start the real init script and then just reap zombies:

#define _XOPEN_SOURCE 700
#include
#include

int main()
{
sigset_t set;
int status;

if (getpid() != 1) return 1;

sigfillset(&set);
sigprocmask(SIG_BLOCK, &set, 0);

if (fork()) for (;;) wait(&status);

sigprocmask(SIG_UNBLOCK, &set, 0);

setsid();
setpgid(0, 0);
return execve("/etc/rc", (char *[]){ "rc", 0 }, (char *[]){ 0 });
}

Yes, that's really all that belongs in PID 1. Then there's no way it can
fail at runtime, and no need to upgrade it once it's successfully
running.

Next, from the init script, run a process supervision system outside of
PID 1 to manage daemons as immediate child processes (no backgrounding).
As mentioned above are several existing choices here. It's not clear to
me that any of them are sufficiently polished or robust to satisfy major
distributions at this time. But neither is systemd; its backers are just
better at sweeping that under the rug.

What the existing choices do have, though, is better design, mainly in
the way of having clean, well-defined scope rather than Katamari Damacy.

If none of them are ready for prime time, then the folks eager to
replace legacy init in their favorite distributions need to step up and
either polish one of the existing solutions or write a better
implementation based on the same principles. Either of these options
would be a lot less work than fixing what's wrong with systemd.

Whatever system is chosen, the most important criterion is that it be
transparent to applications. For 30+ years, the choice of init system
used has been completely irrelevant to everybody but system integrators
and administrators. User applications have had no reason to know or care
whether you use sysvinit with runlevels, upstart, my minimal init with a
hard-coded rc script or a more elaborate process-supervision system, or
even /bin/sh. Ironically, this sort of modularity and interchangibility
is what made systemd possible; if we were starting from the kind of
monolithic, API-lock-in-oriented product systemd aims to be, swapping
out the init system for something new and innovative would not even be
an option.
Update: license on code

Added December 21, 2014.

There has been some interest in having a proper free software license on
the trivial init code included above. I originally considered it too
trivial to even care about copyright or need a license on it, but I
don't want this to keep anyone from using or reusing it, so I'm
explicitly licensing it under the following terms (standard MIT
license):

Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation files
(the "Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.



Ruben


--
So many immigrant groups have swept through our town
that Brooklyn, like Atlantis, reaches mythological
proportions in the mind of the world - RI Safir 1998
http://www.mrbrklyn.com

DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002
http://www.nylxs.com - Leadership Development in Free Software
http://www2.mrbrklyn.com/resources - Unpublished Archive
http://www.coinhangout.com - coins!
http://www.brooklyn-living.com

Being so tracked is for FARM ANIMALS and and extermination camps,
but incompatible with living as a free human being. -RI Safir 2013

  1. 2015-02-01 Ruben <ruben.safir-at-my.liu.edu> Subject: [NYLXS - HANGOUT] Your next laptop
  2. 2015-02-01 Robert Menes <viewtiful.icchan-at-gmail.com> Re: [NYLXS - HANGOUT] Your next laptop
  3. 2015-02-01 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] The Mayor and WINS
  4. 2015-02-02 Ruben <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: [Perlweekly] #184 - Larry gave his talk at FOSDEM
  5. 2015-02-02 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Linux Job Crunch
  6. 2015-02-02 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Linux Job Crunch
  7. 2015-02-02 Paul Robert Marino <prmarino1-at-gmail.com> Re: [NYLXS - HANGOUT] Linux Job Crunch
  8. 2015-02-02 mrbrklyn-at-panix.com Subject: [NYLXS - HANGOUT] [announce-at-lists.isoc-ny.org: [isoc-ny] Two NYC meetups Tuesday]
  9. 2015-02-04 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] vitamins?
  10. 2015-02-04 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Essentail Database tools
  11. 2015-02-04 mrbrklyn-at-panix.com Subject: [NYLXS - HANGOUT] [info-at-fsf.org: Gorgeous animated video against DRM]
  12. 2015-02-05 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Very Good Jobs
  13. 2015-02-05 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] sorry -n new postfix conf
  14. 2015-02-05 Ruben Safir <mrbrklyn-at-panix.com> Subject: [mrbrklyn-at-panix.com: [NYLXS - HANGOUT] sorry -n new postfix conf]
  15. 2015-02-05 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Journal Shaping Up
  16. 2015-02-05 prmarino1-at-gmail.com Re: [NYLXS - HANGOUT] Journal Shaping Up
  17. 2015-02-05 prmarino1-at-gmail.com Re: [NYLXS - HANGOUT] Journal Shaping Up
  18. 2015-02-06 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] New Live vaudeville Theater opening in Brooklyn
  19. 2015-02-06 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] antartic Linux
  20. 2015-02-08 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Community Based Education Programs we can emulate
  21. 2015-02-08 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Re: GNU education
  22. 2015-02-09 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Re: GNU education
  23. 2015-02-09 Ruben <ruben.safir-at-my.liu.edu> Re: [NYLXS - HANGOUT] Re: GNU education
  24. 2015-02-09 Ruben <ruben.safir-at-my.liu.edu> Re: [NYLXS - HANGOUT] Re: GNU education
  25. 2015-02-09 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Re: Rafi from NYLUG
  26. 2015-02-09 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Journal Shaping Up
  27. 2015-02-09 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Journal Shaping Up
  28. 2015-02-09 Elfen Magix <elfen_magix-at-yahoo.com> Subject: [NYLXS - HANGOUT] Email resend, warning - with headers
  29. 2015-02-09 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Re: Email resend, warning - with headers
  30. 2015-02-09 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Re: Article discussion
  31. 2015-02-10 Elfen Magix <elfen_magix-at-yahoo.com> Re: [NYLXS - HANGOUT] Community Based Education Programs we can emulate
  32. 2015-02-10 Elfen Magix <elfen_magix-at-yahoo.com> Re: [NYLXS - HANGOUT] Community Based Education Programs we can emulate
  33. 2015-02-10 mrbrklyn-at-panix.com Subject: [NYLXS - HANGOUT] [elana-at-flatironschool.com: [nyc-on-rails] Meetup Cancelled: February
  34. 2015-02-10 Ruben <ruben.safir-at-my.liu.edu> Re: [NYLXS - HANGOUT] Community Based Education Programs we can emulate
  35. 2015-02-10 Ruben <ruben.safir-at-my.liu.edu> Re: [NYLXS - HANGOUT] Community Based Education Programs we can emulate
  36. 2015-02-11 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Friday Morning Volunteer
  37. 2015-02-11 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Freee Software Job market
  38. 2015-02-11 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Community Based Education Programs we can emulate
  39. 2015-02-11 Paul Robert Marino <prmarino1-at-gmail.com> Re: [NYLXS - HANGOUT] Community Based Education Programs we can emulate
  40. 2015-02-11 prmarino1-at-gmail.com Re: [NYLXS - HANGOUT] Friday Morning Volunteer
  41. 2015-02-11 prmarino1-at-gmail.com Re: [NYLXS - HANGOUT] Friday Morning Volunteer
  42. 2015-02-11 Robert Menes <viewtiful.icchan-at-gmail.com> Re: [NYLXS - HANGOUT] Friday Morning Volunteer
  43. 2015-02-11 Robert Menes <viewtiful.icchan-at-gmail.com> Re: [NYLXS - HANGOUT] Friday Morning Volunteer
  44. 2015-02-11 Ruben <ruben.safir-at-my.liu.edu> Re: [NYLXS - HANGOUT] Friday Morning Volunteer
  45. 2015-02-11 Ruben <ruben.safir-at-my.liu.edu> Re: [NYLXS - HANGOUT] Friday Morning Volunteer
  46. 2015-02-11 Ruben Safir <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] Re: [NYLXS - HANGOUT] Friday Morning Volunteer
  47. 2015-02-11 Ruben Safir <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] Re: [NYLXS - HANGOUT] Friday Morning Volunteer
  48. 2015-02-11 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Community Based Education Programs we can emulate
  49. 2015-02-11 Robert Menes <viewtiful.icchan-at-gmail.com> Re: [NYLXS - HANGOUT] Friday Morning Volunteer
  50. 2015-02-11 Paul Robert Marino <prmarino1-at-gmail.com> Re: [NYLXS - HANGOUT] Community Based Education Programs we can emulate
  51. 2015-02-11 Ruben <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Community Based Education Programs we can emulate
  52. 2015-02-11 Ruben <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Friday Morning Volunteer
  53. 2015-02-11 prmarino1-at-gmail.com Re: [NYLXS - HANGOUT] Community Based Education Programs we can emulate
  54. 2015-02-11 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Community Based Education Programs we can
  55. 2015-02-12 mrbrklyn-at-panix.com Subject: [NYLXS - HANGOUT] [dmarti-at-zgp.org: Re: [linux-elitists] Potential conference for 2015?]
  56. 2015-02-12 mrbrklyn-at-panix.com Subject: [NYLXS - HANGOUT] [dmarti-at-zgp.org: [linux-elitists] Potential conference for 2015?]
  57. 2015-02-12 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Community Based Education Programs we can emulate
  58. 2015-02-12 Rick Moen <rick-at-linuxmafia.com> Subject: [NYLXS - HANGOUT] On the art of mailing lists
  59. 2015-02-12 Rick Moen <rick-at-linuxmafia.com> Subject: [NYLXS - HANGOUT] And another thing
  60. 2015-02-12 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Community Based Education Programs we can emulate
  61. 2015-02-12 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] And another thing
  62. 2015-02-12 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] sorry - host looks ok?
  63. 2015-02-15 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Movie of the week
  64. 2015-02-15 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Journal Article
  65. 2015-02-15 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Re: Journal Article
  66. 2015-02-16 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Chinese Backdoors
  67. 2015-02-16 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Autoencryption and table security
  68. 2015-02-16 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] get off the grid
  69. 2015-02-16 Rick Moen <rick-at-linuxmafia.com> Subject: [NYLXS - HANGOUT] Matching this mailing list for procmail rules
  70. 2015-02-16 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] Matching this mailing list for procmail rules
  71. 2015-02-16 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Matching this mailing list for procmail rules
  72. 2015-02-16 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Matching this mailing list for procmail rules
  73. 2015-02-16 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] fly anywahere robot
  74. 2015-02-17 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] Matching this mailing list for procmail rules
  75. 2015-02-17 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Matching this mailing list for procmail rules
  76. 2015-02-17 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] Matching this mailing list for procmail rules
  77. 2015-02-17 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Matching this mailing list for procmail rules
  78. 2015-02-17 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Matching this mailing list for procmail rules
  79. 2015-02-17 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] Matching this mailing list for procmail rules
  80. 2015-02-17 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] CCNY Hackers Meeting Tomorrow
  81. 2015-02-17 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Matching this mailing list for procmail rules
  82. 2015-02-17 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] Matching this mailing list for procmail rules
  83. 2015-02-17 mrbrklyn-at-panix.com Subject: [NYLXS - HANGOUT] systemd and process 1
  84. 2015-02-17 Elfen Magix <elfen_magix-at-yahoo.com> Re: [NYLXS - HANGOUT] CCNY Hackers Meeting Tomorrow
  85. 2015-02-17 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: Re: Wayland support?
  86. 2015-02-17 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] CCNY Hackers Meeting Tomorrow
  87. 2015-02-17 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] CCNY Hackers Meeting Tomorrow
  88. 2015-02-17 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Journal Meeting
  89. 2015-02-17 Michael Richardson <mlr52-at-michaellrichardson.com> Re: [NYLXS - HANGOUT] Journal Meeting
  90. 2015-02-17 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Journal Meeting
  91. 2015-02-18 Michael Richardson <mlr52-at-michaellrichardson.com> Re: [NYLXS - HANGOUT] Journal Meeting
  92. 2015-02-18 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Journal Meeting
  93. 2015-02-18 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Recursive Allorithms
  94. 2015-02-18 einker <eminker-at-gmail.com> Subject: [NYLXS - HANGOUT] In New York City, Jobs Come Back Without Wall Street
  95. 2015-02-18 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] In New York City, Jobs Come Back Without Wall
  96. 2015-02-18 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] In New York City, Jobs Come Back Without Wall
  97. 2015-02-18 mrbrklyn-at-panix.com Subject: [NYLXS - HANGOUT] [techinsider-at-ieee.org: Open Source Software Security - Mitigating
  98. 2015-02-18 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] [ruben-at-mrbrklyn.com: Re: Have you connected with your local Chapter?]
  99. 2015-02-18 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] [techinsider-at-ieee.org: Open Source Software
  100. 2015-02-18 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] [techinsider-at-ieee.org: Open Source Software
  101. 2015-02-18 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Journal Meeting
  102. 2015-02-20 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] a little C++ insite
  103. 2015-02-21 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] movie of the week
  104. 2015-02-22 Ruben <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] movie of the week
  105. 2015-02-22 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] movie of the week
  106. 2015-02-22 Rick Moen <rick-at-linuxmafia.com> Subject: [NYLXS - HANGOUT] Majordomo2: from-scratch improved version, real open source, still
  107. 2015-02-22 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] movie of the week
  108. 2015-02-22 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] movie of the week
  109. 2015-02-24 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Journal Articles
  110. 2015-02-24 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] What Jews here??
  111. 2015-02-24 mrbrklyn-at-panix.com Subject: [NYLXS - HANGOUT] [groups-noreply-at-linkedin.com: Great opportunity for QA Engineers in
  112. 2015-02-25 Paul Robert Marino <prmarino1-at-gmail.com> Re: [NYLXS - HANGOUT] Fwd: Re: Wayland support?
  113. 2015-02-25 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] one url
  114. 2015-02-25 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: Entrepreneurship & Innovation March 2 & 3rd Pratt 110
  115. 2015-02-25 Ruben <ruben.safir-at-my.liu.edu> Re: [NYLXS - HANGOUT] movie of the week
  116. 2015-02-25 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Fwd: Re: Wayland support?
  117. 2015-02-25 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] funding sources
  118. 2015-02-26 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] movie of the week
  119. 2015-02-26 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: Deadline Extended: Save $100 at Rock Stars of 3D Printing &
  120. 2015-02-26 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Lenovo Superfish
  121. 2015-02-26 Robert Menes <viewtiful.icchan-at-gmail.com> Re: [NYLXS - HANGOUT] Lenovo Superfish
  122. 2015-02-26 Ruben <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Lenovo Superfish
  123. 2015-02-26 Robert Menes <viewtiful.icchan-at-gmail.com> Re: [NYLXS - HANGOUT] Lenovo Superfish
  124. 2015-02-26 Ruben <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Lenovo Superfish
  125. 2015-02-26 Elfen Magix <elfen_magix-at-yahoo.com> Re: [NYLXS - HANGOUT] funding sources
  126. 2015-02-26 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] funding sources
  127. 2015-02-26 Michael Richardson <mlr52-at-michaellrichardson.com> Subject: [NYLXS - HANGOUT] Bios
  128. 2015-02-26 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] jobs
  129. 2015-02-26 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] shaping the future
  130. 2015-02-26 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] one more read
  131. 2015-02-26 Michael Richardson <mlr52-at-michaellrichardson.com> Subject: [NYLXS - HANGOUT] look at your submissions
  132. 2015-02-26 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] look at your submissions
  133. 2015-02-26 Michael Richardson <mlr52-at-michaellrichardson.com> Re: [NYLXS - HANGOUT] look at your submissions
  134. 2015-02-26 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] look at your submissions
  135. 2015-02-26 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] look at your submissions
  136. 2015-02-26 Michael Richardson <mlr52-at-michaellrichardson.com> Re: [NYLXS - HANGOUT] look at your submissions
  137. 2015-02-26 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Re: GNU education
  138. 2015-02-26 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] look at your submissions
  139. 2015-02-27 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Re: IEEE Seminar Friday 27th, 2015
  140. 2015-02-27 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Spock
  141. 2015-02-27 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] a hate this question
  142. 2015-02-27 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Read the FUCKING NOTES
  143. 2015-02-27 prmarino1-at-gmail.com Re: [NYLXS - HANGOUT] Read the FUCKING NOTES
  144. 2015-02-27 prmarino1-at-gmail.com Re: [NYLXS - HANGOUT] Read the FUCKING NOTES
  145. 2015-02-28 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: Gary McGraw Speaks
  146. 2015-02-28 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Read the FUCKING NOTES

NYLXS are Do'ers and the first step of Doing is Joining! Join NYLXS and make a difference in your community today!