Sat Nov 17 23:17:07 2018
EVENTS
 FREE
SOFTWARE
INSTITUTE

POLITICS
JOBS
MEMBERS'
CORNER

MAILING
LIST

NYLXS Mailing Lists and Archives
NYLXS Members have a lot to say and share but we don't keep many secrets. Join the Hangout Mailing List and say your peice.

DATE 2015-05-01

HANGOUT

2018-11-17 | 2018-10-17 | 2018-09-17 | 2018-08-17 | 2018-07-17 | 2018-06-17 | 2018-05-17 | 2018-04-17 | 2018-03-17 | 2018-02-17 | 2018-01-17 | 2017-12-17 | 2017-11-17 | 2017-10-17 | 2017-09-17 | 2017-08-17 | 2017-07-17 | 2017-06-17 | 2017-05-17 | 2017-04-17 | 2017-03-17 | 2017-02-17 | 2017-01-17 | 2016-12-17 | 2016-11-17 | 2016-10-17 | 2016-09-17 | 2016-08-17 | 2016-07-17 | 2016-06-17 | 2016-05-17 | 2016-04-17 | 2016-03-17 | 2016-02-17 | 2016-01-17 | 2015-12-17 | 2015-11-17 | 2015-10-17 | 2015-09-17 | 2015-08-17 | 2015-07-17 | 2015-06-17 | 2015-05-17 | 2015-04-17 | 2015-03-17 | 2015-02-17 | 2015-01-17 | 2014-12-17 | 2014-11-17 | 2014-10-17 | 2014-09-17 | 2014-08-17 | 2014-07-17 | 2014-06-17 | 2014-05-17 | 2014-04-17 | 2014-03-17 | 2014-02-17 | 2014-01-17 | 2013-12-17 | 2013-11-17 | 2013-10-17 | 2013-09-17 | 2013-08-17 | 2013-07-17 | 2013-06-17 | 2013-05-17 | 2013-04-17 | 2013-03-17 | 2013-02-17 | 2013-01-17 | 2012-12-17 | 2012-11-17 | 2012-10-17 | 2012-09-17 | 2012-08-17 | 2012-07-17 | 2012-06-17 | 2012-05-17 | 2012-04-17 | 2012-03-17 | 2012-02-17 | 2012-01-17 | 2011-12-17 | 2011-11-17 | 2011-10-17 | 2011-09-17 | 2011-08-17 | 2011-07-17 | 2011-06-17 | 2011-05-17 | 2011-04-17 | 2011-03-17 | 2011-02-17 | 2011-01-17 | 2010-12-17 | 2010-11-17 | 2010-10-17 | 2010-09-17 | 2010-08-17 | 2010-07-17 | 2010-06-17 | 2010-05-17 | 2010-04-17 | 2010-03-17 | 2010-02-17 | 2010-01-17 | 2009-12-17 | 2009-11-17 | 2009-10-17 | 2009-09-17 | 2009-08-17 | 2009-07-17 | 2009-06-17 | 2009-05-17 | 2009-04-17 | 2009-03-17 | 2009-02-17 | 2009-01-17 | 2008-12-17 | 2008-11-17 | 2008-10-17 | 2008-09-17 | 2008-08-17 | 2008-07-17 | 2008-06-17 | 2008-05-17 | 2008-04-17 | 2008-03-17 | 2008-02-17 | 2008-01-17 | 2007-12-17 | 2007-11-17 | 2007-10-17 | 2007-09-17 | 2007-08-17 | 2007-07-17 | 2007-06-17 | 2007-05-17 | 2007-04-17 | 2007-03-17 | 2007-02-17 | 2007-01-17 | 2006-12-17 | 2006-11-17 | 2006-10-17 | 2006-09-17 | 2006-08-17 | 2006-07-17 | 2006-06-17 | 2006-05-17 | 2006-04-17 | 2006-03-17 | 2006-02-17 | 2006-01-17 | 2005-12-17 | 2005-11-17 | 2005-10-17 | 2005-09-17 | 2005-08-17 | 2005-07-17 | 2005-06-17 | 2005-05-17 | 2005-04-17 | 2005-03-17 | 2005-02-17 | 2005-01-17 | 2004-12-17 | 2004-11-17 | 2004-10-17 | 2004-09-17 | 2004-08-17 | 2004-07-17 | 2004-06-17 | 2004-05-17 | 2004-04-17 | 2004-03-17 | 2004-02-17 | 2004-01-17 | 2003-12-17 | 2003-11-17 | 2003-10-17 | 2003-09-17 | 2003-08-17 | 2003-07-17 | 2003-06-17 | 2003-05-17 | 2003-04-17 | 2003-03-17 | 2003-02-17 | 2003-01-17 | 2002-12-17 | 2002-11-17 | 2002-10-17 | 2002-09-17 | 2002-08-17 | 2002-07-17 | 2002-06-17 | 2002-05-17 | 2002-04-17 | 2002-03-17 | 2002-02-17 | 2002-01-17 | 2001-12-17 | 2001-11-17 | 2001-10-17 | 2001-09-17 | 2001-08-17 | 2001-07-17 | 2001-06-17 | 2001-05-17 | 2001-04-17 | 2001-03-17 | 2001-02-17 | 2001-01-17 | 2000-12-17 | 2000-11-17 | 2000-10-17 | 2000-09-17 | 2000-08-17 | 2000-07-17 | 2000-06-17 | 2000-05-17 | 2000-04-17 | 2000-03-17 | 2000-02-17 | 2000-01-17 | 1999-12-17

Key: archive Value: 2015-05-01

Key: id Value: 543333

MESSAGE
DATE 2015-05-13
FROM Ruben Safir
SUBJECT Subject: [NYLXS - HANGOUT] Malloc systemtap probes: an example
From owner-hangout-outgoing-at-mrbrklyn.com Wed May 13 22:15:46 2015
Return-Path:
X-Original-To: archive-at-mrbrklyn.com
Delivered-To: archive-at-mrbrklyn.com
Received: by mrbrklyn.com (Postfix)
id 95183161162; Wed, 13 May 2015 22:15:45 -0400 (EDT)
Delivered-To: hangout-outgoing-at-mrbrklyn.com
Received: by mrbrklyn.com (Postfix, from userid 28)
id 8055816117A; Wed, 13 May 2015 22:15:45 -0400 (EDT)
Delivered-To: hangout-at-nylxs.com
Received: from mailbackend.panix.com (mailbackend.panix.com [166.84.1.89])
by mrbrklyn.com (Postfix) with ESMTP id 88B95161162;
Wed, 13 May 2015 22:15:21 -0400 (EDT)
Received: from [10.0.0.19] (www.mrbrklyn.com [96.57.23.82])
by mailbackend.panix.com (Postfix) with ESMTPSA id DF33D13043;
Wed, 13 May 2015 22:15:20 -0400 (EDT)
Message-ID: <55540538.3020609-at-panix.com>
Date: Wed, 13 May 2015 22:15:20 -0400
From: Ruben Safir
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0
MIME-Version: 1.0
To: Hangout
Subject: [NYLXS - HANGOUT] Malloc systemtap probes: an example
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Sender: owner-hangout-at-mrbrklyn.com
Precedence: bulk
Reply-To: hangout-at-nylxs.com
[NYLXS: HANGOUT]
X-BeenThere: hangout-at-nylxs.com
X-Mailing-list: hangout-at-nylxs.com
Precedence: list
List-Id: NYLXS General Discussion Forum
List-Unsubscribe:
List-Archive:
List-Post:
List-Help:
List-Subscribe:



Malloc systemtap probes: an example

Posted on January 6, 2015

by Siddhesh


gnu logo
One
feedback I got from my blog post on Understanding malloc behavior using
Systemtap userspace probes

was that I should have included an example script to explain how this
works. Well, better late than never, so here’s an example script. This
script prints some diagnostic information during a program run and also
logs some information to print out a summary at the end. I’ll go through
the script a few related probes at a time.

global sbrk, waits, arenalist, mmap_threshold = 131072, heaplist

First, we declare some globals:

* We record the process heap size in the |sbrk| variable
* We count the number of times a thread had to wait for an arena lock
in the |waits| variable
* We record an association of threads and arena address spaces in the
|arenalist| variable
* We record the mmap threshold in the suitably named |mmap_threshold|
variable. We initialize the value to the known initial value of 128K
* We record the list of heaps in the |heaplist| variable. The address
of the heap is the key to this associative array and the size is the
value

Now, whenever the process heap is increased or decreased, we record the
increment or decrement.

probe process("/lib64/libc.so.6").mark("memory_sbrk_more")
{
sbrk += $arg2
}

probe process("/lib64/libc.so.6").mark("memory_sbrk_less")
{
sbrk -= $arg2
}

Next, we log every time there is a change in the dynamic mmap and trim
thresholds since those values affect how allocations are done. One could
extend this by adding a probe on various allocation functions (malloc,
calloc, etc.) and seeing how future allocations relate to the threshold
in terms of size.

probe process("/lib64/libc.so.6").mark("memory_mallopt_free_dyn_thresholds")
{
printf("%d: New thresholds:: mmap: %ld bytes, trim: %ld bytes\n", tid(), $arg1,
$arg2)
mmap_threshold = $arg1
}

Next, we note every time a new arena is created and also record which
thread is using it. At the same time, a thread may block to access the
arena and we want to record that.

probe process("/lib64/libc.so.6").mark("memory_arena_new")
{
printf ("%d: Created new arena\n", tid())
arenalist[$arg1, tid()] = 1
}

probe process("/usr/lib64/libc.so.6").mark("memory_arena_reuse_wait")
{
waits[tid()]++
}

probe process("/usr/lib64/libc.so.6").mark("memory_arena_reuse")
{
if ($arg2 != 0)
{
printf ("%d: failed to allocate on own arena, trying another\n", tid())
arenalist[$arg1, tid()] = 1
}
}

Next, the good case, where a thread picks up an arena from the free
list. We update the association of that arena with a new thread.

probe process("/usr/lib64/libc.so.6").mark("memory_arena_reuse_free_list")
{
arenalist[$arg1, tid()] = 1
}

And then, when a thread is destroyed, its arena is sent back to the free
list. Find that arena in our records and mark it as unused.

probe process.thread.end
{
/* Find the thread and mark its arena as unused. */
foreach ([a, t] in arenalist)
if (t == tid())
break
arenalist[a, t] = 0
}

Next, we look at arena heap accounting. These probes give the current
size of the heap at all times, so it is easy for us to record it.

probe process("/usr/lib64/libc.so.6").mark("memory_heap_new")
{
printf("%d: New heap\n", tid());
heaplist[$arg1] = $arg2
}

probe process("/usr/lib64/libc.so.6").mark("memory_heap_more")
{
heaplist[$arg1] = $arg2
}

probe process("/usr/lib64/libc.so.6").mark("memory_heap_less")
{
heaplist[$arg1] = $arg2
}

probe process("/usr/lib64/libc.so.6").mark("memory_heap_free")
{
heaplist[$arg1] = 0
}

Finally, at the end of the program, we print out a summary that tells us
how many times a thread had to wait for a lock on an arena, an
association of arenas with their active thread at the time of process
exit, allocated heaps and their sizes, the total size of the process
heap on exit and finally, the mmap threshold on exit.

probe end
{
printf ("Contention: \n")
foreach (t in waits)
printf ("\t%d: %d waits\n", t, waits[t])

print("Active arenas:\n")
foreach ([a, t] in arenalist)
if (arenalist[a, t])
printf ("\t%d -> %p\n", t, a)

print ("Allocated heaps:\n")
foreach (h in heaplist)
if (heaplist[h])
printf ("\t%p -> %ld bytes\n", h, heaplist[h])

printf ("Total sbrk: %ld bytes\n", sbrk)
printf ("Mmap threshold in the end: %ld kb\n", mmap_threshold / 1024)
}

When I run this script on Red Hat Enterprise Linux 7 with gnome-help and
click a couple of links, here is what I get as output:

# stap malloc.stp -c gnome-help
17319: New heap
17319: Created new arena
17320: New heap
17320: Created new arena
17321: New heap
17321: Created new arena
17324: New heap
17324: Created new arena
17325: New heap
17325: Created new arena
17326: New heap
17326: Created new arena
17315: New thresholds:: mmap: 565248 bytes, trim: 1130496 bytes
17329: New heap
17329: Created new arena
17336: New heap
17336: Created new arena
17337: New heap
17337: Created new arena
17335: New heap
17335: Created new arena
17338: New heap
17338: Created new arena
17334: New heap
17334: Created new arena
17315: New thresholds:: mmap: 925696 bytes, trim: 1851392 bytes
17339: New heap
17339: Created new arena
17341: New heap
17341: Created new arena
17342: New heap
17342: Created new arena
17315: New thresholds:: mmap: 1642496 bytes, trim: 3284992 bytes
17315: New thresholds:: mmap: 1642496 bytes, trim: 3284992 bytes
Contention:
Active arenas:
17326 -> 0x7f8a80000020
17328 -> 0x7f8ad4000020
17360 -> 0x7f8b01742760
Allocated heaps:
0x7f8ae4000000 -> 139264 bytes
0x7f8ad8000000 -> 139264 bytes
0x7f8ad0000000 -> 135168 bytes
0x7f8ad4000000 -> 27246592 bytes
0x7f8a88000000 -> 10854400 bytes
0x7f8a80000000 -> 139264 bytes
0x7f8a84000000 -> 135168 bytes
0x7f8a78000000 -> 10919936 bytes
0x7f8a70000000 -> 311296 bytes
0x7f8a74000000 -> 540672 bytes
0x7f8a68000000 -> 352256 bytes
0x7f8a6c000000 -> 335872 bytes
0x7f8a5c000000 -> 5074944 bytes
0x7f8a54000000 -> 6815744 bytes
0x7f8a58000000 -> 5279744 bytes
Total sbrk: 17633280 bytes
Mmap threshold in the end: 1604 kb

This entry was posted in General
, Performance
, RHEL7

and tagged developer ,
glibc , RHEL7
, systemtap
, tools
by Siddhesh
. Bookmark the
permalink
.



2 thoughts on “Malloc systemtap probes: an example”

1.

Pingback: ??????????????? ?????? – 10 | ???????????? ???


2.
Frank Ch. Eigler on January 30, 2015 at 9:00 pm

said:

This script is included in systemtap’s sample library.

https://sourceware.org/systemtap/examples/index.html#memory/glibc-malloc.stp

Liked by 1 person


Reply ?




Leave a Reply

Gravatar
Email (required) (Address never made public)
Name (required)
Website

Notify me of new comments via email.

  1. 2015-05-02 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Great Article on Software Concordance program writing
  2. 2015-05-02 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Excellent article on Virtual Paging and OS memory
  3. 2015-05-02 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Semephores and what the heck are those things?
  4. 2015-05-03 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] digital advert fraud
  5. 2015-05-03 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Go Language tutorials
  6. 2015-05-03 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Journal Club meeting
  7. 2015-05-03 Elfen Magix <elfen_magix-at-yahoo.com> Re: [NYLXS - HANGOUT] Journal Club meeting
  8. 2015-05-03 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Journal Club meeting
  9. 2015-05-03 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Re: Journal Article
  10. 2015-05-03 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] Journal Club meeting
  11. 2015-05-03 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Journal Club meeting
  12. 2015-05-03 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] Journal Club meeting
  13. 2015-05-03 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] Journal Club meeting
  14. 2015-05-03 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Journal Club meeting
  15. 2015-05-03 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] Journal Club meeting
  16. 2015-05-03 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Movie of the week
  17. 2015-05-03 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Journal Club meeting
  18. 2015-05-03 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] Journal Club meeting
  19. 2015-05-04 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] Journal Club meeting
  20. 2015-05-04 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Journal Club meeting
  21. 2015-05-04 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: a second look at storage and pointer fundamentals
  22. 2015-05-04 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: Kernel Scheduling and wait queues
  23. 2015-05-04 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: [Perlweekly] #197 - YAPC::EU Master classes - talks - hackathons
  24. 2015-05-04 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: [ Khal Tiferes Yaakov ] Lag B'Omer in Passaic
  25. 2015-05-05 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Nice possible project for NYLXS or others
  26. 2015-05-06 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: Tomorrow: You're hosting "Journal Club Meetings"
  27. 2015-05-08 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Things to study over the summer
  28. 2015-05-08 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Things to study over the summer
  29. 2015-05-08 adrba-at-nyct.net Re: [NYLXS - HANGOUT] Things to study over the summer
  30. 2015-05-08 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: Re: Kernel Scheduler and wiat queues
  31. 2015-05-09 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Movies of the Week
  32. 2015-05-10 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] scheduler Slides
  33. 2015-05-10 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Re: [LIU Comp Sci] scheduler Slides
  34. 2015-05-11 Ruben Safir <ruben.safir-at-my.liu.edu> Subject: [NYLXS - HANGOUT] Re: [LIU Comp Sci] scheduler Slides
  35. 2015-05-11 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Re: [LIU Comp Sci] scheduler Slides
  36. 2015-05-12 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] jobs
  37. 2015-05-12 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Job sound like this evenings lectures
  38. 2015-05-12 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] LAMP Jobs
  39. 2015-05-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] April Journal is Available
  40. 2015-05-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: Tomorrow: You and 256 others are going to "Btrfs"
  41. 2015-05-13 mrbrklyn-at-panix.com Subject: [NYLXS - HANGOUT] [member-at-linkedin.com: RE: April Journal is Available]
  42. 2015-05-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [mrbrklyn-at-panix.com: Re: [NYLXS - HANGOUT] Things to study over the
  43. 2015-05-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Malloc systemtap probes: an example
  44. 2015-05-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Stackiq - Educational Program
  45. 2015-05-14 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Weekly Education Meeting
  46. 2015-05-17 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] ssh hack
  47. 2015-05-18 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: [manjaro-general] Manjaro 0.8.13-rc1 released
  48. 2015-05-18 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: [Perlweekly] #199 - Rust 1.0 is out!
  49. 2015-05-18 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Check This Out
  50. 2015-05-18 adrba-at-nyct.net Re: [NYLXS - HANGOUT] Check This Out
  51. 2015-05-18 Elfen Magix <elfen_magix-at-yahoo.com> Re: [NYLXS - HANGOUT] Check This Out
  52. 2015-05-18 mrbrklyn-at-panix.com Subject: [NYLXS - HANGOUT] [jkeen-at-verizon.net: ny.pm Technical Meeting Wed May 20 6:15 pm]
  53. 2015-05-19 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] Check This Out
  54. 2015-05-19 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] copyright vrs free speach on utube
  55. 2015-05-19 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Check This Out
  56. 2015-05-19 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Check This Out
  57. 2015-05-19 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Weapons of Mass Destruction my ass
  58. 2015-05-19 Robert Menes <viewtiful.icchan-at-gmail.com> Re: [NYLXS - HANGOUT] Check This Out
  59. 2015-05-19 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] NYLXS
  60. 2015-05-19 Rick Moen <rick-at-linuxmafia.com> Re: [NYLXS - HANGOUT] Check This Out
  61. 2015-05-20 Ruben Safir <mrbrklyn-at-panix.com> Re: [NYLXS - HANGOUT] Check This Out
  62. 2015-05-21 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Free Software Business
  63. 2015-05-25 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Summer NYLXS Study Schedule
  64. 2015-05-26 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] C++ maheim
  65. 2015-05-26 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: IEEE NY Meeting Announcement
  66. 2015-05-27 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Plane Hacking
  67. 2015-05-27 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Oracle attach on Android Copyright
  68. 2015-05-28 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] adblock is legal... who knew?
  69. 2015-05-28 mrbrklyn-at-panix.com Subject: [NYLXS - HANGOUT] Steeley Dan in NYC
  70. 2015-05-28 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: privacy and data security
  71. 2015-05-28 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: Please give to the Houston Relief Fund
  72. 2015-05-28 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Fwd: Please give to the Houston Relief Fund
  73. 2015-05-28 mrbrklyn-at-panix.com Subject: [NYLXS - HANGOUT] [ruben-at-www.mrbrklyn.com: Linux 1 Book]
  74. 2015-05-29 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] Minting Money with Free software
  75. 2015-05-31 Ruben Safir <mrbrklyn-at-panix.com> Subject: [NYLXS - HANGOUT] anyone familiar with STM?

NYLXS are Do'ers and the first step of Doing is Joining! Join NYLXS and make a difference in your community today!