Tue Dec 10 02:50:10 2024
EVENTS
 FREE
SOFTWARE
INSTITUTE

POLITICS
JOBS
MEMBERS'
CORNER

MAILING
LIST

NYLXS Mailing Lists and Archives
NYLXS Members have a lot to say and share but we don't keep many secrets. Join the Hangout Mailing List and say your peice.

DATE 2015-05-01

LEARN

2024-12-10 | 2024-11-10 | 2024-10-10 | 2024-09-10 | 2024-08-10 | 2024-07-10 | 2024-06-10 | 2024-05-10 | 2024-04-10 | 2024-03-10 | 2024-02-10 | 2024-01-10 | 2023-12-10 | 2023-11-10 | 2023-10-10 | 2023-09-10 | 2023-08-10 | 2023-07-10 | 2023-06-10 | 2023-05-10 | 2023-04-10 | 2023-03-10 | 2023-02-10 | 2023-01-10 | 2022-12-10 | 2022-11-10 | 2022-10-10 | 2022-09-10 | 2022-08-10 | 2022-07-10 | 2022-06-10 | 2022-05-10 | 2022-04-10 | 2022-03-10 | 2022-02-10 | 2022-01-10 | 2021-12-10 | 2021-11-10 | 2021-10-10 | 2021-09-10 | 2021-08-10 | 2021-07-10 | 2021-06-10 | 2021-05-10 | 2021-04-10 | 2021-03-10 | 2021-02-10 | 2021-01-10 | 2020-12-10 | 2020-11-10 | 2020-10-10 | 2020-09-10 | 2020-08-10 | 2020-07-10 | 2020-06-10 | 2020-05-10 | 2020-04-10 | 2020-03-10 | 2020-02-10 | 2020-01-10 | 2019-12-10 | 2019-11-10 | 2019-10-10 | 2019-09-10 | 2019-08-10 | 2019-07-10 | 2019-06-10 | 2019-05-10 | 2019-04-10 | 2019-03-10 | 2019-02-10 | 2019-01-10 | 2018-12-10 | 2018-11-10 | 2018-10-10 | 2018-09-10 | 2018-08-10 | 2018-07-10 | 2018-06-10 | 2018-05-10 | 2018-04-10 | 2018-03-10 | 2018-02-10 | 2018-01-10 | 2017-12-10 | 2017-11-10 | 2017-10-10 | 2017-09-10 | 2017-08-10 | 2017-07-10 | 2017-06-10 | 2017-05-10 | 2017-04-10 | 2017-03-10 | 2017-02-10 | 2017-01-10 | 2016-12-10 | 2016-11-10 | 2016-10-10 | 2016-09-10 | 2016-08-10 | 2016-07-10 | 2016-06-10 | 2016-05-10 | 2016-04-10 | 2016-03-10 | 2016-02-10 | 2016-01-10 | 2015-12-10 | 2015-11-10 | 2015-10-10 | 2015-09-10 | 2015-08-10 | 2015-07-10 | 2015-06-10 | 2015-05-10 | 2015-04-10 | 2015-03-10 | 2015-02-10 | 2015-01-10 | 2014-12-10 | 2014-11-10 | 2014-10-10

Key: Value:

Key: Value:

MESSAGE
DATE 2015-05-13
FROM Ruben Safir
SUBJECT Subject: [LIU Comp Sci] Malloc systemtap probes: an example
From owner-learn-outgoing-at-mrbrklyn.com Wed May 13 22:15:45 2015
Return-Path:
X-Original-To: archive-at-mrbrklyn.com
Delivered-To: archive-at-mrbrklyn.com
Received: by mrbrklyn.com (Postfix)
id B334B161165; Wed, 13 May 2015 22:15:45 -0400 (EDT)
Delivered-To: learn-outgoing-at-mrbrklyn.com
Received: by mrbrklyn.com (Postfix, from userid 28)
id A30B916116B; Wed, 13 May 2015 22:15:45 -0400 (EDT)
Delivered-To: learn-at-nylxs.com
Received: from mailbackend.panix.com (mailbackend.panix.com [166.84.1.89])
by mrbrklyn.com (Postfix) with ESMTP id 88B95161162;
Wed, 13 May 2015 22:15:21 -0400 (EDT)
Received: from [10.0.0.19] (www.mrbrklyn.com [96.57.23.82])
by mailbackend.panix.com (Postfix) with ESMTPSA id DF33D13043;
Wed, 13 May 2015 22:15:20 -0400 (EDT)
Message-ID: <55540538.3020609-at-panix.com>
Date: Wed, 13 May 2015 22:15:20 -0400
From: Ruben Safir
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0
MIME-Version: 1.0
To: Hangout
Subject: [LIU Comp Sci] Malloc systemtap probes: an example
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Sender: owner-learn-at-mrbrklyn.com
Precedence: bulk
Reply-To: learn-at-mrbrklyn.com


Malloc systemtap probes: an example

Posted on January 6, 2015

by Siddhesh


gnu logo
One
feedback I got from my blog post on Understanding malloc behavior using
Systemtap userspace probes

was that I should have included an example script to explain how this
works. Well, better late than never, so here’s an example script. This
script prints some diagnostic information during a program run and also
logs some information to print out a summary at the end. I’ll go through
the script a few related probes at a time.

global sbrk, waits, arenalist, mmap_threshold = 131072, heaplist

First, we declare some globals:

* We record the process heap size in the |sbrk| variable
* We count the number of times a thread had to wait for an arena lock
in the |waits| variable
* We record an association of threads and arena address spaces in the
|arenalist| variable
* We record the mmap threshold in the suitably named |mmap_threshold|
variable. We initialize the value to the known initial value of 128K
* We record the list of heaps in the |heaplist| variable. The address
of the heap is the key to this associative array and the size is the
value

Now, whenever the process heap is increased or decreased, we record the
increment or decrement.

probe process("/lib64/libc.so.6").mark("memory_sbrk_more")
{
sbrk += $arg2
}

probe process("/lib64/libc.so.6").mark("memory_sbrk_less")
{
sbrk -= $arg2
}

Next, we log every time there is a change in the dynamic mmap and trim
thresholds since those values affect how allocations are done. One could
extend this by adding a probe on various allocation functions (malloc,
calloc, etc.) and seeing how future allocations relate to the threshold
in terms of size.

probe process("/lib64/libc.so.6").mark("memory_mallopt_free_dyn_thresholds")
{
printf("%d: New thresholds:: mmap: %ld bytes, trim: %ld bytes\n", tid(), $arg1,
$arg2)
mmap_threshold = $arg1
}

Next, we note every time a new arena is created and also record which
thread is using it. At the same time, a thread may block to access the
arena and we want to record that.

probe process("/lib64/libc.so.6").mark("memory_arena_new")
{
printf ("%d: Created new arena\n", tid())
arenalist[$arg1, tid()] = 1
}

probe process("/usr/lib64/libc.so.6").mark("memory_arena_reuse_wait")
{
waits[tid()]++
}

probe process("/usr/lib64/libc.so.6").mark("memory_arena_reuse")
{
if ($arg2 != 0)
{
printf ("%d: failed to allocate on own arena, trying another\n", tid())
arenalist[$arg1, tid()] = 1
}
}

Next, the good case, where a thread picks up an arena from the free
list. We update the association of that arena with a new thread.

probe process("/usr/lib64/libc.so.6").mark("memory_arena_reuse_free_list")
{
arenalist[$arg1, tid()] = 1
}

And then, when a thread is destroyed, its arena is sent back to the free
list. Find that arena in our records and mark it as unused.

probe process.thread.end
{
/* Find the thread and mark its arena as unused. */
foreach ([a, t] in arenalist)
if (t == tid())
break
arenalist[a, t] = 0
}

Next, we look at arena heap accounting. These probes give the current
size of the heap at all times, so it is easy for us to record it.

probe process("/usr/lib64/libc.so.6").mark("memory_heap_new")
{
printf("%d: New heap\n", tid());
heaplist[$arg1] = $arg2
}

probe process("/usr/lib64/libc.so.6").mark("memory_heap_more")
{
heaplist[$arg1] = $arg2
}

probe process("/usr/lib64/libc.so.6").mark("memory_heap_less")
{
heaplist[$arg1] = $arg2
}

probe process("/usr/lib64/libc.so.6").mark("memory_heap_free")
{
heaplist[$arg1] = 0
}

Finally, at the end of the program, we print out a summary that tells us
how many times a thread had to wait for a lock on an arena, an
association of arenas with their active thread at the time of process
exit, allocated heaps and their sizes, the total size of the process
heap on exit and finally, the mmap threshold on exit.

probe end
{
printf ("Contention: \n")
foreach (t in waits)
printf ("\t%d: %d waits\n", t, waits[t])

print("Active arenas:\n")
foreach ([a, t] in arenalist)
if (arenalist[a, t])
printf ("\t%d -> %p\n", t, a)

print ("Allocated heaps:\n")
foreach (h in heaplist)
if (heaplist[h])
printf ("\t%p -> %ld bytes\n", h, heaplist[h])

printf ("Total sbrk: %ld bytes\n", sbrk)
printf ("Mmap threshold in the end: %ld kb\n", mmap_threshold / 1024)
}

When I run this script on Red Hat Enterprise Linux 7 with gnome-help and
click a couple of links, here is what I get as output:

# stap malloc.stp -c gnome-help
17319: New heap
17319: Created new arena
17320: New heap
17320: Created new arena
17321: New heap
17321: Created new arena
17324: New heap
17324: Created new arena
17325: New heap
17325: Created new arena
17326: New heap
17326: Created new arena
17315: New thresholds:: mmap: 565248 bytes, trim: 1130496 bytes
17329: New heap
17329: Created new arena
17336: New heap
17336: Created new arena
17337: New heap
17337: Created new arena
17335: New heap
17335: Created new arena
17338: New heap
17338: Created new arena
17334: New heap
17334: Created new arena
17315: New thresholds:: mmap: 925696 bytes, trim: 1851392 bytes
17339: New heap
17339: Created new arena
17341: New heap
17341: Created new arena
17342: New heap
17342: Created new arena
17315: New thresholds:: mmap: 1642496 bytes, trim: 3284992 bytes
17315: New thresholds:: mmap: 1642496 bytes, trim: 3284992 bytes
Contention:
Active arenas:
17326 -> 0x7f8a80000020
17328 -> 0x7f8ad4000020
17360 -> 0x7f8b01742760
Allocated heaps:
0x7f8ae4000000 -> 139264 bytes
0x7f8ad8000000 -> 139264 bytes
0x7f8ad0000000 -> 135168 bytes
0x7f8ad4000000 -> 27246592 bytes
0x7f8a88000000 -> 10854400 bytes
0x7f8a80000000 -> 139264 bytes
0x7f8a84000000 -> 135168 bytes
0x7f8a78000000 -> 10919936 bytes
0x7f8a70000000 -> 311296 bytes
0x7f8a74000000 -> 540672 bytes
0x7f8a68000000 -> 352256 bytes
0x7f8a6c000000 -> 335872 bytes
0x7f8a5c000000 -> 5074944 bytes
0x7f8a54000000 -> 6815744 bytes
0x7f8a58000000 -> 5279744 bytes
Total sbrk: 17633280 bytes
Mmap threshold in the end: 1604 kb

This entry was posted in General
, Performance
, RHEL7

and tagged developer ,
glibc , RHEL7
, systemtap
, tools
by Siddhesh
. Bookmark the
permalink
.



2 thoughts on “Malloc systemtap probes: an example”

1.

Pingback: ??????????????? ?????? – 10 | ???????????? ???


2.
Frank Ch. Eigler on January 30, 2015 at 9:00 pm

said:

This script is included in systemtap’s sample library.

https://sourceware.org/systemtap/examples/index.html#memory/glibc-malloc.stp

Liked by 1 person


Reply ?




Leave a Reply

Gravatar
Email (required) (Address never made public)
Name (required)
Website

Notify me of new comments via email.

  1. 2015-05-02 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] debuging methods
  2. 2015-05-02 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Excellent article on Virtual Paging and OS memory
  3. 2015-05-02 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Great Article on Software Concordance program writing
  4. 2015-05-02 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Semephores and what the heck are those things?
  5. 2015-05-03 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Go Language tutorials
  6. 2015-05-04 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] ACL and beyound security in linux
  7. 2015-05-04 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: [Perlweekly] #197 - YAPC::EU Master classes - talks - hackathons
  8. 2015-05-05 Ruben <ruben.safir-at-my.liu.edu> Re: [LIU Comp Sci] Fibonacci trees
  9. 2015-05-05 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Examination Question for Allogorthims
  10. 2015-05-05 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fibonacci trees
  11. 2015-05-05 Ruben <ruben.safir-at-my.liu.edu> Subject: [LIU Comp Sci] Fwd: Internships with Oracle, Amtrak, The Nature Conservancy & more
  12. 2015-05-05 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Nice possible project for NYLXS or others
  13. 2015-05-06 Ruben Safir <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] Fibonacci trees
  14. 2015-05-06 Ruben Safir <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] Fibonacci trees
  15. 2015-05-06 Ruben Safir <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] Fibonacci trees
  16. 2015-05-06 Ruben <ruben.safir-at-my.liu.edu> Subject: [LIU Comp Sci] Fwd: Re: [opensuse] Re: no space left on the device
  17. 2015-05-06 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] hashing multiplication
  18. 2015-05-08 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: Kernel Scheduling and wait queues
  19. 2015-05-08 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: Re: Kernel Scheduler and wiat queues
  20. 2015-05-08 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Re: [NYLXS - HANGOUT] Things to study over the summer
  21. 2015-05-08 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Things to study over the summer
  22. 2015-05-10 Ruben Safir <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] scheduler Slides
  23. 2015-05-10 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] scheduler Slides
  24. 2015-05-11 Ruben Safir <ruben.safir-at-my.liu.edu> Re: [LIU Comp Sci] scheduler Slides
  25. 2015-05-11 Ruben Safir <ruben.safir-at-my.liu.edu> Re: [LIU Comp Sci] scheduler Slides
  26. 2015-05-11 Justin Lau <justinml-at-gmail.com> Re: [LIU Comp Sci] scheduler Slides
  27. 2015-05-11 Justin Lau <justinml-at-gmail.com> Re: [LIU Comp Sci] scheduler Slides
  28. 2015-05-11 Ruben Safir <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] scheduler Slides
  29. 2015-05-11 Ruben Safir <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] scheduler Slides
  30. 2015-05-11 Ruben Safir <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] scheduler Slides
  31. 2015-05-12 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Job sound like this evenings lectures
  32. 2015-05-12 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] jobs
  33. 2015-05-12 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] LAMP Jobs
  34. 2015-05-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] April Journal is Available
  35. 2015-05-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: Tomorrow: You and 256 others are going to "Btrfs"
  36. 2015-05-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Malloc systemtap probes: an example
  37. 2015-05-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Stackiq - Educational Program
  38. 2015-05-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Student Lab and Club House
  39. 2015-05-13 mrbrklyn-at-panix.com Subject: [LIU Comp Sci] [member-at-linkedin.com: RE: April Journal is Available]
  40. 2015-05-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] [mrbrklyn-at-panix.com: Re: [NYLXS - HANGOUT] Things to study over the
  41. 2015-05-14 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Weekly Education Meeting
  42. 2015-05-18 mrbrklyn-at-panix.com Subject: [LIU Comp Sci] [jkeen-at-verizon.net: ny.pm Technical Meeting Wed May 20 6:15 pm]
  43. 2015-05-25 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Summer NYLXS Study Schedule
  44. 2015-05-28 Tony Genao <tony.genao-at-my.liu.edu> Re: [LIU Comp Sci] Summer NYLXS Study Schedule
  45. 2015-05-28 Ruben Safir <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] Summer NYLXS Study Schedule
  46. 2015-05-28 Tony Genao <tony.genao-at-my.liu.edu> Re: [LIU Comp Sci] Summer NYLXS Study Schedule
  47. 2015-05-28 Ruben Safir <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] Summer NYLXS Study Schedule
  48. 2015-05-28 Ruben <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: Re: Programming Position
  49. 2015-05-28 mrbrklyn-at-panix.com Subject: [LIU Comp Sci] [ruben-at-www.mrbrklyn.com: Linux 1 Book]
  50. 2015-05-31 Ruben Safir <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] Summer NYLXS Study Schedule

NYLXS are Do'ers and the first step of Doing is Joining! Join NYLXS and make a difference in your community today!