Sun Feb 28 08:09:43 2021
EVENTS
 FREE
SOFTWARE
INSTITUTE

POLITICS
JOBS
MEMBERS'
CORNER

MAILING
LIST

NYLXS Mailing Lists and Archives
NYLXS Members have a lot to say and share but we don't keep many secrets. Join the Hangout Mailing List and say your peice.

DATE 2015-04-01

LEARN

2021-02-28 | 2021-01-28 | 2020-12-28 | 2020-11-28 | 2020-10-28 | 2020-09-28 | 2020-08-28 | 2020-07-28 | 2020-06-28 | 2020-05-28 | 2020-04-28 | 2020-03-28 | 2020-02-28 | 2020-01-28 | 2019-12-28 | 2019-11-28 | 2019-10-28 | 2019-09-28 | 2019-08-28 | 2019-07-28 | 2019-06-28 | 2019-05-28 | 2019-04-28 | 2019-03-28 | 2019-02-28 | 2019-01-28 | 2018-12-28 | 2018-11-28 | 2018-10-28 | 2018-09-28 | 2018-08-28 | 2018-07-28 | 2018-06-28 | 2018-05-28 | 2018-04-28 | 2018-03-28 | 2018-02-28 | 2018-01-28 | 2017-12-28 | 2017-11-28 | 2017-10-28 | 2017-09-28 | 2017-08-28 | 2017-07-28 | 2017-06-28 | 2017-05-28 | 2017-04-28 | 2017-03-28 | 2017-02-28 | 2017-01-28 | 2016-12-28 | 2016-11-28 | 2016-10-28 | 2016-09-28 | 2016-08-28 | 2016-07-28 | 2016-06-28 | 2016-05-28 | 2016-04-28 | 2016-03-28 | 2016-02-28 | 2016-01-28 | 2015-12-28 | 2015-11-28 | 2015-10-28 | 2015-09-28 | 2015-08-28 | 2015-07-28 | 2015-06-28 | 2015-05-28 | 2015-04-28 | 2015-03-28 | 2015-02-28 | 2015-01-28 | 2014-12-28 | 2014-11-28 | 2014-10-28

Key: Value:

Key: Value:

MESSAGE
DATE 2015-04-17
FROM Ruben
SUBJECT Subject: [LIU Comp Sci] more advanced coding musings ...
From owner-learn-outgoing-at-mrbrklyn.com Fri Apr 17 20:31:01 2015
Return-Path:
X-Original-To: archive-at-mrbrklyn.com
Delivered-To: archive-at-mrbrklyn.com
Received: by mrbrklyn.com (Postfix)
id E92A1161190; Fri, 17 Apr 2015 20:31:00 -0400 (EDT)
Delivered-To: learn-outgoing-at-mrbrklyn.com
Received: by mrbrklyn.com (Postfix, from userid 28)
id D73371612DF; Fri, 17 Apr 2015 20:31:00 -0400 (EDT)
Delivered-To: learn-at-nylxs.com
Received: from mail-qg0-f46.google.com (mail-qg0-f46.google.com [209.85.192.46])
by mrbrklyn.com (Postfix) with ESMTP id 05033161190
for ; Fri, 17 Apr 2015 20:30:34 -0400 (EDT)
Received: by qgeb100 with SMTP id b100so29507706qge.3
for ; Fri, 17 Apr 2015 17:30:33 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20130820;
h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to
:subject:content-type:content-transfer-encoding;
bh=nEmnwpFBaw1Ftnw/UHEee2OYkICsiXDQD3INokov0LM=;
b=Q9caSEPGz+tXgQA7CinHz1lIanwPozbXuXMLEK9sFJcjjjF/WXiSbdNQ3/zlLj5/g1
Itij64ahUggIO94wnQmhiXRKmNml3W4RjOboItLAi+59i1Lfbh3tCkw2BG4VZn+xY4PC
x1uj/hTURvLO7VC2L8Ltko4607SUrRXZKI4kD37S/dwSFoFNpom3HHjseMmnnUBhw6hg
N1557S/yijYZSjPcEPx+veqnCML5E219BQAratWR+VhTv2dgNbiFqdz2Bk1gZGOPO27P
12AgXn4W/dFtBuvN3MaNM2ZNgSDukdzWCIjM4eQkmIRVQFFwGsDxg79AAK27VuheWKme
k60g==
X-Gm-Message-State: ALoCoQnBAmYacmMZHx2CTqixOxRx8paOyXgd3/UQDV425Mq99R/3yWdZZCAFonb2xKIJ49m/n+7/
X-Received: by 10.140.151.75 with SMTP id 72mr6810450qhx.89.1429317033012;
Fri, 17 Apr 2015 17:30:33 -0700 (PDT)
Received: from [10.0.0.19] (www.mrbrklyn.com. [96.57.23.82])
by mx.google.com with ESMTPSA id q34sm9273470qkq.4.2015.04.17.17.30.32
for
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Fri, 17 Apr 2015 17:30:32 -0700 (PDT)
Message-ID: <5531A5A7.8030501-at-my.liu.edu>
Date: Fri, 17 Apr 2015 20:30:31 -0400
From: Ruben
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0
MIME-Version: 1.0
To: learn-at-nylxs.com
Subject: [LIU Comp Sci] more advanced coding musings ...
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Sender: owner-learn-at-mrbrklyn.com
Precedence: bulk
Reply-To: learn-at-mrbrklyn.com


Binder vs. kdbus

A lot of people have asked about replacing Android’s binder
code with kdbus. I originally thought
this could be done, but as time has gone by, I’ve come to the conclusion
that this will not happen with the first version of kdbus, and possibly
can never happen.

First off, go read that link describing binder
that I pointed to above, especially
all of the links to different resources from that page. That should give
you more than you ever wanted to know about binder.


Short answer

Binder is bound to the CPU, D-Bus (and hence kdbus), is bound to RAM.


Long answer


Binder

Binder is an interface that Android uses to provide synchronous calling
(CPU) from one task to a thread of another task. There is no queueing
involved in these calls, other than the caller process is suspended
until the answering process returns. RAM is not interesting besides the
fact that it is used to share the data between the different callers.
The fact that the caller process gives up its CPU slice to the answering
process is key for how Android works with the binder library.

This is just like a syscall, and it behaves a lot like a mutex. The
communicating processes are directly connected to each other. There is
an upper limit of how many different processes can be using binder at
once, and I think it’s around 16 for most systems.


D-Bus

D-Bus is asynchronous, it queues (RAM) messages, keeps the messages in
order, and the receiver dequeues the messages. The CPU does not matter
at all other than it is used to do the asynchronous work of passing the
RAM around between the different processes.

This is a lot like network communication protocols. It is a very
“disconnected” communication method between processes. The upper limit
of message sizes and numbers is usually around 8Mb per connection and a
normal message is around 200-800 bytes.


Binder

The model of Binder was created for a microkernel-like device (side
note, go read this wonderful article
about the history of Danger
written by one of the engineers at that company for a glimpse into where
the Android internals came from, binder included.) The model of binder
is very limited, inflexible in its use-cases, but very powerful and
/extremely/ low-overhead and fast. Binder ensures that the same CPU
timeslice will go from the calling process into the called process’s
thread, and then come back into the caller when finished. There is
almost no scheduling involved, and is much like a syscall into the
kernel that does work for the calling process. This interface is very
well suited for cheap devices with almost no RAM and very low CPU resources.

So, for systems like Android, binder makes total sense, especially given
the history of it and where it was designed to be used.


D-Bus

D-Bus is a create-store-forward, compose reply and then
create-store-forward messaging model which is more complex than binder,
but because of that, it is extremely flexible, versatile, network
transparent, much easier to manage, and very easy to let fully untrusted
peers take part of the communication model (hint, never let this happen
with binder, or bad things will happen…) D-Bus can scale up to huge
amounts of data, and with the implementation of kdbus it is possible to
pass gigabytes of buffers to every connection on the bus if you really
wanted to. CPU-wise, it is not as efficient as binder, but is a much
better general-purpose solution for general-purpose machines and workloads.


CPU vs. RAM

Yes, it’s an over simplification of a different set of complex IPC
methods, but these 3 words should help you explain the differences
between binder and D-Bus and why kdbus isn’t going to be able to easily
replace binder anytime soon.


Never say never

Ok, before you start to object to the above statements, yes, we could
add functionality to kdbus to have some blocking ioctl calls that
implement something like: |write question -> block for reply and read
reply one answer| for the request side, and then on the server side do:
|write answer -> block in read| That would get kdbus a tiny bit closer
to the binder model, by queueing stuff in RAM instead of relying on a
thread pool.

That might work, but would require a lot of work on the binder library
side in Android, and as a very limited number of people have write
access to that code (they all can be counted on one hand), and it’s a
non-trivial amount of work for a core function of Android that is
working very well today, I don’t know if it will ever happen.

But anything is possible, it’s just software you know…


Thanks

Many thanks to Kay Sievers who came up with the CPU vs. RAM description
of binder and D-Bus and whose email I pretty much just copied into this
post. Also thanks to Kay and Lennart for taking the time and energy to
put up with my silly statements about how kdbus could replace binder,
and totally proving me wrong, sorry for having you spend so much time on
this, but I now know you are right.

Also thanks to Daniel Mack and Kay for doing so much work on the kdbus
kernel code, that I don’t think any of my original implementation is
even present anymore, which is probably a good thing. Also thanks to
Tejun Heo for help with the memfd implementation and cgroups help in kdbus.


Binary Blobs to C Structures


Sep 11th, 2013

Sometimes you don’t have access to vim’s wonderful xxd
tool, and you need to use
it to generate some .c code based on a binary file. This happened to me
recently when packaging up the EFI signing tools for Gentoo.

Adding a build requirement of vim for a single autogenerated file was
not an option for some users, so I created a perl version of the |xxd
-i| command line tool.

This works because everyone has perl in their build systems, whether
they like it or not. Instead of burying it in the efitools package,
here’s a copy of it for others to use if they want/need it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48



|#!/usr/bin/env perl
#
# xxdi.pl - perl implementation of 'xxd -i' mode
#
# Copyright 2013 Greg Kroah-Hartman
# Copyright 2013 Linux Foundation
#
# Released under the GPLv2.
#
# Implements the "basic" functionality of 'xxd -i' in perl to keep build
# systems from having to build/install/rely on vim-core, which not all
# distros want to do. But everyone has perl, so use it instead.

use strict;
use warnings;
use File::Slurp qw(slurp);

my $indata = slurp(-at-ARGV ? $ARGV[0] : \*STDIN);
my $len_data = length($indata);
my $num_digits_per_line = 12;
my $var_name;
my $outdata;

# Use the variable name of the file we read from, converting '/' and '.
# to '_', or, if this is stdin, just use "stdin" as the name.
if (-at-ARGV) {
$var_name = $ARGV[0];
$var_name =~ s/\//_/g;
$var_name =~ s/\./_/g;
} else {
$var_name = "stdin";
}

$outdata .= "unsigned char $var_name\[] = {";

# trailing ',' is acceptable, so instead of duplicating the logic for
# just the last character, live with the extra ','.
for (my $key= 0; $key < $len_data; $key++) {
if ($key % $num_digits_per_line == 0) {
$outdata .= "\n\t";
}
$outdata .= sprintf("0x%.2x, ", ord(substr($indata, $key, 1)));
}

$outdata .= "\n};\nunsigned int $var_name\_len = $len_data;\n";

binmode STDOUT;
print {*STDOUT} $outdata;
|

Yes, I know I write perl code like a C programmer, that’s not an insult
to me.



  1. 2015-04-01 Ruben <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] external paths
  2. 2015-04-01 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] omp pthread madness
  3. 2015-04-02 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: Fw: Perl Developer / Linux SysAdmin Opportunities in South Florida
  4. 2015-04-02 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: omp pthread madness
  5. 2015-04-02 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: omp pthread madness
  6. 2015-04-02 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: [New discussion] which technologies/skills are good to have
  7. 2015-04-02 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Re: omp pthread madness
  8. 2015-04-05 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Re: Depths and heights
  9. 2015-04-09 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Re: Kernel thread scheduling
  10. 2015-04-09 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Re: Kernel thread scheduling
  11. 2015-04-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] BASh Shell Scripting
  12. 2015-04-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Chapter 5 Monitor
  13. 2015-04-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: [Perlweekly] #194 - The CPAN PR Challenge Now Has a Mini-Me!
  14. 2015-04-13 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Kernel Hacking Resources
  15. 2015-04-14 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: Invitation: Vintage Computer Festival
  16. 2015-04-14 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] hashing slots sizes
  17. 2015-04-15 Ruben <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] BASh Shell Scripting
  18. 2015-04-15 Maneesh Kongara <maneeshkongara-at-gmail.com> Re: [LIU Comp Sci] BASh Shell Scripting
  19. 2015-04-15 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: Monday: Join 121 HackerNestologists at "HackerNest NYC April
  20. 2015-04-15 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: You're confirmed for: NYLUG Open hacker hours
  21. 2015-04-15 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Hash Tables Examples
  22. 2015-04-15 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Re: hashing quadradic probes
  23. 2015-04-15 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] [ruben-at-mrbrklyn.com: Re: Heap Management Problem]
  24. 2015-04-16 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: Re: virtualbox
  25. 2015-04-16 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] linsched
  26. 2015-04-16 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] scheduler project thus far
  27. 2015-04-17 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Come Sunday Morning
  28. 2015-04-17 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Kernel Scheduling Project
  29. 2015-04-17 Ruben <ruben.safir-at-my.liu.edu> Subject: [LIU Comp Sci] more advanced coding musings ...
  30. 2015-04-19 Tony Genao <tony.genao-at-my.liu.edu> Re: [LIU Comp Sci] Fwd: [NYLXS - HANGOUT] Summers Here
  31. 2015-04-19 Ruben Safir <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] Fwd: [NYLXS - HANGOUT] Summers Here
  32. 2015-04-19 Tony Genao <tony.genao-at-my.liu.edu> Re: [LIU Comp Sci] Fwd: [NYLXS - HANGOUT] Summers Here
  33. 2015-04-19 Ruben <mrbrklyn-at-panix.com> Re: [LIU Comp Sci] Fwd: [NYLXS - HANGOUT] Summers Here
  34. 2015-04-19 Tony Genao <tony.genao-at-my.liu.edu> Re: [LIU Comp Sci] Fwd: [NYLXS - HANGOUT] Summers Here
  35. 2015-04-19 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: [NYLXS - HANGOUT] Summers Here
  36. 2015-04-19 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Re: The more I learn the less I know
  37. 2015-04-22 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: Re: wait queues semiphores kernel implementations
  38. 2015-04-22 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] linux locks and mutex
  39. 2015-04-23 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: Re: Kernel Scheduler and wiat queues
  40. 2015-04-25 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: Kernel Scheduling and wait queues
  41. 2015-04-27 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: OS Cpt 9 27 question
  42. 2015-04-27 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Fwd: Tomorrow: You and 30 other Flatirons are going to "Flatiron
  43. 2015-04-28 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Re: DMCA exemption commenting process broken beyond repair
  44. 2015-04-29 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Do you have notes for this question
  45. 2015-04-29 Ruben Safir <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Professional Troll job available
  46. 2015-04-29 Ruben <mrbrklyn-at-panix.com> Subject: [LIU Comp Sci] Re: DMCA exemption commenting process broken beyond repair

NYLXS are Do'ers and the first step of Doing is Joining! Join NYLXS and make a difference in your community today!