MESSAGE
DATE | 2021-01-26 |
FROM | Richard Purdie
|
SUBJECT | Re: [Hangout - NYLXS] Future plans for Autotools
|
From hangout-bounces-at-nylxs.com Tue Jan 26 19:01:50 2021 Return-Path: X-Original-To: archive-at-mrbrklyn.com Delivered-To: archive-at-mrbrklyn.com Received: from www2.mrbrklyn.com (www2.mrbrklyn.com [96.57.23.82]) by mrbrklyn.com (Postfix) with ESMTP id A7D34163FB8; Tue, 26 Jan 2021 19:01:49 -0500 (EST) X-Original-To: hangout-at-www2.mrbrklyn.com Delivered-To: hangout-at-www2.mrbrklyn.com Received: by mrbrklyn.com (Postfix, from userid 1000) id 72730163FBA; Tue, 26 Jan 2021 19:01:45 -0500 (EST) Resent-From: Ruben Safir Resent-Date: Tue, 26 Jan 2021 19:01:45 -0500 Resent-Message-ID: <20210127000145.GA7593-at-www2.mrbrklyn.com> Resent-To: hangout-at-mrbrklyn.com X-Original-To: ruben-at-mrbrklyn.com Delivered-To: ruben-at-mrbrklyn.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mrbrklyn.com (Postfix) with ESMTP id 0D121163FBB for ; Tue, 26 Jan 2021 16:31:34 -0500 (EST) Received: from localhost ([::1]:54710 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l4Vvx-0004Hk-G2 for ruben-at-mrbrklyn.com; Tue, 26 Jan 2021 16:31:33 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:51070) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l4VPR-0007bq-AP for autoconf-at-gnu.org; Tue, 26 Jan 2021 15:57:57 -0500 Received: from mail-ed1-x52a.google.com ([2a00:1450:4864:20::52a]:44124) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l4VPN-0007oj-Sj for autoconf-at-gnu.org; Tue, 26 Jan 2021 15:57:56 -0500 Received: by mail-ed1-x52a.google.com with SMTP id c2so21140051edr.11 for ; Tue, 26 Jan 2021 12:57:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=message-id:subject:from:to:date:in-reply-to:user-agent:mime-version :content-transfer-encoding; bh=9SqzP5fgmHhuVCN6rNOp/ZXtG448eQWntPCPv3Q2h4o=; b=IMvjj1xQSao0hcKyr+iL/lRveJmdlLiPo80deVHSwJTQ6faGyDS7L05aspNKnaFqjh Nh+bvr65FCMEIsci14F3t3UIwbGnk0epGOpaOPysIvGZuXDJWj5ZC8yC+OWT+pPRlDaH z+x6F5WbJfzygicz7ymOKLGWR6BcBSYLNJdE4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:date:in-reply-to :user-agent:mime-version:content-transfer-encoding; bh=9SqzP5fgmHhuVCN6rNOp/ZXtG448eQWntPCPv3Q2h4o=; b=uXc3/PltYet9lOCBajvTclRyVlAnx4onrpppkGTBKqwzX8DBGiW52eMylzD1tCOSKU 3sctbrTd1eTqKtUy2mgKkGqOlYykjm2fCxtAbGBemZ880l1aUM+LzC04y4Jl/4VmSN60 Il6teoILOwpcjp7ao6cPfEKg8Dm4e7xY64yJVVLfi1PKO5rJBPcZavRZNuN6MpwLFeBo OkhSTXHXAq4be5WF4hvGPmb8hiDW6DcycXHSc8NGwIvxwtzJi/GQm9SeGDHt+VsS86C+ oeo54OThyVddDOHtoaLAQu6c9g5hsu20uowU1SUGX8xyOeFxfimXhFJ61kzbB+U5s87A j52g== X-Gm-Message-State: AOAM531Ipx5LftxEX8ymnALyl4zkWNUEF6wzDCrCPYjFAK3lugWB6+6K mFWcPwIvX8vgcmmQ3OoskeNQuFo7DNLmlw== X-Google-Smtp-Source: ABdhPJyP33JFDfYcuVxP/oyqSDj+QOtYVlTyYPITqrAQJP1hnluakkHI72bmGdLTwkxaWCXTgzfVfA== X-Received: by 2002:a05:6402:3186:: with SMTP id di6mr6075485edb.45.1611694670686; Tue, 26 Jan 2021 12:57:50 -0800 (PST) Received: from ?IPv6:2001:8b0:aba:5f3c:740b:f7b2:9a90:fbc0? ([2001:8b0:aba:5f3c:740b:f7b2:9a90:fbc0]) by smtp.gmail.com with ESMTPSA id s22sm10193705ejd.106.2021.01.26.12.57.49 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Jan 2021 12:57:50 -0800 (PST) Message-ID: From: Richard Purdie To: autoconf-at-gnu.org Date: Tue, 26 Jan 2021 20:57:46 +0000 In-Reply-To: CAKCAbMiDiBPd3jy2gxMKkuxvCx4VwcAoUta6YXLq8t-sHT13gg-at-mail.gmail.com User-Agent: Evolution 3.38.1-1 MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::52a; envelope-from=richard.purdie-at-linuxfoundation.org; helo=mail-ed1-x52a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Tue, 26 Jan 2021 16:31:09 -0500 X-BeenThere: autoconf-at-gnu.org X-Mailman-Version: 2.1.23 Precedence: list Subject: Re: [Hangout - NYLXS] Future plans for Autotools X-BeenThere: hangout-at-nylxs.com List-Id: NYLXS Tech Talk and Politics List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: hangout-bounces-at-nylxs.com Sender: "Hangout"
Firstly, thanks for the 2.70 release! Yocto Project hasn't switched yet as whilst we tested it and reported issues, there was a last minute change which broke things for us. We'll get that sorted and upgrade.
I am one of the lead developers of the Yocto Project and at its heart is OpenEmbedded which is basically a cross compiling operating system build tool. We primarily target Linux and specialise in allowing complete customisation of the end resulting images.
We try and solve a complex problem, pulling together all the pieces of open source needed for an operating system including the toolchain, cross compiling them and allowing the user to customise them in infinitely different ways. Where the software uses autotools, we actually regenerate the configure scripts and makefiles from the source files before configuring and building the software. We do this since we may need to include arch support that isn't part of the original configure script, or to include fixes we carry to components like libtool, which I'm say to say doesn't work properly cross compiling out the box.
One scary statistic is that if we look at where our builds spend time, 40% of our wall clock build time is spent in "do_configure" tasks, which is mostly running autotools and is single threaded. The compile takes much less time as we can do things in parallel. Of that 40%, about half of it is in autoreconf and half is running configure itself.
I did profile configure and the problem is that its basically a fork bomb. Tons of forks to do trivial shell operations. I have less insight how to speed up the autoreconf side of things as that mostly seems to be in m4, although an m4 deletion party of obsolete things we don't care about has come to mind before.
I like autotools, I've spent a lot of time debugging them but I also know they solve some hard problems. I do wish there were more active releases and an upstream we could engage with and get issues discussed and resolved somehow.
I will observe that I once worked on a Makefile based build system (buildroot) and I then somehow ultimately became the bitbake maintainer which can be thought of as a custom python tool for parsing and dealing with metadata. Over the years I have come to love python for handling our recipes and configuration data. I saw suggestion of using some other language for configure than shell and the idea sounds intriguing, particularly after my positive experiences with python.
In terms of CI, Yocto Project had a challenge there as we have huge possible numbers of configurations. Its taken us a few years but we do now have a CI system I'm quite proud of. In around 6 hours we can take a configuration and build it out from scratch for 4 principle architectures in 32, x32/n32 and 64 bit variants for multiple init systems, kernel versions, libc implementations and much much more. As an idea, here is a link to one of our test reports:
https://autobuilder.yocto.io/pub/non-release/20210120-9/testresults/testresult-report.txt
That does correctly say it attempted about 2.6 million tests. Included there are booting images under qemu for all our principle target architectures and running the test suites included with many pieces of software, including the testsuites with gcc, glibc, binutils, perl and python and the ltp suite. We also cover things like reproducibility.
When we heard about the new release we did run 2.70 under this test matrix and reported issues back so hopefully the release benefited from this indirectly.
Doing something simpler with build testing for autoconf and other GNU projects would be entirely possible, I mention this mainly to show what can be done!
I do worry about the future for autotools, I see projects moving over to alternatives and I am happy to see the performance gains we get from it.
I'd mention that the copyright assignment situation needed for contributions is prohibitive to people being able to participate (including me in the past). GNU's thoughts on version control and project interconnection is also somewhat discouraging to people in the wider open source ecosystem although I don't follow it closely.
I've also not seen mention of it but the fact that GCC and the toolchain use an ancient version of autoconf has always been rather sad to me.
I do think autotools has a proud heritage but to survive into the next decade, I do think it needs to change direction, stop supporting old obsolete systems and embrace more new technology, much like it would have done originally. Reducing to the lowest common denominator is ok for a while but ultimately signs a death warrant as I've seen observed on the libtool list by devlopers in recent years too.
Everything here is meant as constructive support of trying to find a way to allow autotools to succeed into the future.
Cheers,
Richard
_______________________________________________ Hangout mailing list Hangout-at-nylxs.com http://lists.mrbrklyn.com/mailman/listinfo/hangout
|
|