bitbake.git
9 years agobitbake/fetch2: Rename Fetch class to FetchMethod
Richard Purdie [Fri, 4 Feb 2011 10:49:27 +0000 (10:49 +0000)]
bitbake/fetch2: Rename Fetch class to FetchMethod

(From Poky rev: ab0dd1397491478ee6149283e5ba8775dd8cdc3b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: Drop old md5 handling code
Richard Purdie [Fri, 4 Feb 2011 10:30:54 +0000 (10:30 +0000)]
bitbake/fetch2: Drop old md5 handling code

Drop some old md5 functions since we have improved functionality now which includes
sha256 checksum support. This stops each download being md5 checksumed twice.

Also change ".md5" stamp extentions to ".done" to better describe its use as a
download complete marker file and no longer write the md5 sum to the files.

(From Poky rev: 74b71864fed79ce60e721945c8e239b3ebf49200)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: Rewrite and improve exception handling, reusing core functions for...
Richard Purdie [Fri, 4 Feb 2011 10:26:21 +0000 (10:26 +0000)]
bitbake/fetch2: Rewrite and improve exception handling, reusing core functions for common operations where possible

(From Poky rev: d08397ba4d1331993300eacbb2f78fcfef19c1cf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: Have all fetcher exceptions derived from a common BBFetchException...
Richard Purdie [Thu, 3 Feb 2011 23:23:11 +0000 (23:23 +0000)]
bitbake/fetch2: Have all fetcher exceptions derived from a common BBFetchException class

(From Poky rev: f6eefb3ca3bb2a5ea0ec1364bdb0bc41ae58c815)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: Drop unused parameter for localpath() and update comments
Richard Purdie [Thu, 3 Feb 2011 22:02:09 +0000 (22:02 +0000)]
bitbake/fetch2: Drop unused parameter for localpath() and update comments

(From Poky rev: 8daab5b95157dda6854fe6bf1929f911fe3cf25e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: Simplfy setup_srcrevs
Richard Purdie [Thu, 3 Feb 2011 21:43:01 +0000 (21:43 +0000)]
bitbake/fetch2: Simplfy setup_srcrevs

(From Poky rev: 7e4fbfc1c1887a1a0507b60244aa53b8b1994edd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: Simplify localpath variable handling FetchData init
Richard Purdie [Thu, 3 Feb 2011 21:40:03 +0000 (21:40 +0000)]
bitbake/fetch2: Simplify localpath variable handling FetchData init

(From Poky rev: 49a022d25d35115e7286e2ca2530566da2d71aa8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: Define a sane localpath function and remove code duplication
Richard Purdie [Thu, 3 Feb 2011 21:20:39 +0000 (21:20 +0000)]
bitbake/fetch2: Define a sane localpath function and remove code duplication

(From Poky rev: 5a52cb80d6d62c704692ad6cfa93426c4a17bc52)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: Move ud.localfile setup into urldata_init
Richard Purdie [Thu, 3 Feb 2011 21:17:26 +0000 (21:17 +0000)]
bitbake/fetch2: Move ud.localfile setup into urldata_init

(From Poky rev: 972eb5fababb33b5537fcfbbaf8e33ea820f0fee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: Drop horrible SRCREV hack which is no longer needed
Richard Purdie [Thu, 3 Feb 2011 20:33:07 +0000 (20:33 +0000)]
bitbake/fetch2: Drop horrible SRCREV hack which is no longer needed

(From Poky rev: 0e5404cedadaea6738ef6468d7eb8a24d23d9bab)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2/git: Switch to completely use bare clones
Richard Purdie [Sun, 6 Feb 2011 10:20:18 +0000 (10:20 +0000)]
bitbake/fetch2/git: Switch to completely use bare clones

We no longer need index/workdir support in the mirror tree and it causes all
kind of reference naming problems.Simplifying the code to remove this and use
just bare clones addresses this problem.

We increase the "version" number on the mirror tarballs to reflect the change
and ensure older mirror tarballs are not used as they would break.

(From Poky rev: e4c1e99f234491af1c0fc886e73deedaef108a1a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agofetch2/git.py: Fix problem when fetching with no origin
Saul Wold [Sat, 5 Feb 2011 22:11:25 +0000 (14:11 -0800)]
fetch2/git.py: Fix problem when fetching with no origin

When updating a git repo, its possible to not have an origin,
so fail gracefully with a debug message.

(From Poky rev: 192f631129e25b959d501c46c3156b7037b49c22)

Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2/git: Ensure we prune origin to remove stale branches
Richard Purdie [Sat, 5 Feb 2011 13:16:45 +0000 (14:16 +0100)]
bitbake/fetch2/git: Ensure we prune origin to remove stale branches

Thanks to Bruce Ashfield for the tip.

(From Poky rev: 96810cade04cad44d70a7c6e51aacdf89caef92b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2/git: Ensure deleted branches upstream are removed, all branch tags...
Richard Purdie [Thu, 3 Feb 2011 20:16:05 +0000 (20:16 +0000)]
bitbake/fetch2/git: Ensure deleted branches upstream are removed, all branch tags are forced to the upstream revisions and all revisions and tags are always fetched

(From Poky rev: dfc5faf717ac1f917af398097d5295128548a55d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: Remove old local file acceleration code
Richard Purdie [Thu, 3 Feb 2011 18:24:25 +0000 (18:24 +0000)]
bitbake/fetch2: Remove old local file acceleration code

Since local mirror fetches are always symlinked from the download directory
directly, there is no need for this premirrors hack which doesn't cover
mirrors and also abuses the localpath variable with inconsistent results.

(From Poky rev: 282a828f3dc373d8f1397827ebbe1be1c54f2d2a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: Ensure that mirror fetches are symlinked from the download directory
Richard Purdie [Thu, 3 Feb 2011 18:22:06 +0000 (18:22 +0000)]
bitbake/fetch2: Ensure that mirror fetches are symlinked from the download directory

When files are fetched from a mirror source that happens to be local,
ensure links are created for the file since subsequent fetch calls
can then follow the links to find files.

Any other approach such as the existing manipulations of localpath
internally to the fetcher are prone to errors, races and other issues.

(From Poky rev: fa9fda05e1d269446b51050195b891346482e8bb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: When encoding a file:// url, drop user and host information
Richard Purdie [Thu, 3 Feb 2011 18:19:23 +0000 (18:19 +0000)]
bitbake/fetch2: When encoding a file:// url, drop user and host information

When processing a cvs SRC_URI to a file:// mirror, the user and host information
will break the mirror processing. This patch addresses it by only constructing
valid urls.

(From Poky rev: 7f99605562119a13a2510a3c990e3cf577ad764e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/lib/bb/fetch2: Assign a default name in the names array as empty strings...
Richard Purdie [Thu, 3 Feb 2011 00:16:23 +0000 (00:16 +0000)]
bitbake/lib/bb/fetch2: Assign a default name in the names array as empty strings as names cause problems for python

(From Poky rev: a55d86b4051f6d063f9a67db026f7da6f9b50cc6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: Add missing parameter to localcount_internal_helper
Richard Purdie [Wed, 2 Feb 2011 18:48:13 +0000 (18:48 +0000)]
bitbake/fetch2: Add missing parameter to localcount_internal_helper

(From Poky rev: 08cd6c1cb9639e958e0c60a02e317a22cf43f880)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2/git: Ensure target directory exists when copying files
Richard Purdie [Wed, 2 Feb 2011 17:46:37 +0000 (17:46 +0000)]
bitbake/fetch2/git: Ensure target directory exists when copying files

(From Poky rev: adfa6c40dadebb18bfd457859300d8c093b007f7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agofetch2: Allow multiple src rev to be used in one src_uri
Yu Ke [Wed, 26 Jan 2011 12:14:06 +0000 (20:14 +0800)]
fetch2: Allow multiple src rev to be used in one src_uri

* SRC_URI format:
the SRC_URI are extended to allow multiple src rev:
name=<name1>,<name2>,...<name-n>
branch=<branch1>,<branch2>,...,<branch-n>
also SRCREV can be defined with
SRCREV_<name1> = xxxxx
SRCREV_<name2> = xxxxx

* FetchData extention
to support multiple src rev, several FetchData data are added:
- FetchData.names: list of name in SRC_URI,  one name per srcrev. name is the index of revision and branch
- FetchData.revisions: dictionary of name->revision.
- FetchData.branches: dictionary of name->branch.

For example, linux-yocto recipes becomes:
SRC_URI = "git://git.pokylinux.org/linux-yocto-2.6.37;protocol=git;branch=${KBRANCH},meta;name=machine,meta"

FetchData.names = ['machine', 'meta']
FetchData.revisions = { 'machine':xxxxx, 'meta':xxxxxx }
FetchData.branches = { 'machine':${KBRANCH}, 'meta':'meta'}

* generic revision handling extension
the related revision handling code in fetch2.__init__.py are changed accordingly. the major change is add name parameter to indicate which src rev to handling. originally there is one src rev per FetchData, so FetchData parameter is enough. now since one FetchData has multiple src rev, it is necessary to use FetchData + name to specifiy src rev.

* git extension
git fetcher are also revised to take advantage of the multiple src rev in FetchData. especially the download() method are enhanced to fetch multiple src rev.

* other fetcher (svn, hg, ...) does not support multiple src rev. they just sync the API to add name, and then simply ignore the name. no actually functional change

(From Poky rev: 0e837e6844be449659bb96a1498e54a8b9442d13)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agogit.py: Remove the source tree tarball archives
Yu Ke [Tue, 18 Jan 2011 16:58:42 +0000 (00:58 +0800)]
git.py: Remove the source tree tarball archives

Since we're now always providing the git source control files it becomes
pointless to handle the tarballs of specific git revisions so drop this
part of the fetcher.

(From Poky rev: affcfe468a0c2993c0c4ace446248f85aee4315f)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2/git: Add nocheckout option to disable checkouts and always add scm...
Richard Purdie [Wed, 2 Feb 2011 12:48:32 +0000 (12:48 +0000)]
bitbake/fetch2/git: Add nocheckout option to disable checkouts and always add scm data to checkouts

(From Poky rev: 05048e0f9168ceb7d66f6ac79dcae42f1e696826)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobb.fetch2: Add git unpack
Yu Ke [Tue, 18 Jan 2011 16:22:13 +0000 (00:22 +0800)]
bb.fetch2: Add git unpack

The git download method clones the git repository to the local machine. The unpack process
can be optimised to be a local to local machine clone or a direct readtree operation to the
destination using git.will clone git repo to local, so git unpack can be simplified
to only checkouting the code to the work dir. For fullclone case, we also
need to manually copy all the ref info, which is needed by the later do_kernel_checkout().
Rather than use hardlinks, we reference the repository using alternatives since the
download directory may be on a different filesystem.

[Change to use -s by Richard Purdie]

(From Poky rev: 3a3fc6b071f09970e7f79fb4f1a8f0013917371e)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: Ensure the local revision counter takes a default value of 0, not...
Richard Purdie [Tue, 1 Feb 2011 23:33:19 +0000 (23:33 +0000)]
bitbake/fetch2: Ensure the local revision counter takes a default value of 0, not None

(From Poky rev: 05954ef4d7d882f34e2bb1f3bc44ad6f80c11d4c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/fetch2: Instrument fetchers when making network access
Yu Ke [Mon, 24 Jan 2011 07:56:54 +0000 (15:56 +0800)]
bitbake/fetch2: Instrument fetchers when making network access

(From Poky rev: 029f8584d547c0792ffbe2f83451dcdfe2a3db7f)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobb.fetch2: add "BB_NO_NETWORK" option
Yu Ke [Mon, 24 Jan 2011 06:58:26 +0000 (14:58 +0800)]
bb.fetch2: add "BB_NO_NETWORK" option

Sometime user want a purely local fetching, i.e. using local mirror without
any remote netowrk access. BB_NO_NETWORK option is introduced for this purpose

check_network_access() is the guard for BB_NO_NETWOKR option. it should be
put in any place that fetcher use network access

(From Poky rev: 098e8ded339f3bf864f3bad9871028176f70b12b)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobb.fetch2: remove the obsolate Fetch.try_mirrors referrence
Yu Ke [Tue, 18 Jan 2011 16:17:39 +0000 (00:17 +0800)]
bb.fetch2: remove the obsolate Fetch.try_mirrors referrence

Fetch.try_mirrors is no longer exists, so the code is obsolate

(From Poky rev: 733de7596c2ed78a846541b3290f01c21ff96606)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agogit.py: split download to download() and build_mirror_data()
Yu Ke [Tue, 18 Jan 2011 15:35:30 +0000 (23:35 +0800)]
git.py: split download to download() and build_mirror_data()

the download is to fetch the source from URL, the build_mirror_data is
to create the mirror tar ball. the original go() method mix them together,
it is more clean to split them.

(From Poky rev: ef918a72de97fd2189e1c8487e371d13e18088fd)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobb.fetch2: rename "go" with "download" to better reflect its functionality
Yu Ke [Tue, 18 Jan 2011 15:03:53 +0000 (23:03 +0800)]
bb.fetch2: rename "go" with "download" to better reflect its functionality

no functional change

(From Poky rev: e05918937c515dff845fcb4c9e94f8ecbea8c957)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobb.fetch: add fetch version to distinguish bb.fetch and bb.fetch2
Yu Ke [Tue, 18 Jan 2011 14:41:23 +0000 (22:41 +0800)]
bb.fetch: add fetch version to distinguish bb.fetch and bb.fetch2

there is case that we need to distingush bb.fetch and bb.fetch2,
and use different API for bb.fetch and bb.fetch2. so it is necessary
to add version info for distinguish purpose

(From Poky rev: f12e71484593039cd58b6e7fadd038b28b05d849)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobb.fetch2: revise the Fetch.unpack API
Yu Ke [Tue, 18 Jan 2011 14:08:09 +0000 (22:08 +0800)]
bb.fetch2: revise the Fetch.unpack API

change the unpack to use the urldata and rootdir parameter
- urldata is the FetchData instance
- rootdir is the dir to put the extracted source. the original unpack
  use current dir (os.getcwd) as destination dir, which is not flexible
  and error-prone (error will occur if caller not chdir to dest dir)

(From Poky rev: 84ff79413a3b56f287f42b8ca1dd2ab194337c42)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobb.fetch2: add unpack method in fetcher
Yu Ke [Tue, 18 Jan 2011 13:53:36 +0000 (21:53 +0800)]
bb.fetch2: add unpack method in fetcher

copy exactly the base.bbclass:oe_unpack_file() to bb.fetch2 as the code base

(From Poky rev: d8698b92ffcc2cbd29b57d3dc8e851d3d6c137f2)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agoFetcher: only set __BB_DONT_CACHE when SRCREV = "${AUTOREV}"
Yu Ke [Mon, 27 Dec 2010 08:20:34 +0000 (16:20 +0800)]
Fetcher: only set __BB_DONT_CACHE when SRCREV = "${AUTOREV}"

we should cache SRCREV whenever possible, the only exception is
when SREREV is auto rev. so change the logic to only set __BB_DONT_CACHE
at SRCREV = "${AUTOREV}" case

(From Poky rev: 2b2dbf76126650750cb8a6778b837af6fe4c8b21)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agoFetcher: break the "SRCREVINACTION" deadlock
Yu Ke [Mon, 27 Dec 2010 01:31:38 +0000 (09:31 +0800)]
Fetcher: break the "SRCREVINACTION" deadlock

Current fetcher has annoying "SRCREVINACTION" deadlock,
which occurs when SRCREV=${AUTOREV}=@bb.fetch.get_srcrev():
get_srcrev()->setup_localpath()->srcrev_internal_helper()
->evaluate SRCREV->get_srcrev()

current fetcher resolve the deadlock by introducing a
"SRCREVINACTION" condition check. Althoguh it works, it is
indeed not clean.

This patch use antoehr idea to break the deadlock: break
the dependency among SRCREV and get_srcrev(), i.e. assign
a specific keyword "AUTOINC" to AUTOREV. when Fetcher meet
this keyword, it will check and set the latest revision to
urldata.revision. get_srcrev later can use the urldata.revision
for value evaluation(SRCPV etc). In this case, SRCREV no longer
depends on get_srcrev, and there is not deadlock anymore.

(From Poky rev: 1589a1172f9432aed1cc9ce006f68cddf3073774)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobb.fetch2.bzr: add bzr urldata_init
Yu Ke [Mon, 27 Dec 2010 02:48:34 +0000 (10:48 +0800)]
bb.fetch2.bzr: add bzr urldata_init

move the bzr specific urldata init from localpath to urldata_init
so that it can be called early

(From Poky rev: 550c3bd82115b4bdb8235da53cfc18b1dc39ad96)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobb.fetch2.hg: add hg urldata_init
Yu Ke [Mon, 27 Dec 2010 02:35:19 +0000 (10:35 +0800)]
bb.fetch2.hg: add hg urldata_init

move the hg specific urldata init from localpath to urldata_init
so that it can be called early

(From Poky rev: f684ff18a2b9565823a41750af369dee392f6142)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobb.fetch2.svn.py: add urldata_init
Yu Ke [Mon, 27 Dec 2010 02:07:04 +0000 (10:07 +0800)]
bb.fetch2.svn.py: add urldata_init

move the svn specific urldata init from localpath to urldata_init
so that it can be called early

(From Poky rev: 7420e70f4f03a18f9d5a0d3226918088af1032bd)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobb.fetch2.git.py: add git urldata_init
Yu Ke [Mon, 27 Dec 2010 01:50:25 +0000 (09:50 +0800)]
bb.fetch2.git.py: add git urldata_init

move the git specific urldata init from localpath to urldata_init
so that it can be called early

(From Poky rev: 54e34f6e255d1717beada23638a5783c9dda42ea)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobb.fetch2: add urldata_init call for Fetch class
Yu Ke [Sat, 8 Jan 2011 08:40:18 +0000 (16:40 +0800)]
bb.fetch2: add urldata_init call for Fetch class

FetchData has some fetch method specific data, and only fetch method knows how
to initialize it. originally it is mostly initialized in Fetch.localpath().
But now there is requirement to call Fetch.latest_revision() before
Fetch.localpath(), thus require another earlier place for initialization. so
urldata_init is introduced for this purpose. it will be called in FetchData:__init__
and make all the Fetch functions useable after that.

(From Poky rev: 2892cbfb114de1d2e2b5b479d5f928646ce5c831)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agoBBHandler: remove bb.fetch referrence
Yu Ke [Sat, 8 Jan 2011 02:26:28 +0000 (10:26 +0800)]
BBHandler: remove bb.fetch referrence

BBHandler.py no longer use bb.fetch, so remove its import statement

(From Poky rev: 9e3a92c04a5be4932a0324af4a74bce871126e9d)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/utils.py: add glob name matching to remove
Saul Wold [Tue, 8 Feb 2011 17:24:12 +0000 (09:24 -0800)]
bitbake/utils.py: add glob name matching to remove

(From Poky rev: 07088f7711e0b53c4fac3685d15d406cf4793602)

Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/msg: Ensure lower level debug messages have DEBUG prefix and reuse log level...
Richard Purdie [Mon, 7 Feb 2011 14:49:10 +0000 (14:49 +0000)]
bitbake/msg: Ensure lower level debug messages have DEBUG prefix and reuse log level values from formatter

(From Poky rev: 7586adb360d8075d3e97184dfcafb1b13ce5f838)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years ago__init__.py: Fix debug log level handling to correct debug output
Richard Purdie [Mon, 7 Feb 2011 14:46:22 +0000 (14:46 +0000)]
__init__.py: Fix debug log level handling to correct debug output

(From Poky rev: d7eebbe9dbf0d790d4af93466f5c27127cae0999)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agoExport KRB5CCNAME variable
Javier Martin [Mon, 31 Jan 2011 11:27:54 +0000 (12:27 +0100)]
Export KRB5CCNAME variable

This allows fetching git repositories using Kerberos authentication.

Signed-off-by: Javier Martin <javier.martin@vista-silicon.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years ago.gitignore: add *.pyo
Andreas Oberritter [Tue, 4 Jan 2011 15:32:07 +0000 (15:32 +0000)]
.gitignore: add *.pyo

* python generates pyo files if PYTHONOPTIMIZE is set

Signed-off-by: Andreas Oberritter <obi@opendreambox.org>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agoFix comparison with SRCREVINACTION constant
Javier Martin [Thu, 27 Jan 2011 08:38:21 +0000 (09:38 +0100)]
Fix comparison with SRCREVINACTION constant

Use '==' instead of 'is', otherwise it will always return
true since 'rev' and "SRCREVINACTION" are not the same object.

Signed-off-by: Javier Martin <javier.martin@vista-silicon.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agoevent: fix exception format
Chris Larson [Tue, 8 Feb 2011 22:35:03 +0000 (15:35 -0700)]
event: fix exception format

Forgot to concatenate the lines returned by format_exception.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agoevent: limit the number of traceback entries to 5
Chris Larson [Tue, 8 Feb 2011 20:51:46 +0000 (13:51 -0700)]
event: limit the number of traceback entries to 5

This is slightly more sane.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agopersist_data: kill unreachable break line
Chris Larson [Tue, 8 Feb 2011 20:42:46 +0000 (13:42 -0700)]
persist_data: kill unreachable break line

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agoShift traceback pre-formatting into LogHandler
Chris Larson [Tue, 8 Feb 2011 20:41:36 +0000 (13:41 -0700)]
Shift traceback pre-formatting into LogHandler

It's cleaner to leave it behaving as usual, passing the exception data in the
exc_info attribute of the LogRecord where it normally lies, and then let
LogHandler make it pickleable so it can be sent to the UI.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agopersist_data: loop on database lock for table creation
Chris Larson [Tue, 8 Feb 2011 17:41:58 +0000 (10:41 -0700)]
persist_data: loop on database lock for table creation

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agobuild: improve set -x handling
Chris Larson [Tue, 8 Feb 2011 16:53:47 +0000 (09:53 -0700)]
build: improve set -x handling

- set -x always, not just when debugging
- set -x immediately before executing the function, rather than at the top of
  the script, so the output is no longer cluttered with all the variable and
  function definitions

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agoMerge branch '1.10'
Chris Larson [Mon, 7 Feb 2011 20:38:10 +0000 (13:38 -0700)]
Merge branch '1.10'

9 years agobitbake-layers: fix bug with env vars leaking in
Chris Larson [Mon, 7 Feb 2011 17:58:40 +0000 (10:58 -0700)]
bitbake-layers: fix bug with env vars leaking in

I forgot that bin/bitbake is what does the environment filtering based upon
BB_ENV_WHITELIST, etc.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agoAdd initial bitbake-layers script
Chris Larson [Mon, 7 Feb 2011 17:50:27 +0000 (10:50 -0700)]
Add initial bitbake-layers script

This script has subcommands which operate against your bitbake layers, either
displaying useful information, or acting against them.  Currently, it only
provides a show_appends command, which shows you what bbappends are in effect,
and warns you if you have appends which are not being utilized.

Currently, a bug exists when using this due to the DataContext stuff, but I'm
not certain as to the root cause, it appears to be the bb package relying
implicitly on the way the bitbake script does things.  A fix for that issue
will be forthcoming, as will further subcommands.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agobitbake/runqueue.py: Avoid starvation of events to the server
Richard Purdie [Fri, 28 Jan 2011 10:21:41 +0000 (10:21 +0000)]
bitbake/runqueue.py: Avoid starvation of events to the server

The server UI was reading 1024 bytes, then sleeping for 0.25 seconds. Since
most new LogRecord events are larger than this it leads to a build up of data
which is only processed slowly, leading to a bottleneck and a slow down of
all bitbake processes.

Thanks to Dongxiao Xu <dongxiao.xu@intel.com> for the great work in debugging
this. A large value has been left in for the read() command just to ensure some
fairness amongst process handling if a task tries to log truly huge amounts of
data to the server, or goes crazy and ensures the main loop doesn't stall.

(From Poky rev: 06c6db7929c75f576a395fb442abe447b833fc3b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agoAdd a few items to .gitignore
Chris Larson [Fri, 21 Jan 2011 01:51:52 +0000 (18:51 -0700)]
Add a few items to .gitignore

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agocache: fix docstring usage
Chris Larson [Thu, 20 Jan 2011 21:32:16 +0000 (14:32 -0700)]
cache: fix docstring usage

Use comments instead of docstrings where appropriate in CacheData.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agocooker: also respond to SystemExit for execute_runqueue
Chris Larson [Mon, 17 Jan 2011 15:50:40 +0000 (08:50 -0700)]
cooker: also respond to SystemExit for execute_runqueue

Without this, bitbake appears to hang when encountering an invalid task.  In
reality, the SystemExit flows up uncaught, the server may go away, and the UI
doesn't know about it, so requires us to ^C to exit.

This is a temporary measure.  Poky's bitbake handles invalid tasks
differently, and avoids this *particular* issue, but we do need to ensure that
the server responds to SystemExit from the idle function in a sane way.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agobitbake/knotty.py: Ensure task note messages are only surpressed at default log levels
Richard Purdie [Wed, 12 Jan 2011 13:45:39 +0000 (13:45 +0000)]
bitbake/knotty.py: Ensure task note messages are only surpressed at default log levels

(From Poky rev: 3f08b7f1eadc8d181c76339d14c7abb4f49cad57)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agocodeparser: fix spacing in diagnostic messages
Bernhard Reutner-Fischer [Tue, 11 Jan 2011 18:21:55 +0000 (19:21 +0100)]
codeparser: fix spacing in diagnostic messages

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
9 years ago*: use utils.remove() some more
Bernhard Reutner-Fischer [Mon, 10 Jan 2011 22:05:19 +0000 (23:05 +0100)]
*: use utils.remove() some more

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
9 years agobuild: remove duplicate import of utils
Bernhard Reutner-Fischer [Mon, 10 Jan 2011 15:32:56 +0000 (16:32 +0100)]
build: remove duplicate import of utils

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
9 years agorunqueue.py: Call bb.cache.Cache.loadDataFull function directly when dumping signatur...
Richard Purdie [Mon, 10 Jan 2011 22:34:01 +0000 (22:34 +0000)]
runqueue.py: Call bb.cache.Cache.loadDataFull function directly when dumping signatures as per the rest of runqueue

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/runqueue.py: Fix del_stamp calls and -f option to bitbake with the BasicHash...
Richard Purdie [Mon, 10 Jan 2011 21:59:34 +0000 (21:59 +0000)]
bitbake/runqueue.py: Fix del_stamp calls and -f option to bitbake with the BasicHash siggen code

(From Poky rev: 48e567bb477ad468d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agomsg: no need to hardcode the logging levels
Chris Larson [Mon, 10 Jan 2011 19:30:05 +0000 (12:30 -0700)]
msg: no need to hardcode the logging levels

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agovim: properly highlight python inside of ${@}
Chris Larson [Mon, 10 Jan 2011 19:00:41 +0000 (12:00 -0700)]
vim: properly highlight python inside of ${@}

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agovim: handle highlighting exports without assignment
Chris Larson [Mon, 10 Jan 2011 19:00:09 +0000 (12:00 -0700)]
vim: handle highlighting exports without assignment

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agovim: add an ftplugin for indentation settings
Chris Larson [Mon, 10 Jan 2011 18:57:59 +0000 (11:57 -0700)]
vim: add an ftplugin for indentation settings

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agobitbake: handle IOError from event_queue.get
Chris Larson [Mon, 10 Jan 2011 18:54:52 +0000 (11:54 -0700)]
bitbake: handle IOError from event_queue.get

These can occur in a particular ^C shutdown, so just break out of the event
queue empty loop if we encounter it.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agoInject taskpid into log records via our log handler
Chris Larson [Mon, 10 Jan 2011 16:20:50 +0000 (09:20 -0700)]
Inject taskpid into log records via our log handler

It turns out that while log filters added with addFilter are only associated
with that logger, and not its children, handlers are inherited, and handlers
can be filters.   So, let's add filtering to our existing LogHandler class
which dispatches our log records as bitbake events.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agoSwitch to fetch2 fetcher code based on the environment variable BBFETCH2
Richard Purdie [Mon, 10 Jan 2011 18:12:53 +0000 (18:12 +0000)]
Switch to fetch2 fetcher code based on the environment variable BBFETCH2

(From Poky rev: af06e4a8efbcaff657c397c34f2cb893d2d7f7f7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobb.fetch2: replace bb.fetch with bb.fetch2 in the bb.fetch
Yu Ke [Mon, 10 Jan 2011 18:05:07 +0000 (18:05 +0000)]
bb.fetch2: replace bb.fetch with bb.fetch2 in the bb.fetch

bb.fetch2 is copied from bb.fetch, and has many bb.fetch referrence.
Fix these referrence with bb.fetch2 referrence

(From Poky rev: c64b445f38179a117d39319f762f9dc2435da0ea)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agofetch2/git: Remove duplicated code only added for backwards compatibility
Richard Purdie [Mon, 10 Jan 2011 14:26:56 +0000 (14:26 +0000)]
fetch2/git: Remove duplicated code only added for backwards compatibility

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake: copy bb.fetch to bb.fetch2 as initial code base for fetcher overhaul
Yu Ke [Mon, 10 Jan 2011 14:25:04 +0000 (14:25 +0000)]
bitbake: copy bb.fetch to bb.fetch2 as initial code base for fetcher overhaul

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agoutils.py: Update log message to use the logger calls
Richard Purdie [Mon, 10 Jan 2011 13:19:29 +0000 (13:19 +0000)]
utils.py: Update log message to use the logger calls

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agoevent.py: Improve debug about invalid events
Richard Purdie [Mon, 10 Jan 2011 13:08:09 +0000 (13:08 +0000)]
event.py: Improve debug about invalid events

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agopersist_data: Don't loop forever waiting on database locks
Richard Purdie [Mon, 10 Jan 2011 13:00:39 +0000 (13:00 +0000)]
persist_data: Don't loop forever waiting on database locks

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/runqueue.py: Fix a bug where do_setscene dependencies would be ignored
Richard Purdie [Thu, 6 Jan 2011 19:48:47 +0000 (19:48 +0000)]
bitbake/runqueue.py: Fix a bug where do_setscene dependencies would be ignored

(From Poky rev: df8569b4d89ce83e3cafd87f2f37b795d1bfbd6d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agorunqueue.py: Fix undefined schedulers variable reference
Richard Purdie [Mon, 10 Jan 2011 12:53:11 +0000 (12:53 +0000)]
runqueue.py: Fix undefined schedulers variable reference

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agorunqueue.py: Add debug to help resolve partial log message failure
Richard Purdie [Mon, 10 Jan 2011 12:51:56 +0000 (12:51 +0000)]
runqueue.py: Add debug to help resolve partial log message failure

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agorunqueue.py: Add missing result return value in setscene failure path
Richard Purdie [Mon, 10 Jan 2011 12:51:00 +0000 (12:51 +0000)]
runqueue.py: Add missing result return value in setscene failure path

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agorunqueue.py: Fix up some lost range->xrange conversions misplaced as part of the...
Richard Purdie [Mon, 10 Jan 2011 12:50:10 +0000 (12:50 +0000)]
runqueue.py: Fix up some lost range->xrange conversions misplaced as part of the poky sync up

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agofetch/__init__.py: Add missing debug level argument
Richard Purdie [Mon, 10 Jan 2011 12:17:14 +0000 (12:17 +0000)]
fetch/__init__.py: Add missing debug level argument

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/siggen.py: Add a signature generator which adds the hash to the stamp files
Richard Purdie [Fri, 7 Jan 2011 11:04:38 +0000 (11:04 +0000)]
bitbake/siggen.py: Add a signature generator which adds the hash to the stamp files

(From Poky rev: 97789a160ad21ccc614f976fa5e2b306d816aabf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake build/siggen/runqueue: Fix stampfile parameters
Richard Purdie [Thu, 6 Jan 2011 19:47:00 +0000 (19:47 +0000)]
bitbake build/siggen/runqueue: Fix stampfile parameters

The current parameters are not useful to the stampfile generator function
as they can't uniquely define a task. This updated things so the
parameters can identify unique tasks.

(From Poky rev: 66c6200ff34e8eeca5d1a689bbf9d6a83818248f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobb.utils: check if lock file is writable, to fix Yocto bug 606
Yu Ke [Wed, 29 Dec 2010 07:28:48 +0000 (15:28 +0800)]
bb.utils: check if lock file is writable, to fix Yocto bug 606

Bug 606 report that if $DL_DIR is read-only, do_fetch will
simply hang without any error message.

The root cause is that: bb.fetch.go()->bb.utils.lockfile()
will try to lock file ${DL_DIR}/xxxxx.lock. Since ${DL_DIR}
is read-only, it will cause IOError exception. Although
lockfile() can catch the exception, currently code simply
ignore all the exception and continue the loop. it make
sense if the exception is caused by locking contention,
but in the read-only $DL_DIR case, it cause endless waiting
unfortunately.

So this patch add read-only check for lockfile to avoid the
silent hang.

(From Poky rev: 6ee0c26e21f48dcd47af88ed5c174e76e76a3e42)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/knotty.py: Don't show log messages of NOTE and below for task processes
Richard Purdie [Fri, 7 Jan 2011 23:54:42 +0000 (23:54 +0000)]
bitbake/knotty.py: Don't show log messages of NOTE and below for task processes

This cleans up the knotty console messages to be a lot quieter and cleaning,
in keeping with the expectations of most users.

(From Poky rev: b22e345e05efcc3f66278af8f09fb083afe32b68)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/__init__.py: Add taskpid to all LogRecords (subclassed to be BBLogRecords)
Richard Purdie [Fri, 7 Jan 2011 23:52:19 +0000 (23:52 +0000)]
bitbake/__init__.py: Add taskpid to all LogRecords (subclassed to be BBLogRecords)

This allows us to identify which task messages are from.

(From Poky rev: f8e7215f6c4ece8c0a74ceee8da707cf791038e8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/msg.py: Add named definitions for all logging levels
Richard Purdie [Fri, 7 Jan 2011 23:50:17 +0000 (23:50 +0000)]
bitbake/msg.py: Add named definitions for all logging levels

(From Poky rev: a13352cfaf4b5d19196151e362851b1516e6af15)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agoMerge branch 'poky-sync'
Chris Larson [Fri, 7 Jan 2011 15:51:43 +0000 (08:51 -0700)]
Merge branch 'poky-sync'

* poky-sync:
  build: kill unneeded environment setup bits
  parse: save python functions into the metadata
  Include exported variables in task process env
  runqueue: start implementing quieterrors
  Overhaul environment handling
  runqueue: resurrect use of file objects for pipein/pipeout
  runqueue: clean up message log levels
  runqueue: use correct task ID when checking validity of setscene tasks
  Add support for 'noexec' tasks
  runqueue: implement cache checking from metadata
  runqueue: pass task hash information to tasks
  runqueue: avoid unnecessary delays
  Implement task signatures
  data: Add emit_func and generate_dependencies
  codeparser: Implement persistent cache
  data_smart: track variable references
  codeparser: add module for parsing shell/python
  runqueue: implement scenequeue
  fetch: add SRC_URI checksum

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agocache: don't expand variables for skipped recipes
Chris Larson [Fri, 7 Jan 2011 15:38:41 +0000 (08:38 -0700)]
cache: don't expand variables for skipped recipes

Errors can result from these expansions, but for skipped recipes, we
shouldn't care about those failures.  This fixes the same issue which
Richard Purdie fixed in poky, commit 847b717.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agobitbake/build.py: Fix del_stamp work correctly after recent stamp function changes
Richard Purdie [Fri, 7 Jan 2011 14:03:27 +0000 (14:03 +0000)]
bitbake/build.py: Fix del_stamp work correctly after recent stamp function changes

(From Poky rev: 5da9747)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/data.py: corrected the output for shell syntax.
Lianhao Lu [Fri, 7 Jan 2011 06:17:10 +0000 (14:17 +0800)]
bitbake/data.py: corrected the output for shell syntax.

[BUGID# 645], modify the emit_var()
1. Added "#" to the beginning of each line if the comment contains
multiple lines.

2. Added "\" to the end of each line if the shell variable value
contains multiple lines.

(From Poky rev: 6f454c10bcdd5)

Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake build.py: Stamp handling improvements
Richard Purdie [Thu, 6 Jan 2011 19:43:41 +0000 (19:43 +0000)]
bitbake build.py: Stamp handling improvements

* Move stamp file deletion out of the internal stamp helper function
* Add a new function to return the path to a stamp for a given task

(From Poky rev: ec6e4793dc79c61b780b74571db4cd66c1c04251)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake build.py: Use localdata for stamp handling, not d
Richard Purdie [Thu, 6 Jan 2011 19:40:41 +0000 (19:40 +0000)]
bitbake build.py: Use localdata for stamp handling, not d

(From Poky rev: 4a10a5b438f49577fb29f4e277e2300cdce10ab8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agobitbake/build.py: Ditch the extract_stamp function as its just going to complicate...
Richard Purdie [Thu, 6 Jan 2011 15:41:39 +0000 (15:41 +0000)]
bitbake/build.py: Ditch the extract_stamp function as its just going to complicate improving the stamp handling functions

(From Poky rev: 51a2e8712a9b94de53253e198c64c0fa428e4349)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
9 years agoparse: pass filename, lineno into the ast
Chris Larson [Tue, 4 Jan 2011 20:34:08 +0000 (13:34 -0700)]
parse: pass filename, lineno into the ast

We will be needing this information to improve the tracebacks of python code
from the metadata, as well as to give the user information about where
variables were defined, so they know how it ended up the way it is.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
9 years agoparse.ast: drop __word__ regular expression
Chris Larson [Tue, 4 Jan 2011 20:07:27 +0000 (13:07 -0700)]
parse.ast: drop __word__ regular expression

We can use the string split method for this instead.

Signed-off-by: Chris Larson <chris_larson@mentor.com>