bitbake.git
8 years agobitbake/cooker: Ensure delays are accurately transfered to the idle loops from runqueue
Richard Purdie [Tue, 15 Feb 2011 23:18:39 +0000 (23:18 +0000)]
bitbake/cooker: Ensure delays are accurately transfered to the idle loops from runqueue

(From Poky rev: 579ecccc2244da43b9bc03fb09bb71d8a0219a8b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/runqueue: When we have successfully collected a pid value return True to...
Richard Purdie [Tue, 15 Feb 2011 23:20:20 +0000 (23:20 +0000)]
bitbake/runqueue: When we have successfully collected a pid value return True to ensure we don't sleep in the main loop

(From Poky rev: 5a178b547bbf6d99c42cba6821b3f559e585a079)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/runqueue: When we successfully fork off a task and there are more processes...
Richard Purdie [Tue, 15 Feb 2011 23:20:54 +0000 (23:20 +0000)]
bitbake/runqueue: When we successfully fork off a task and there are more processes we can start do so immediately

(From Poky rev: 974ea1a190167dcfd831ba1fc5f733e0dc9a6fda)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/runqueue: show correct task name for setscene task failure
Paul Eggleton [Mon, 28 Feb 2011 14:28:25 +0000 (14:28 +0000)]
bitbake/runqueue: show correct task name for setscene task failure

If a setscene task failed previously it was showing an incorrect task
name in the error line. This patch ensures we show the correct name, also
including the "_setscene" suffix.

(From Poky rev: 9d9c68e3536387150e3ce7261f4770576e724dd9)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/runqueue: fix clash when setscene & real tasks done in same build
Paul Eggleton [Fri, 25 Mar 2011 13:22:01 +0000 (13:22 +0000)]
bitbake/runqueue: fix clash when setscene & real tasks done in same build

If a build causes a real task to be run when the setscene task has already
run then it was possible for dependent packages to be rebuilding at the same
time as a rebuild of the packages they depended on, resulting in failures
when files were missing. This change looks in the setscene covered list and
removes anything where a dependency of the real task is going to be run (e.g.
do_install is going to be run even though the setscene equivalent of
do_populate_sysroot has already been run).

As an additional safeguard we also delete the stamp file for the setscene
task under these circumstances.

Fixes [YOCTO #792]

(From Poky rev: b4268c08c350a7928a0b1a041b04ffe5a44e77b4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agocooker/process.py: Move profiling code to a place it can be reused by different serve...
Richard Purdie [Wed, 8 Jun 2011 10:44:50 +0000 (11:44 +0100)]
cooker/process.py: Move profiling code to a place it can be reused by different server mechanisms

The cooker profiling code isn't server specific so move it to a place
where different server backends can use it.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agocache: Implement multiple extra cache fields request support
Liping Ke [Fri, 3 Jun 2011 00:22:40 +0000 (08:22 +0800)]
cache: Implement multiple extra cache fields request support

This patch is to support extra cache. If user needs to request
extra cache fields besides CoreRecipeInfo fields, just add a new
XXXRecipeInfo class definition as Hob Does.
Currently supported Extra RecipeInfo name is an array. We can
support multiple extra cache fields at the same time besides
CoreRecipeInfo which is needed by all clients.

Signed-off-by: Liping Ke <liping.ke@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agocache: Introduce new param caches_array into Cache impl.
Liping Ke [Fri, 3 Jun 2011 00:21:44 +0000 (08:21 +0800)]
cache: Introduce new param caches_array into Cache impl.

When using hob ui interface, we need extra cache fields.
We will save ui required extra cache fields into a separate
cache file. This patch introduce this caches_array parameter.
It will be used in the extra cache implementation (following patch).
Caches_array at least contains CoreRecipeInfo. If users need extra
cache fields support, such as 'hob', caches_array will contain
more relevant elements such as HobRecipeInfo.

Signed-off-by: Liping Ke <liping.ke@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agocache: Introduce extra cache class for image creator
Liping Ke [Fri, 3 Jun 2011 00:20:15 +0000 (08:20 +0800)]
cache: Introduce extra cache class for image creator

Extra RecipeInfo will be all defined in this file. Currently,
Only Hob (Image Creator) Requests some extra fields. So
HobRecipeInfo is defined. It's named HobRecipeInfo because it
is introduced by 'hob'. Users could also introduce other
RecipeInfo or simply use those already defined RecipeInfo.
In the following patch, this newly defined new extra RecipeInfo
will be dynamically loaded and used for loading/saving the extra
cache fields.

Signed-off-by: Liping Ke <liping.ke@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agocache.py: Refactory Current Cache implementation
Liping Ke [Fri, 3 Jun 2011 00:17:23 +0000 (08:17 +0800)]
cache.py: Refactory Current Cache implementation

This patch is for refactorying current cache implementation, the
main reason is for introducing extra cache fields requests for
image creator as well as other users. The refactory parts include:
Move cache data retrieve methods out of Cache Data Fields
Definition. Since this retrieve methods will be shared for
both CoreRecipeInfo as well as the new introduced extra RecipeInfo
in the following patches.

Signed-off-by: Liping Ke <liping.ke@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agocache.py: Sync what amounts to whitespace with bitbake in poky, pending merge of...
Richard Purdie [Tue, 7 Jun 2011 21:32:59 +0000 (22:32 +0100)]
cache.py: Sync what amounts to whitespace with bitbake in poky, pending merge of cache patches

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/cooker: don't error in prepareTreeData for unbuildable targets
Joshua Lock [Sat, 5 Mar 2011 17:13:32 +0000 (17:13 +0000)]
bitbake/cooker: don't error in prepareTreeData for unbuildable targets

Set abort to False in prepareTreeData so that unbuildable targets do not
raise an exception.

(From Poky rev: 3c113f355bce012d55dac873189a410657016a62)

Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agouniverse target: add a new target to collect all recipe targets
Saul Wold [Thu, 21 Apr 2011 18:44:45 +0000 (11:44 -0700)]
universe target: add a new target to collect all recipe targets

This new universe target is not intended to be used for
compiling or building everything, it use is for sanity checking
and other tasks that need to find all targets. This does not
exclude any broken or virtual targets.

(From Poky rev: 28e7041a9c110be2ac5dea1eb1f55ca8f056111e)

Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobb.exceptions: don't choke on frames without arguments
Chris Larson [Fri, 3 Jun 2011 18:29:44 +0000 (14:29 -0400)]
bb.exceptions: don't choke on frames without arguments

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agosiggen: don't choke with traceback when data is None
Chris Larson [Thu, 2 Jun 2011 21:46:13 +0000 (14:46 -0700)]
siggen: don't choke with traceback when data is None

Given we use bb.error, not bb.fatal, here, it seems this was intended to be
non-fatal, yet we'd end up trying to concatenate None. Fix this by setting an
empty task to the empty string, for the purposes of hashing. Also str() the
value we get from the datastore, just in case something other than a string
was stored there.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agoparse: pass a useful (if mangled) key to bb.data.expand for :=
Chris Larson [Thu, 2 Jun 2011 21:45:37 +0000 (14:45 -0700)]
parse: pass a useful (if mangled) key to bb.data.expand for :=

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agocooker: handle ExpansionError the same way we do ParseError
Chris Larson [Thu, 2 Jun 2011 21:45:13 +0000 (14:45 -0700)]
cooker: handle ExpansionError the same way we do ParseError

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agoevent: don't catch systemexit from handler execution
Chris Larson [Thu, 2 Jun 2011 21:32:17 +0000 (14:32 -0700)]
event: don't catch systemexit from handler execution

This means that anyone firing an event can get a systemexit and result in
their process exiting, which isn't ideal, but behaves the way it used to (in
particular, ensures that a sanity check failure will halt the build). This
should be revisited in the future.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agoMerge remote branch 'github/exceptions'
Chris Larson [Thu, 2 Jun 2011 21:20:26 +0000 (14:20 -0700)]
Merge remote branch 'github/exceptions'

* github/exceptions:
  Shift exception formatting into the UI
  cooker: don't show a traceback for ParseError
  cooker: don't show a useless traceback for SyntaxError
  cooker: use logger.exception for config file parse errors
  cooker: pass traceback back from parsing thread
  cooker: show a useful message for ParsingFailure
  bb.exceptions: don't show a repr of 'self'
  codeparser.py: fix syntax error in exception handling
  codeparser.py: Ignore incomplete cache files
  bb.exceptions: handle tb entries without context
  bb.exceptions: add to_string convenience function
  bb.exceptions: add code to create pickleable traceback entries
  bb.namedtuple_with_abc: add useful util from activestate

8 years agodata_smart: Adding overrides of zero length is pointless so lets not
Richard Purdie [Wed, 1 Jun 2011 13:01:32 +0000 (14:01 +0100)]
data_smart: Adding overrides of zero length is pointless so lets not

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake: use layer priority when applying bbappends
Paul Eggleton [Wed, 1 Jun 2011 10:09:34 +0000 (11:09 +0100)]
bitbake: use layer priority when applying bbappends

If the priority of a layer has been specified with BBFILE_PRIORITY_ then
use that to sort the list of BBFILES entries, which will affect the order
in which .bbappend files are applied.

Fixes [YOCTO #1125]

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agoproviders.py: Correct PREFERRED_VERSION handling
Richard Purdie [Tue, 31 May 2011 22:02:01 +0000 (23:02 +0100)]
providers.py: Correct PREFERRED_VERSION handling

Overrides on the right are the highest priority and in this case, pn-PN
and PN should take priority over any other override so fix the code to
do this.

Also, since overrides will have been processed by bitbake, we shouldn't
then be specifically looking up PREFERRED_VERSION_${PN} but just using
PREFERRED_VERSION.

This patch corrects the behaviours to match what the code is expected
to do.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agoserver: add updated/fixed xmlrpc server from Poky
Joshua Lock [Fri, 27 May 2011 20:03:55 +0000 (13:03 -0700)]
server: add updated/fixed xmlrpc server from Poky

Signed-off-by: Joshua Lock <josh@linux.intel.com>
8 years agouievent: fix queueing of events for xmlrpc before UI has loaded
Joshua Lock [Fri, 27 May 2011 20:04:44 +0000 (13:04 -0700)]
uievent: fix queueing of events for xmlrpc before UI has loaded

The change to Queue up events before the UI is spawned broke the xmlrpc
server because the uievent implementation of BBUIEventQueue expects pickled
strings for its queue_event() method.
This is because the RPC exposed event.send() method must accept pickled
strings, but for xmlrpc event.send() is just mapped to queue_event(). Work
around this by adding a send_event method which unpickles strings and hands
them off to queue_event() which can then be used for the remapping.

Signed-off-by: Joshua Lock <josh@linux.intel.com>
8 years agofetch2/git: ensure network check log matches actual command
Darren Hart [Fri, 20 May 2011 23:06:12 +0000 (16:06 -0700)]
fetch2/git: ensure network check log matches actual command

The git command string logged via check_network_access() does not match
the actual command executed in a few places. Ensure that it does.

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agofetch2/git: use logging.debug() and clarify messages
Darren Hart [Fri, 20 May 2011 23:06:14 +0000 (16:06 -0700)]
fetch2/git: use logging.debug() and clarify messages

Replace a call to print() with logging.debug() and flesh out the
message to clarify the state being reported.

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/codeparser: Improve cache handling
Richard Purdie [Fri, 27 May 2011 15:14:48 +0000 (16:14 +0100)]
bitbake/codeparser: Improve cache handling

The current codeparser cache handling hurts performance badly even
over a couple of cores and certainly on many core systems, it can
spent huge amounts of time in the codeparser cache save functions.

This patch reworks the cache handling so that each parsing thread
saves out its own "differences" file compared to any existing core
cache and then the main bitbake thread picks these up and merges
things back together.

This was tested on systems with small and large numbers of cores
and was found to perform orders of magnitude better in all cases
despite the more complex code.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agorunqueue.py: Set BB_WORKERCONTEXT in task execution context
Richard Purdie [Fri, 27 May 2011 16:16:11 +0000 (17:16 +0100)]
runqueue.py: Set BB_WORKERCONTEXT in task execution context

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/runqueue.py: Ensure existing setscene stamp files are taken into account
Richard Purdie [Fri, 27 May 2011 14:03:51 +0000 (15:03 +0100)]
bitbake/runqueue.py: Ensure existing setscene stamp files are taken into account

JaMa reported issues where bitbake would rebuild things instead of using the
existing built tasks. This was tracked to a case where:

a) rm_work is uses
b) A depends on B
c) B has a version change (e.g. PR bump)

and A *and* B would then rebuild.

It turns out that rm_work was correctly turning stamp files into the correct
_setscene varients but bitbake was then ignoring them during setscene processing.
If the correct sstate checksumed files didn't exist, everything would seemingly
rebuild.

The fix is to check for existing *_setscene stamps and if present, honour them.
If "basichash" is enabled, the hash is included with the stamps so everything
should then function as intended.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/utils.py: Add option to lockfiles to return immediately rather than wait
Richard Purdie [Fri, 27 May 2011 15:13:27 +0000 (16:13 +0100)]
bitbake/utils.py: Add option to lockfiles to return immediately rather than wait

There are usecases where we don't want to block waiting for a lockfile
so enhance the lockfile handling functions to support this.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/cooker.py: Ensure BBFILES is processed in order
Richard Purdie [Fri, 27 May 2011 14:13:54 +0000 (15:13 +0100)]
bitbake/cooker.py: Ensure BBFILES is processed in order

The files found by collect_bbfiles should be processed in order but due
to being processed using python's set(), the order was not being preserved.

Use a list instead as whilst the code is slightly more ugly, order
is preserved.

Addresses [YOCTO #1100]

Acked-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/cooker.py: Fix -b option regexp handling
Richard Purdie [Fri, 27 May 2011 14:11:49 +0000 (15:11 +0100)]
bitbake/cooker.py: Fix -b option regexp handling

bitbake -b core-image was showing no matches when it should list all targets
containing the fragment "core-image". This patch only calls os.path.abspath()
on things that look like paths and passed the path around more consistently to
various functions to get this use case working again.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agofetch2/git: enhance the unpack by using "git checkout"
Yu Ke [Thu, 26 May 2011 07:37:56 +0000 (15:37 +0800)]
fetch2/git: enhance the unpack by using "git checkout"

current git fetcher unpack method only checkout index and working tree,
but did not did not update the git branch in ref/heads, so user may not
get right info in ${S} by using git.

this patch enhance the unpack by using git checkout to fix this issue.

Fix bug [YOCTO #1089]

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agofetch2/git: add document for git fetcher supported options
Yu Ke [Tue, 24 May 2011 06:22:52 +0000 (14:22 +0800)]
fetch2/git: add document for git fetcher supported options

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agofetch2/git: unify the nocheckout option format
Yu Ke [Thu, 26 May 2011 07:29:37 +0000 (15:29 +0800)]
fetch2/git: unify the nocheckout option format

make the nocheckout option format to be: default is "0",
use nocheckou=1 to set this option

with this patch, the format will be consistant with other bitbake options
like rebaseable

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agofetch2/git: change default protocol from rsync to git
Yu Ke [Tue, 24 May 2011 07:41:14 +0000 (15:41 +0800)]
fetch2/git: change default protocol from rsync to git

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agoAdd PR service deamon to bitbake
Lianhao Lu [Fri, 27 May 2011 06:31:45 +0000 (14:31 +0800)]
Add PR service deamon to bitbake

Added the initial implementation of the server side PR service.

Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agocooker.py: Don't show spurious warnings for collections of .bbappend files
Richard Purdie [Wed, 25 May 2011 22:45:31 +0000 (23:45 +0100)]
cooker.py: Don't show spurious warnings for collections of .bbappend files

Seeing warnings like:

WARNING: No bb files matched BBFILE_PATTERN_yocto '^/xxx/meta-yocto/'

are not encouraging to users and we shouldn't show these if we found
.bbappend files (but no .bb files). This change stops these warnings
from appearing.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobuild/siggen: Ensure a task signature file is generated for each task that is executed
Richard Purdie [Wed, 25 May 2011 22:19:41 +0000 (23:19 +0100)]
build/siggen: Ensure a task signature file is generated for each task that is executed

Tracing down signature changes and dependencies is hard unless the complete task
chain can be evaultated. Since we have the data available at task exeuction
time, writing it to disk makes sense.

This change is suboptimal internal knowledge stamp_internal() has leaks into
make_stamp() and the concepts used by this code need revisiting but the change
brings enough usability benefit to warrant its inclusion despite that.

Addresses issue [YOCTO #1074]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agogit fetcher: add support for rebaseable git repo
Yu Ke [Sun, 15 May 2011 14:33:30 +0000 (22:33 +0800)]
git fetcher: add support for rebaseable git repo

Some upstream git repo may rebase in the future, which means current
revision may disappear from the upstream repo after the rebase.

current git fetcher can not handle this case, because the git mirror
tar ball is per repo, and may also change in the rebase and lost the
current revision info.

To fix this issue, this patch
- add rebaseable tag in the SRC_URI
- for rebaseable repo, make git mirror tar ball per revision, in this
  case, even upstream rebase, the git mirror still has the current
  revision info.
- for rebaseable repo, generate mirror tar ball by default, since the
  repo may change in the future.

Signed-off-by: Yu Ke <ke.yu@intel.com>
8 years agobitbake/ui: Fix Gtk+ GUI's after recent cache changes
Joshua Lock [Wed, 11 May 2011 00:07:56 +0000 (17:07 -0700)]
bitbake/ui: Fix Gtk+ GUI's after recent cache changes

Signed-off-by: Joshua Lock <josh@linux.intel.com>
8 years agoShift exception formatting into the UI
Chris Larson [Fri, 6 May 2011 00:43:38 +0000 (17:43 -0700)]
Shift exception formatting into the UI

Now we use bb.exceptions to pass pickleable traceback entries to the UI, and
the UI is free to do whatever it wants to do with this information. By
default, the log formatter for the UIs formats it with bb.exceptions.  This
also means that all exceptions should now show 3 lines of context and limit to
5 entries.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agocooker: don't show a traceback for ParseError
Chris Larson [Fri, 6 May 2011 14:18:44 +0000 (07:18 -0700)]
cooker: don't show a traceback for ParseError

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agocooker: don't show a useless traceback for SyntaxError
Chris Larson [Fri, 6 May 2011 14:15:34 +0000 (07:15 -0700)]
cooker: don't show a useless traceback for SyntaxError

Note: we rely on the fact that better_compile has already output information
about the exception, so don't do it ourselves.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agocooker: use logger.exception for config file parse errors
Chris Larson [Fri, 6 May 2011 02:21:12 +0000 (19:21 -0700)]
cooker: use logger.exception for config file parse errors

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agocooker: pass traceback back from parsing thread
Chris Larson [Thu, 5 May 2011 23:55:33 +0000 (16:55 -0700)]
cooker: pass traceback back from parsing thread

Uses bb.exceptions to get a traceback back from the parsing thread to the main
thread, where it is then formatted.  Also enables 3 lines of context for the
formatted traceback, and limits the number of entries displayed to 5.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agocooker: show a useful message for ParsingFailure
Chris Larson [Thu, 5 May 2011 23:50:34 +0000 (16:50 -0700)]
cooker: show a useful message for ParsingFailure

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agobb.exceptions: don't show a repr of 'self'
Chris Larson [Sat, 14 May 2011 00:35:37 +0000 (17:35 -0700)]
bb.exceptions: don't show a repr of 'self'

Rather than treating self like an ordinary argument, showing a repr of its
value in the function spec when formatting the traceback entry, now we show
the class name for the method as a part of the function name. Example:

Old: bar(self=<some repr of Fooclass>, f=5)
New: Fooclass.bar(f=5)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agocodeparser.py: fix syntax error in exception handling
Otavio Salvador [Fri, 13 May 2011 16:46:32 +0000 (16:46 +0000)]
codeparser.py: fix syntax error in exception handling

Commit 036cf3cd11b3a6836b77f5ffa760ceee6b71b1ef missed the needed
brackets to handle more then a type of exception.

Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
8 years agocodeparser.py: Ignore incomplete cache files
Richard Purdie [Thu, 12 May 2011 22:35:53 +0000 (23:35 +0100)]
codeparser.py: Ignore incomplete cache files

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobb.exceptions: handle tb entries without context
Chris Larson [Fri, 6 May 2011 02:07:21 +0000 (19:07 -0700)]
bb.exceptions: handle tb entries without context

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agobb.exceptions: add to_string convenience function
Chris Larson [Fri, 6 May 2011 02:06:35 +0000 (19:06 -0700)]
bb.exceptions: add to_string convenience function

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agobb.exceptions: add code to create pickleable traceback entries
Chris Larson [Thu, 5 May 2011 23:53:52 +0000 (16:53 -0700)]
bb.exceptions: add code to create pickleable traceback entries

This lets you get as much useful information as possible from traceback
entries while staying pickleable.

In addition, it has improved traceback formatting.  It shows the values of the
arguments for the functions, lines of context from the file for the code, and
has an optional formatter to do things like syntax highlighting for the code
lines.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agobb.namedtuple_with_abc: add useful util from activestate
Chris Larson [Thu, 5 May 2011 23:53:44 +0000 (16:53 -0700)]
bb.namedtuple_with_abc: add useful util from activestate

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agocodeparser.py: fix syntax error in exception handling
Otavio Salvador [Fri, 13 May 2011 16:46:32 +0000 (16:46 +0000)]
codeparser.py: fix syntax error in exception handling

Commit 036cf3cd11b3a6836b77f5ffa760ceee6b71b1ef missed the needed
brackets to handle more then a type of exception.

Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
8 years agocodeparser.py: Ignore incomplete cache files
Richard Purdie [Thu, 12 May 2011 22:35:53 +0000 (23:35 +0100)]
codeparser.py: Ignore incomplete cache files

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/utils.py: Only try and add read access to a file if we don't have it
Richard Purdie [Fri, 25 Feb 2011 17:22:00 +0000 (17:22 +0000)]
bitbake/utils.py: Only try and add read access to a file if we don't have it

A file we're copying might be on a readonly filesystem so if we can already read
it, don't try and add read permission.

Fixes BUGID #771 in Yocto.

(From Poky rev: 9166b9e32fd6f618f9597b07d88cef09a88916a1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/fetch2/local: Fix inverted update required logic
Richard Purdie [Wed, 9 Mar 2011 19:01:35 +0000 (11:01 -0800)]
bitbake/fetch2/local: Fix inverted update required logic

(From Poky rev: 4f28cd2d1854df8e6f56544fe509fb2e3ddce9aa)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/fetch2: Allow local file:// urls to be found on mirrors
Richard Purdie [Tue, 8 Mar 2011 19:23:34 +0000 (11:23 -0800)]
bitbake/fetch2: Allow local file:// urls to be found on mirrors

With the current implementation, file:// urls as used by sstate don't access the
mirror code, breaking sstate mirror support. This change enables the usual
mirror handling. To do this, we remove the localfile special case, using the basename
paramemter instead. We also ensure the downloads directory is checked for files.

The drawback of this change is that file urls containing "*" globing require special
casing in the core.

(From Poky rev: a778fb311540580476976e43f9c0576284f8dc38)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/fetch2: Fix the problems introduced by the git fetcher AUTOREV fix
Richard Purdie [Wed, 20 Apr 2011 01:13:23 +0000 (02:13 +0100)]
bitbake/fetch2: Fix the problems introduced by the git fetcher AUTOREV fix

The ordering constrains on the urldata_init functions are not straight
forward. To avoid further problems, create a helper function to setup
the source revisions which the init functions can all at the appropriate
point.

(From Poky rev: c4371138f7444ecaa1fdd2b1ee4949fbc819f886)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/fetch2/git: Fix a bug where AUTOREV and the git fetcher interact badly
Richard Purdie [Tue, 19 Apr 2011 22:31:09 +0000 (23:31 +0100)]
bitbake/fetch2/git: Fix a bug where AUTOREV and the git fetcher interact badly

Fix a bug where ud.branches were being referenced before it was set by
the git fetcher when using AUTOREV. To do this some ordering needed
to be changed. This fixes errors like:

ERROR: Error parsing /recipes-kernel/linux/rt-tests_git.bb: Failure expanding variable
SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
AttributeError: 'FetchData' object has no attribute 'branches'

(From Poky rev: 2d2d07275c8a2d652a221c1cd74d28c2f6f226d5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobuild.py: Fix ordering bug introduced in 7a29ab534388c0095f7f826b16c5cff343927d10
Richard Purdie [Fri, 6 May 2011 14:46:35 +0000 (15:46 +0100)]
build.py: Fix ordering bug introduced in 7a29ab534388c0095f7f826b16c5cff343927d10

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake docs: use dblatex to build the pdf bitbake manual
Darren Hart [Wed, 9 Mar 2011 20:55:20 +0000 (12:55 -0800)]
bitbake docs: use dblatex to build the pdf bitbake manual

Fix [BUGID #593]

The current manual build fails for printing formats which use latex as an
intermediate format. This bug has been reported in multiple locations and I
haven't found a solution posted to any of them.

Using --with-dblatex uses dblatex to make the conversion and successfully
generates the pdf. It adds a dependency on dblatex and its dependencies.

(From Poky rev: b26640f30a4c80fcedda0d19bcf96ed412062a33)

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
CC: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agocorrect typo in ??= documentation
Darren Hart [Wed, 13 Apr 2011 00:11:52 +0000 (17:11 -0700)]
correct typo in ??= documentation

??= is a lazy version of ?=

(From Poky rev: 2ee0be82d065aeee716a9c0289bf111ea121e6dc)

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agogit fetcher: make tag back to work, fix Yocto bug 972
Yu Ke [Mon, 18 Apr 2011 05:53:53 +0000 (13:53 +0800)]
git fetcher: make tag back to work, fix Yocto bug 972

In current git fetcher, tag does not work due to commit http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=5920e85c561624e657c126df58f5c378a8950bbc. Tag is not in sha256 form, so it will be treated invalid, and silently replaced by latest revision.

To fix it, this patch treat tag name as branches name, thus it will be handled correctly later. Thanks Richard for reviewing and proposing the better approach.

Fix [YOCTO #972]

CC: Richard Purdie <richard.purdie@linuxfoundation.org>
(From Poky rev: a2030c8d1a58c79428cba80158982aea4d2d9f6d)

Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/codeparser: Correctly handle a missing/empty cache file
Richard Purdie [Thu, 5 May 2011 18:27:46 +0000 (19:27 +0100)]
bitbake/codeparser: Correctly handle a missing/empty cache file

(From Poky rev: 72875493b8bbb5d6793380ee71c6bca4f438ca04)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/cache.py: Ensure skipped recipes make it into the cache to avoid reparsing
Richard Purdie [Sun, 10 Apr 2011 17:55:48 +0000 (10:55 -0700)]
bitbake/cache.py: Ensure skipped recipes make it into the cache to avoid reparsing

(From Poky rev: 001a555c2f755d4f8a69b113656d9307ca7ee597)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/cooker/codeparser: Ensure the code parser cache is saved for each parsing...
Richard Purdie [Sun, 10 Apr 2011 17:52:29 +0000 (10:52 -0700)]
bitbake/cooker/codeparser: Ensure the code parser cache is saved for each parsing process

Before this change, the codeparser cache was only being saved for the main
server process. This is suboptimal as it leaves code being re-evaluated at
task execution time and increases parse time.

We use the multiprocess Finalize() functionality to ensure each process
saves out its cache. We need to update the cache save function to be multiprocess
friendly with locking.

(From Poky rev: c8928e93dd8a08d6768623f6491c9ba31d0aa06f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agobitbake/ast.py: Only run finalise() for the specified variant
Richard Purdie [Sun, 10 Apr 2011 17:50:16 +0000 (10:50 -0700)]
bitbake/ast.py: Only run finalise() for the specified variant

Allows the heavy finalise function to only be run for the case we're
interested in when running tasks, saving some processing time.

(From Poky rev: 9211fd9c375489c73924fd43f1f8a0da2c4290bb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
8 years agomanual: fix typo
Bernhard Reutner-Fischer [Mon, 2 May 2011 09:45:56 +0000 (11:45 +0200)]
manual: fix typo

"is used signify" -> "is used to signify"

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
8 years agobuild: run fakeroot if FAKEROOT is set
Chris Larson [Fri, 8 Apr 2011 14:10:40 +0000 (07:10 -0700)]
build: run fakeroot if FAKEROOT is set

FAKEROOT contains the fakeroot command to be run.  This should not be set for
those using pseudo + FAKEROOTENV, but should be set for people like upstream
OE, and its value should be 'fakeroot' for them, for compatibility with the
old mechanisms.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agofetch.git: fix a remnant wrt persist + keyerror
Christopher Larson [Thu, 7 Apr 2011 03:34:53 +0000 (20:34 -0700)]
fetch.git: fix a remnant wrt persist + keyerror

Signed-off-by: Christopher Larson <kergoth@gmail.com>
8 years agopersist_data: implement comparison, same as dict
Chris Larson [Mon, 4 Apr 2011 16:36:45 +0000 (09:36 -0700)]
persist_data: implement comparison, same as dict

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agopersist_data: make SQLTable a context manager
Chris Larson [Thu, 10 Feb 2011 00:57:03 +0000 (17:57 -0700)]
persist_data: make SQLTable a context manager

This can be used for more control over the underlying transactions.  Unlike
the context manager of, say, a file object, we can still use the object even
after the end of a given with block, as the context manager exit only ensures
we've committed to the database, not that we have closed the database.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agopersist_data: don't allow non-string keys/values
Chris Larson [Mon, 4 Apr 2011 16:37:07 +0000 (09:37 -0700)]
persist_data: don't allow non-string keys/values

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agopersist_data: add has_key
Chris Larson [Mon, 4 Apr 2011 16:36:22 +0000 (09:36 -0700)]
persist_data: add has_key

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agopersist_data: raise KeyError on missing elements
Chris Larson [Mon, 4 Apr 2011 16:36:10 +0000 (09:36 -0700)]
persist_data: raise KeyError on missing elements

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agosetup.py: install fetch2 module
Martin Jansa [Thu, 3 Mar 2011 18:07:41 +0000 (18:07 +0000)]
setup.py: install fetch2 module

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agosetup.py: also install bitbake-{layers,diffsigs} scripts
Martin Jansa [Thu, 3 Mar 2011 22:52:48 +0000 (22:52 +0000)]
setup.py: also install bitbake-{layers,diffsigs} scripts

Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agocooker: fix -b with BBCLASSEXTEND
Chris Larson [Mon, 4 Apr 2011 21:37:59 +0000 (14:37 -0700)]
cooker: fix -b with BBCLASSEXTEND

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agoevent: improve output for syntax errors in handlers
Chris Larson [Mon, 4 Apr 2011 20:51:51 +0000 (13:51 -0700)]
event: improve output for syntax errors in handlers

Note: this includes IndentationError, which is a subclass of SyntaxError.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agoevent: improve output when eventhandler exec fails
Chris Larson [Mon, 4 Apr 2011 18:31:39 +0000 (11:31 -0700)]
event: improve output when eventhandler exec fails

- Name the event handler by its actual name, so the traceback shows it rather
  than 'tmpHandler'.
- Rather than immediately aborting when encountering an event handler error,
  display an error message and try to continue.
- Show a traceback for ordinary exceptions, skipping the first entry in the
  traceback, so it only shows the useful information.
- Show an error, but no traceback, for SystemExit with a code other than 0.
- For for SystemExit with a code of 0, simply continue silently.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agoevent: register event handler functions, not code objects
Chris Larson [Mon, 4 Apr 2011 16:41:41 +0000 (09:41 -0700)]
event: register event handler functions, not code objects

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agobuild.py: Merge fakeroot changes from Poky
Mark Hatle [Fri, 1 Apr 2011 21:30:11 +0000 (16:30 -0500)]
build.py: Merge fakeroot changes from Poky

Changes imported from poky repository, master branch as of commit:
bcc817ee058bbab91d6e88a3e1e867a4ff09aeca

In Poky fakeroot processing is handled exclusively by FAKEROOTENV.  The
fakeroot binary itself is never invoked directly.  (This is due to Poky
using pseudo, a fakeroot replacement that is more efficiently run by
setting environment variables.)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agobuild: don't choke if olddir no longer exists
Chris Larson [Fri, 25 Mar 2011 16:11:34 +0000 (09:11 -0700)]
build: don't choke if olddir no longer exists

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agobuild: fix dir removal traceback
Chris Larson [Fri, 25 Mar 2011 15:58:52 +0000 (08:58 -0700)]
build: fix dir removal traceback

This one is to cover the case where the current directory vanishes out from
under us, so os.getcwd() raises an OSError.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agobuild: in exec_func, mkdirhier ${T}
Chris Larson [Fri, 25 Mar 2011 14:55:26 +0000 (07:55 -0700)]
build: in exec_func, mkdirhier ${T}

This should fix the -c clean traceback people are seeing.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agorunqueue: simplify fakeroot environment handling
Chris Larson [Fri, 18 Mar 2011 16:42:33 +0000 (09:42 -0700)]
runqueue: simplify fakeroot environment handling

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agobuild: add missing bit necessary for mhatle's fakeroot fix
Chris Larson [Fri, 18 Mar 2011 16:56:50 +0000 (09:56 -0700)]
build: add missing bit necessary for mhatle's fakeroot fix

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agorunqueue.py: Enable PSEUDO (fakeroot) before the fork
Mark Hatle [Thu, 17 Mar 2011 23:10:18 +0000 (19:10 -0400)]
runqueue.py: Enable PSEUDO (fakeroot) before the fork

PSEUDO when used as the fakeroot program is usually preloaded into memory,
but disabled by default.  The trigger for enabling it is a set of environment
variables and a fork() or exec*() operation.

We need to setup the environment, specifically PSEUDO_DISABLED=0, prior to
the fork() so that python tasks can be run under PSEUDO control.

This patch is based on the work in Poky primarily from Richard Purdie.

See the following Poky commits for detailed history on this change:

commit 1f3e313fd5cc5ae8ea838bf8fcdedace3cb72584
Author: Richard Purdie <rpurdie@linux.intel.com>
Date:   Wed Dec 8 00:08:04 2010 +0000

    bitbake Revert bitbake exec() and go back to fork() for performace wins (first draft)

Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
commit 52aada2eaf08d57d5f715f155f2d878831dbaab0
Author: Richard Purdie <rpurdie@linux.intel.com>
Date:   Tue Dec 21 00:51:24 2010 +0000

    bitbake/runqueue.py: Somehow the python environment mapping is failing so do it manually

Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
commit 1b08a7eb8b708f4d0fc119cf89deb450fa62fea1
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Mon Feb 28 15:31:20 2011 +0000

    bitbake/cache/runqueue.py: Move workload for recipe parsing to the child process

    Parsing the recipe in the parent before forking off the child worker
    can mean the parent doesn't hit the idle loop and becomes a bottleneck
    when lauching many short lived processes.

    The reason we need this in the parent is to figure out the fakeroot
    environmental options. To address this, add the fakeroot variables
    to the cache and move recipe loadData into the child task.

    For a poky-image-sato build this results in about a 2 minute speedup
    (1.8%).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Integrated and merged into bitbake upstream

Signed-off-by: Mark Hatle <Mark.Hatle@windriver.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agoInitial work on getting bitbake working under pypy
Chris Larson [Wed, 16 Mar 2011 15:07:29 +0000 (08:07 -0700)]
Initial work on getting bitbake working under pypy

- use os.chmod, not os.fchmod, as the latter is missing under pypy
- rearrange our imports a bit
- don't die if sqlite3 is missing shared cache support

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agoMore 'is' fixups
Chris Larson [Wed, 16 Mar 2011 15:07:37 +0000 (08:07 -0700)]
More 'is' fixups

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agoFix more incorrect usages of 'is'
Chris Larson [Tue, 8 Mar 2011 19:07:24 +0000 (12:07 -0700)]
Fix more incorrect usages of 'is'

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agocodeparser: use ==, not 'is' to compare strings
Chris Larson [Tue, 8 Mar 2011 18:54:45 +0000 (11:54 -0700)]
codeparser: use ==, not 'is' to compare strings

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agocodeparser: fix ShellSyntaxError usage
Chris Larson [Wed, 16 Mar 2011 14:38:18 +0000 (07:38 -0700)]
codeparser: fix ShellSyntaxError usage

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agobb.siggen: import os
Chris Larson [Tue, 8 Mar 2011 23:29:22 +0000 (16:29 -0700)]
bb.siggen: import os

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agoutils: fix typo in lockfile
Chris Larson [Tue, 8 Mar 2011 16:39:40 +0000 (09:39 -0700)]
utils: fix typo in lockfile

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agolockfile: ask for forgiveness, not permission
Chris Larson [Tue, 8 Mar 2011 16:33:40 +0000 (09:33 -0700)]
lockfile: ask for forgiveness, not permission

Create the lockfile directory if it doesn't exist, rather than erroring out if
it doesn't exist (was also racy).

Also improve the wording of the error message shown when the lockfile's
directory is not writable.

Note for the future, this function should be improved, particularly with
regard to its exception handling. It should be catching the *exact*
exception(s) it will encounter when the file is locked, and continuing in that
case only. If it did that, there'd be no need for the proactive directory
writability check, as bb.utils.lockfile() would raise an appropriate IOError
for that case.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agohob: adjust for master's eventHandler API
Chris Larson [Mon, 7 Mar 2011 23:01:21 +0000 (16:01 -0700)]
hob: adjust for master's eventHandler API

Signed-off-by: Chris Larson <chris_larson@mentor.com>
8 years agogoggle: exit quietly on ^C
Chris Larson [Mon, 7 Mar 2011 22:54:29 +0000 (15:54 -0700)]
goggle: exit quietly on ^C

Signed-off-by: Chris Larson <chris_larson@mentor.com>