bitbake.git
5 years agofetch2: Don't allow '/' in user:pass, fix branch containing '@' 1.20 origin/1.20
Martin Jansa [Tue, 21 Jan 2014 15:46:28 +0000 (16:46 +0100)]
fetch2: Don't allow '/' in user:pass, fix branch containing '@'

* currently decode_url regexp parses branch=@foo as username so it ends like this:
  - ('git', '', 'foo', 'git.openembedded.org/bitbake;branch=', '', {})
  + ('git', 'git.openembedded.org', '/bitbake', '', '', {'branch': '@foo'})
* http://hg.python.org/cpython/file/2.7/Lib/urlparse.py also assumes
  that there is at least one '/' as separator between netloc and path,
  params, so it looks reasonable to prevent including '/' in username

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
5 years agohob/hoblistmodel: check if vals of packages/recipes names are not None
Cristiana Voicu [Wed, 8 Jan 2014 17:40:04 +0000 (17:40 +0000)]
hob/hoblistmodel: check if vals of packages/recipes names are not None

[YOCTO #5053]

(Corresponds to BitBake master rev: ba9fe77e37be31e8246431578902e871dd94515e)

Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
5 years agoimagedetailspage: Fix crash with more than 15 layers
Richard Purdie [Mon, 2 Dec 2013 17:41:58 +0000 (17:41 +0000)]
imagedetailspage: Fix crash with more than 15 layers

If you had more than 15 layers the system would crash since one more
value is added to one array than the other. This fixes the code
so equal numbers of values are added to the arrays and hence
doesn't crash when many layers are enabled.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
5 years agoparse/ConfHander/BBHandler/utils: Fix cache dependency bugs
Richard Purdie [Fri, 29 Nov 2013 23:15:56 +0000 (23:15 +0000)]
parse/ConfHander/BBHandler/utils: Fix cache dependency bugs

Currently bitbake only adds files to its dependency list if they exist.
If you add 'include foo.inc' to your recipe and the file doesn't exist,
then later you add the file, the cache will not be invalidated.

This leads to another bug which is that if files don't exist and then
you add them and they should be found first due to BBPATH, again the
cache won't invalidate.

This patch adds in tracking of files we check for the existence of so
that if they are added later, the cache correctly invalidates. This
necessitated a new version of bb.utils.which which returns a list of
files tested for.

The patch also adds in checks for duplicate file includes and for now
prints a warning about this. That will likely become a fatal error at
some point since its never usually desired to include a file twice.

The same issue is also fixed for class inheritance. Now when a class
is added which would be found in the usual search path, it will cause
the cache to be invalidated.

Unfortunately this is old code in bitbake and the patch isn't the
neatest since we have to work within that framework.

[YOCTO #5611]
[YOCTO #4425]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
5 years agodata: Fix output inconsistencies for emit_var
Richard Purdie [Fri, 22 Nov 2013 16:17:39 +0000 (16:17 +0000)]
data: Fix output inconsistencies for emit_var

VAL = ""     (not shown)
VAL = " "    (shown as "")
VAL = " x"   (shown as "x")

would all show up rather differently to what would be expected in the
bitbake -e output. This fixes things so they appear consistently.

The output for running some shell functions may also change slightly
but shouldn't change in a way that is likely to cause problems.

[YOCTO #5507]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
5 years agorunqueue/bitbake-worker: Fix dry run fakeroot issues
Richard Purdie [Fri, 22 Nov 2013 13:31:47 +0000 (13:31 +0000)]
runqueue/bitbake-worker: Fix dry run fakeroot issues

When using the dry run option (-n), bitbake would still try and fire
a specific fakeroot worker. This is doomed to failure since it might
well not have been built.

Add in some checks to prevent the failures.

[YOCTO #5367]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
5 years agobitbake: runqueue: Fix hole in setsceneverify skipped task logic
Richard Purdie [Tue, 10 Dec 2013 15:09:39 +0000 (10:09 -0500)]
bitbake: runqueue: Fix hole in setsceneverify skipped task logic

We have do_bundle_initramfs which is a task inserted after compile and
before build. It is not covered by sstate.

If we run a build with a valid sstate cache present, the setsceneverify
function realises it will rerun the do_compile step (due to the
bundle_initramfs task) and hence marks do_populate_sysroot to rerun.
do_install, a dependency of do_populate_sysroot is left as marked as
covered by sstate.

What we need to do is traverse the dependency tree for any setsceneverify
invalided task and ensure any dependencies are also invalidated. We can
stop at any point we reach another setscene task though.

This means the do_populate_sysroot task has the data from do_install
available and doesn't crash.

(Bitbake master rev: f21910157d873c030b149c4cdc5b57c5062ab5a6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
5 years agobitbake: build.py: add single-quotes around already-expanded directory name
Peter Seebach [Tue, 10 Dec 2013 15:09:38 +0000 (10:09 -0500)]
bitbake: build.py: add single-quotes around already-expanded directory name

If the computed name of a directory contains an undefined variable
reference, bitbake dutifully creates a directory with a name that has
${...} in it. However, the actual task script created then tries to cd
to that directory, and the cd command fails, because no such directory
exists -- because the shell has helpfully removed the ${...} which did
not match any actual variables.

Since we want the name to be used exactly-as-is, add single quotes around
the name so this doesn't cause strange failures running tasks, which
allows us to progress past such failures and get to a point where they
can be diagnosed.

(Bitbake master rev: 2809c2e6f2f35f9b08058950be896947ab5a0284)

Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
5 years agobitbake: fetch2: Fix handling of SCM mirrors in MIRRORS
Richard Purdie [Tue, 10 Dec 2013 15:09:37 +0000 (10:09 -0500)]
bitbake: fetch2: Fix handling of SCM mirrors in MIRRORS

If an SCM mirror is in PREMIRRORS, the tarball is downloaded and then found
by the "upstream" check and handled correctly.

If an SCM mirror is in MIRRORS, the tarball is downloaded but not used
since there is no "upstream" run after MIRRORS completes. It therefore
sits there useless and unused. This code change forces the upstream to
run after a mirror tarball is found and fixes the usage of SCM mirrors
in MIRRORS.

(Bitbake master rev: a66ee0994645aa5658b2f5ea134ed17d89f8751a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
5 years agobitbake: monitordisk: lower inode check warning to note
Olof Johansson [Tue, 10 Dec 2013 15:09:36 +0000 (10:09 -0500)]
bitbake: monitordisk: lower inode check warning to note

Filesystems like btrfs and reiserfs sets the inode count to 0, since
they don't have an inode concept. This is expected, and having a warning
show up every time you run bitbake can cause undue concern.

(Bitbake master rev: f3ac2d3678f48c68a250a0a20c08cf8687322d38)

Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
5 years agobitbake: cooker/command: Add error state for the server and use for pre_serve errors
Richard Purdie [Tue, 10 Dec 2013 15:09:35 +0000 (10:09 -0500)]
bitbake: cooker/command: Add error state for the server and use for pre_serve errors

Currently if errors occur when starting the PR service, there is a race that
occurs since the UI runs various commands including starting builds before
processing the CookerExit(). By adding the error state and refusing to run
async commands in this mode, builds are prevented from starting and the
UI reaches the exit code with the system shutting down cleanly.

(Bitbake master rev: 42fa34142ea685f91115a551e74416ca28ef1c91)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
5 years agoperforce: Fix path subdirectory issues
Richard Purdie [Fri, 22 Nov 2013 16:22:28 +0000 (16:22 +0000)]
perforce: Fix path subdirectory issues

With a SRC_URI = " \
p4://depot/folder/...;module=localfolder/localsubfolder;changeslist=${P4CHANGELIST} \
"

the subfolders of //depot/folder/... get renamed when mapped to the
local folder structure. They lose the first 3 letters. This
patch fixes that.

Issue reported by and patch sent from katutxakurra@gmail.com

[YOCTO #5380]

(Bitbake master rev: 40e06dc459d9c0b5d42d65b2d2c846196fd36b1f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
5 years agofetch/hg: Improve user/password handling
Volker Vogelhuber [Fri, 1 Nov 2013 17:53:09 +0000 (17:53 +0000)]
fetch/hg: Improve user/password handling

Trying to use a server with username and password authentication
within the URL of the SRC_URI variable doesn't appear to work.

This patch adds the missing parts to the hg fetcher to make this
work properly.

(Bitbake master rev: dc3d6d73e44802c203b3f7247f6f212acc2f69bf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
5 years agofetch2/svn.py: use log instead of info to retrieve revision
Nicolas Dechesne [Fri, 1 Nov 2013 00:36:25 +0000 (17:36 -0700)]
fetch2/svn.py: use log instead of info to retrieve revision

We have faced a corner case situation where the 'last changed
revision' returned from svn info is wrong. It happens when the last
revision is a directory move. e.g. if we assume that the svn
repository at revA has root/x/y/z/foo/bar and it is moved to
root/a/b/c/foo/bar in revB, then svn info 'last change revision' will
return revA. As such when using AUTOREV, we are going to attempt to
retrieve root/a/b/c/foo/bar (as per SRC_URI) but at revA when it did
not exist.

So this patch changes how we retrieve the latest revision and uses
'svn log --limit 1' which gives correct result in all tested cases.

(Bitbake master rev: 17d8ef0b813a05c231e3dbe6e8bc82a4a9b1d2f8)

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agohob: change tooltip for live images
Cristiana Voicu [Mon, 7 Oct 2013 15:21:46 +0000 (18:21 +0300)]
hob: change tooltip for live images

This change is needed to inform the user that selecting
live type means that the system will build a hddimg and iso
image.

Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake: replace comment added in conf files
Cristiana Voicu [Mon, 7 Oct 2013 11:28:55 +0000 (14:28 +0300)]
bitbake: replace comment added in conf files

The patch that implements removeConfigurationVar method was made before
merging the patch that replaces "added by bitbake" with "added by hob".
This patch corrects this issue.

[YOCTO #5284]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake-dumpsig: introduce command line and error handling
Paul Eggleton [Fri, 4 Oct 2013 15:36:47 +0000 (15:36 +0000)]
bitbake-dumpsig: introduce command line and error handling

This utility doesn't take any special arguments, but it's nice if it at
least knows how to deal with no arguments, --help and errors properly.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake-diffsigs: improve error handling
Paul Eggleton [Fri, 4 Oct 2013 15:35:29 +0000 (15:35 +0000)]
bitbake-diffsigs: improve error handling

* Set up a logger independent of BitBake so we can log errors ourselves
* Handle common errors without printing a traceback

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake-diffsigs: refactor argument parsing slightly
Paul Eggleton [Fri, 4 Oct 2013 15:06:36 +0000 (15:06 +0000)]
bitbake-diffsigs: refactor argument parsing slightly

* Use OptionParser to parse the two options to -t rather than trying to
  pick them out ourselves.
* Add a description shown with --help output

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake-diffsigs: handle if task name is specified without do_ prefix
Paul Eggleton [Fri, 4 Oct 2013 11:13:46 +0000 (11:13 +0000)]
bitbake-diffsigs: handle if task name is specified without do_ prefix

Adjust the task name automatically if the -t option is specified with
a task name that doesn't start with do_ (e.g. "configure" instead of
"do_configure").

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobin/bitbake: Clarify server error message
Richard Purdie [Fri, 4 Oct 2013 13:24:50 +0000 (14:24 +0100)]
bin/bitbake: Clarify server error message

Getting an error message about --remote-server being set when really
BBSERVER was is confusing, clarify the message.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobin/bitbake: Clear ui-queue when starting a server
Richard Purdie [Fri, 4 Oct 2013 13:17:45 +0000 (14:17 +0100)]
bin/bitbake: Clear ui-queue when starting a server

When starting a server we don't want to show the debug event queue which
is reserved for when when errors have occurred. This patch copies the UI
code to ensure the user doesn't see confusing output.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobin/bitbake: No need to show tracebacks for BBHandledExceptions
Richard Purdie [Fri, 4 Oct 2013 13:16:15 +0000 (14:16 +0100)]
bin/bitbake: No need to show tracebacks for BBHandledExceptions

For BBHandledExceptions, we've already displaced a sensible error to
the user so we don't need to do it again. Just exit with an error
value.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoxmlrpc: Show a better error message for connection failures
Richard Purdie [Fri, 4 Oct 2013 13:15:36 +0000 (14:15 +0100)]
xmlrpc: Show a better error message for connection failures

Showing a traceback when unable to connect to a bitbake server is
rather ugly. This change allows us to show a sensible error message.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobin/bitbake: Catch establish connection log messages
Richard Purdie [Fri, 4 Oct 2013 13:14:52 +0000 (14:14 +0100)]
bin/bitbake: Catch establish connection log messages

If for example you try "bitbake -m" with an invalid BBSERVER, error
messages are not displayed. This change ensures logging is in place
to catch and display such errors.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake/hob: removing extra parameters from conf files using hob
Cristiana Voicu [Fri, 4 Oct 2013 13:19:45 +0000 (16:19 +0300)]
bitbake/hob: removing extra parameters from conf files using hob

In Hob settings, there is a tab to add/remove extra settings. This
patch implements a way to "remove" variables from conf files, through
bitbake. But, to keep the history assigment of the variables synchronized,
instead of removing, it replaces the lines with blank lines.

[YOCTO #5284]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoHOB: MACHINE should be saved in conf files using ?=
Valentin Popa [Fri, 27 Sep 2013 14:10:41 +0000 (17:10 +0300)]
HOB: MACHINE should be saved in conf files using ?=

MACHINE var is saved using early assignment operator.
Calling  MACHINE=x bitbake core-image-... works properly.
Comment "#added by bitbake" is replaced with "#added by hob".

[YOCTO #5070]
Signed-off-by: Valentin Popa <valentin.popa@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoutils: use logger.warn instead of print in copyfile
Ross Burton [Wed, 2 Oct 2013 16:47:28 +0000 (17:47 +0100)]
utils: use logger.warn instead of print in copyfile

print disappears into the ether, so use logger.warn and clean up the messages.

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agolib/bb/monitordisk.py: fix spelling in error text
Erik Bot [Wed, 2 Oct 2013 14:37:37 +0000 (16:37 +0200)]
lib/bb/monitordisk.py: fix spelling in error text

Signed-off-by: Erik Bot <erik.boto@pelagicore.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoHOB: Busy cursor when saving configuration
Valentin Popa [Tue, 10 Sep 2013 12:58:52 +0000 (15:58 +0300)]
HOB: Busy cursor when saving configuration

Display a busy cursor while the configuration is saved.

[YOCTO #4846]
Signed-off-by: Valentin Popa <valentin.popa@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake: Update to version 1.20.0
Richard Purdie [Tue, 24 Sep 2013 17:04:02 +0000 (18:04 +0100)]
bitbake: Update to version 1.20.0

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agotests/data: Whitespace in key names is a really bad idea
Richard Purdie [Sat, 21 Sep 2013 10:44:16 +0000 (10:44 +0000)]
tests/data: Whitespace in key names is a really bad idea

The parser never has supported it, the datastore API did happen to work
but whitespace in key names is a really bad idea and not something I think we
should encourage or support.

Fix test case failures after excplitly ignoring it for variable expansion
purposes.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata_smart: Variable references don't contain newlines, spaces or tabs
Richard Purdie [Fri, 20 Sep 2013 15:26:32 +0000 (15:26 +0000)]
data_smart: Variable references don't contain newlines, spaces or tabs

The code is happily trying to expand variable names containing newlines,
spaces and tabs which are illegal characters in variable names. This
patch stops it doing this. This will change dependency checksums
since some rather weird dependencies were being attempted to be expanded.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agohob: populate "Save image" dialog when saving changes to a custom image recipe
Cristiana Voicu [Tue, 17 Sep 2013 10:51:14 +0000 (13:51 +0300)]
hob: populate "Save image" dialog when saving changes to a custom image recipe

If my build starts from a YP standard image recipe, the 'Save image
recipe' dialog should be empty.

If my build starts from a custom image recipe, the 'Save image recipe'
dialog should populate the 'Name' and 'Description' fields with the values
set for the custom image recipe.

[YOCTO #5004]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobin/bitbake: Improve --help text
Richard Purdie [Wed, 18 Sep 2013 22:12:10 +0000 (22:12 +0000)]
bin/bitbake: Improve --help text

The --help text was rather inconsistent in style and plain incorrect in places,
using confusing terminology in others. I guess most people know what the options
do and don't read this but its confusing to new users.

This updates it to use the terms recipe and task consistently, remove
the references to stage, bbread and generally try and make the output
more useful.

[YOCTO #4856]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobuild: Add BB_TASK_NICE_LEVEL to task code
Richard Purdie [Fri, 20 Sep 2013 09:54:40 +0000 (09:54 +0000)]
build: Add BB_TASK_NICE_LEVEL to task code

On Linux its not possible for processes to regain a previous nice level after
it has changed. Its therefore not possible to have a core low priority and
then raise the priorities of individual tasks.

This variable allows us to do something like:

BB_TASK_NICE_LEVEL = "5"
BB_TASK_NICE_LEVEL_task-testimage = "0"

to give priority to specific tasks which the BB_NICE_LEVEL functionality
doesn't give us the option of.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake: runqueue: add task hash to Queue events
Alexandru DAMIAN [Wed, 18 Sep 2013 12:15:53 +0000 (13:15 +0100)]
bitbake: runqueue: add task hash to Queue events

Adding the sstate-related hash for all runqueue and
scenequeue tasks, as it's needed in the WebHob data.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake: event: adding generic event for metadata usage
Alexandru DAMIAN [Wed, 18 Sep 2013 12:15:52 +0000 (13:15 +0100)]
bitbake: event: adding generic event for metadata usage

Adding the generic bb.event.MetadataEvent that is
targeted specifically at metadata usage. This is
needed in order to let the metadata code send and receive
events during asynchrous execution without having
to define each event specifically in Bitbake.

Metadata code should subscribe to and fire the MetadataEvent
in order to communicate asynchronously, and identify
the object using event.type field, and parse the
data in the event.data field.

Knotty UI will ignore these event by default.

This deprecates RequestPackageInfo/PackageInfo, and that
event pair will be removed in the future.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake: cooker, command: add a command to return global data
Alexandru DAMIAN [Wed, 18 Sep 2013 12:15:51 +0000 (13:15 +0100)]
bitbake: cooker, command: add a command to return global data

Adding the 'getAllKeysWithFlags' read-only command that will
return a dump of the global data state, together with specified
flags for each key. The flag list is passed in as the first
parameter to the command.

This will be used by UI clients to get the build configuration.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake: cooker: get extra information from recipe cache
Alexandru DAMIAN [Wed, 18 Sep 2013 12:15:50 +0000 (13:15 +0100)]
bitbake: cooker: get extra information from recipe cache

The loaded cache modules may add extra attributes to
the recipecache, that will be populated by the cache
classes required by the UI. These attributes
will be used by the UI to display relevant information.

Adds cachefields cache class field to specify
for each cache class which attributes will be set
in the recipecache.

Adds code to automatically expand depends tree with the
fields exported by the extra cache class.

Fixes a cache field name in the HOB UI.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake: build, runqueue: adds info to the *runQueue* events
Alexandru DAMIAN [Wed, 18 Sep 2013 12:15:49 +0000 (13:15 +0100)]
bitbake: build, runqueue: adds info to the *runQueue* events

This patch adds task identifying information for all
runQueue and sceneQueue events, and for bb.build.Task* events.
This will allow matching event to specific tasks in the UI
handlers processing these events.

Adds RunQueueData functions to get the task name and task
file for usage with the runQueue* events.

Adds taskfile and taskname properties to bb.build.TaskBase.

Adds taskfile and taskname properties to the *runQueue* events

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake: cooker,runqueue: send the task dependency tree
Alexandru DAMIAN [Wed, 18 Sep 2013 12:15:48 +0000 (13:15 +0100)]
bitbake: cooker,runqueue: send the task dependency tree

Adding a CookerFeature that allows UIs to enable
receving a dependency tree once the task data has been
computed and the runQueue is ready to start.

This will allow the clients to display dependency
data in an efficient manner, and not recompute the runqueue
specifically to get the dependency data.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake: cooker,xmlrpc,servers: implement CookerFeatures
Alexandru DAMIAN [Wed, 18 Sep 2013 12:15:47 +0000 (13:15 +0100)]
bitbake: cooker,xmlrpc,servers: implement CookerFeatures

Implementing feature set selection that allows a client
to enable specific features in the server at connection time.

Only enabling of features is supported, as there is
no way to safely remove data loaded into the cooker.
Once enabled, a feature will remain enabled for the
life of the cooker.

Client-server connection now supports specifying the feature
set required by the client. This is implemented in the Process
server using a managed proxy list, so the server cooker
will now load dynamically needed features based on what client
connects to it.

In the XMLRPC server the feature set is requested by
using a parameter for registerUIHandler function.
This allows observer-only clients to also specify features
for the server.

The server code configuration now is completly separated
from the client code. All hardcoding of client knowledge is
removed from the server.

The extra_caches is removed as the client can now specify
the caches it needs using the feature. The UI modules
now need to specify the desired featureSet. HOB is modified
to conform to the featureSet specification.

The only feature available is CookerFeatures.HOB_EXTRA_CACHES
which forces loading the bb.cache_extra:HobRecipeInfo class.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata_smart: Fix variable reference issues
Richard Purdie [Wed, 18 Sep 2013 12:23:54 +0000 (12:23 +0000)]
data_smart: Fix variable reference issues

The change to use the expansion cache in VariableParse was incorrect as
it was adding in references it shouldn't have been. This patch corrects
the codepaths and ensures the references are correct.

The cache version is bumped since the previous bug could have leave
to invalid checksum calculations and a clean cache is therefore desireable.

The impact of the bug was that sstate was not getting reused when it should
and some tasks were also being rerun when they should not have been.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agocooker: return a copy checkPackages
Richard Purdie [Wed, 18 Sep 2013 07:38:44 +0000 (07:38 +0000)]
cooker: return a copy checkPackages

The syntax used for checkPackages implies a copy is returned but it
did not do so. Make it so.

This is fixes universe builds where error messaages were being shown
but there should have only been warnings.

[YOCTO #5222]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agousermanual.xml: Two new sections added to BitBake "Description"
Scott Rifenbark [Tue, 17 Sep 2013 21:02:18 +0000 (14:02 -0700)]
usermanual.xml: Two new sections added to BitBake "Description"

1. Added a new section "Appending and Prepending (override style
   syntax)".  This section shows how the append and prepend operators
   work using the override style syntax.

2. Added a new section "Removing (override style syntax)".  This
   section describes the new "_remove" operator.

Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoRevert "bb.fatal: Raise a BBHandledException instead of exiting"
Richard Purdie [Tue, 17 Sep 2013 18:15:26 +0000 (19:15 +0100)]
Revert "bb.fatal: Raise a BBHandledException instead of exiting"

Sanity test failures are no longer fatal with this change so whilst
its the right idea, the code paths need more work.

This reverts commit a50017ba71250e1710a6425b60ac7e3f03d88295.

6 years agobb.fatal: Raise a BBHandledException instead of exiting
Richard Purdie [Sat, 14 Sep 2013 16:59:24 +0000 (16:59 +0000)]
bb.fatal: Raise a BBHandledException instead of exiting

With new bitbake UIs having the cooker exit at 'random' points
in the codebase is problematic. This patch raises an exception
which matches the siutation instead.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata: Optimise flag exclusion list handling
Richard Purdie [Mon, 16 Sep 2013 07:54:58 +0000 (07:54 +0000)]
data: Optimise flag exclusion list handling

Move the variable lookup to the outer loop for performance, replacing
a now unneeded parameter (after the previous changes).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata: Optimise flag lookup in build_dependencies
Richard Purdie [Mon, 16 Sep 2013 07:53:40 +0000 (07:53 +0000)]
data: Optimise flag lookup in build_dependencies

When looking up flag variable dependencies, large chunks of the function
aren't needed. Optimise the function flow accordingly for speed.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata: Optimise build_dependencies a little
Richard Purdie [Mon, 16 Sep 2013 07:51:18 +0000 (07:51 +0000)]
data: Optimise build_dependencies a little

Instead of multiple calls to getVarFlag, make one call to getVarFlags, only expanding
the flags that need to be expanded. This improves performance.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata_smart: Cache the fact a variable accesses another even if its unset
Richard Purdie [Mon, 16 Sep 2013 07:45:50 +0000 (07:45 +0000)]
data_smart: Cache the fact a variable accesses another even if its unset

If a variable references another but it isn't set at present, the
reference wasn't stored. It really should be marked as a reference
and the higher level dependency code can handle as appropriate.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata_smart: Allow flags to use the expand cache
Richard Purdie [Mon, 16 Sep 2013 07:45:20 +0000 (07:45 +0000)]
data_smart: Allow flags to use the expand cache

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata_smart: Allow expansion of flags in getVarFlags
Richard Purdie [Mon, 16 Sep 2013 07:43:30 +0000 (07:43 +0000)]
data_smart: Allow expansion of flags in getVarFlags

Allow a list of flags to expand to be passed into getVarFlags. This
is useful within bitbake itself to optimise performance of the
dependency generation code.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata_smart: Add explict None checks
Alexandru DAMIAN [Mon, 16 Sep 2013 07:40:20 +0000 (07:40 +0000)]
data_smart: Add explict None checks

Simple if xxx checks end up calling len(xxx). We're interested in the specific case
of None which means we can break out the iterator much earlier after the first
item. This adds in the specific tests for None in what is a hot path in the
data store code which gives small performance gains.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata: Cache an list of export variables
Richard Purdie [Mon, 16 Sep 2013 07:36:42 +0000 (07:36 +0000)]
data: Cache an list of export variables

Compute a cache of the list of potential export variables so
that we don't have to compute the list from scratch.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata: Use direct iteration, not keys()
Richard Purdie [Mon, 16 Sep 2013 07:33:48 +0000 (07:33 +0000)]
data: Use direct iteration, not keys()

Profiling shows the creation of keys() has overhead and we're better using
an iterator rather than the memory associated with the huge list of keys
when iterating the whoe datastore. We minimise the number of times
we do this to twice only per recipe.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata_smart: use the expand_cache in VariableParse
Richard Purdie [Mon, 16 Sep 2013 07:25:08 +0000 (07:25 +0000)]
data_smart: use the expand_cache in VariableParse

When in VariableParse, use the expand_cache if possible rather than looking
up data. Ultimately it would come from the same place but this short cuts
a heavily used code block for speed improvements.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata_smart: Improve variable expansion regexp
Richard Purdie [Mon, 16 Sep 2013 07:21:32 +0000 (07:21 +0000)]
data_smart: Improve variable expansion regexp

Debugging showed the variable expansion regexp was catching python
expressions (starting with @). Since these are caught by their own
dedicated regexp, stop matching these for the plain variable expansion
for small performance improvements.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agosiggen: Use lookup cache exclusively
Richard Purdie [Mon, 16 Sep 2013 07:18:34 +0000 (07:18 +0000)]
siggen: Use lookup cache exclusively

All the values we need are already guaranteed to be in the lookupcache
so rather than fetch variables again, just use the cache. This gives a
small performance improvement and simplifies the code.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata: Be explicit in data_db check
Richard Purdie [Mon, 16 Sep 2013 07:13:38 +0000 (07:13 +0000)]
data: Be explicit in data_db check

The if statement current causes the size of parent to be calcuated which
is like a len() operation on a datastore. Since we're only interested
whether the value is none, checking explictly for this gives a
small performance gain.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agocooker: Allow profiling of the parser in profile mode
Richard Purdie [Sun, 15 Sep 2013 08:57:02 +0000 (08:57 +0000)]
cooker: Allow profiling of the parser in profile mode

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agocooker: Avoid duplication for taskdata creation
Alexandru DAMIAN [Mon, 16 Sep 2013 22:50:56 +0000 (22:50 +0000)]
cooker: Avoid duplication for taskdata creation

Clean-up to avoid duplication and promote code reuse to factor
taskdata creation into a common function.

[RP: minor tweaks]

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agorunqueue: add runQueueTaskSkipped event
Alexandru DAMIAN [Mon, 16 Sep 2013 12:46:01 +0000 (12:46 +0000)]
runqueue: add runQueueTaskSkipped event

Adding a runQueueTaskSkipped to notify that the tasks that are not
run either because they are set-scened or they don't need an update
(timestamp was ok).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobuild: Add logfile to add TaskBase events
Richard Purdie [Sat, 14 Sep 2013 16:54:09 +0000 (17:54 +0100)]
build: Add logfile to add TaskBase events

We add the path to the logfile for all Task events except TaskInvalid
so that we can trace back the logfile locations at some future point.

TaskInvalid doesn't ever have a logfile.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agocooker: updateCache should rause exceptions, not sys.exit
Richard Purdie [Fri, 13 Sep 2013 16:34:42 +0000 (17:34 +0100)]
cooker: updateCache should rause exceptions, not sys.exit

Exiting from the server is antisocial, instead we should raise an exception. This
will correctly fail the current command and reset the server state. We use
the handled exception since for these conditions to occur, something was
already displayed to the user.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoknotty: Cleanup error/interruption handling
Richard Purdie [Fri, 13 Sep 2013 16:34:06 +0000 (17:34 +0100)]
knotty: Cleanup error/interruption handling

Only display a CommandFailed ERROR: message if there is an error to display.

Only display an errors summary if we actually displayed errors.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agocommand: Treat empty messages as failures, not CommandCompleted
Richard Purdie [Fri, 13 Sep 2013 16:33:30 +0000 (17:33 +0100)]
command: Treat empty messages as failures, not CommandCompleted

Empty messages should trigger CommandFailed, not CommandCompleted as
otherwise the exit code will be incorrect.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agocooker/command: Add finishcommand to reset cooker state
Richard Purdie [Fri, 13 Sep 2013 16:32:53 +0000 (17:32 +0100)]
cooker/command: Add finishcommand to reset cooker state

After running a command on the server, it needs to reset to the initial
state. This ensures that subsequent clients start from a known state
and notice any configuration changes.

Ultimately we may want to do more than this buts a good start and better
than nothing.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agocooker: Rename confusing 'stop' state to 'forceshutdown'
Richard Purdie [Fri, 13 Sep 2013 16:31:54 +0000 (17:31 +0100)]
cooker: Rename confusing 'stop' state to 'forceshutdown'

The shutdown state causes the server to finish what its doing, stop was
them meant to completely stop it. It doesn't mean the server is stopped
though. Renaming the current stop event for forceshutdown gives more
meaning to what it actually does. The stopped namespace then becomes
available to indicate a completely stopped server.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agocooker: Clean up init/reset configuration code
Richard Purdie [Fri, 13 Sep 2013 16:31:09 +0000 (17:31 +0100)]
cooker: Clean up init/reset configuration code

Currently the cooker event data isn't rebuilt upon reset and the cache
configuration cannot be changed after init. These are both bad things
and this patch refactors the init/reset code so that it is possible
to reconfigure the server.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agocooker: Drop obsolete worker test
Richard Purdie [Fri, 13 Sep 2013 16:30:18 +0000 (17:30 +0100)]
cooker: Drop obsolete worker test

This call only ever happens in cooker context now so we can drop the
nasty worker check from here.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake: xmlrpc: delete unused code BitBakeUIEventServer
Alexandru DAMIAN [Mon, 9 Sep 2013 16:41:03 +0000 (17:41 +0100)]
bitbake: xmlrpc: delete unused code BitBakeUIEventServer

BitBakeUIEventServer is an unused class that pushes UI
events over a separate thread.

The current version of XMLRPC server works just fine with
the classic UI event handlers, so this class is not needed.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata_smart: Account for changes in append/prepend/remove in the config hash
Richard Purdie [Thu, 12 Sep 2013 13:06:22 +0000 (13:06 +0000)]
data_smart: Account for changes in append/prepend/remove in the config hash

bitbake wasn't reparsing when _remove items were added to its configuration
and equally, appends/prepends were also being badly tracked. This
change enrures these variables are accounted for in the configuration
hash.

[YOCTO #5172]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoutils: Fix BB_PRESERVE_ENV
Richard Purdie [Wed, 11 Sep 2013 10:01:47 +0000 (11:01 +0100)]
utils: Fix BB_PRESERVE_ENV

BB_PRESERVE_ENV wasn't working since data.inheritFromOS wasn't getting a
correct list of keys to import into the data store. This fixes
things so it does add all environment variables into the data store
when BB_PRESERVE_ENV is used.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake: runqueue: add sceneQueueTaskCompleted event
Alexandru DAMIAN [Mon, 9 Sep 2013 16:40:56 +0000 (17:40 +0100)]
bitbake: runqueue: add sceneQueueTaskCompleted event

Adding an event to be fired when a scene task is completed.
It is analogous to the run task completed event, and has
been missing for some reason.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agorunqueue.py: check whether multiple versions of the same PN are due to be built
Robert Yang [Mon, 2 Sep 2013 13:47:21 +0000 (09:47 -0400)]
runqueue.py: check whether multiple versions of the same PN are due to be built

There would be an race issue if we:

$ bitbake make-3.81 make-3.82

This because they are being built at the same time which would cause
unexpected problems, for example:

[snip]
ERROR: Package already staged (/path/to/tmp/sstate-control/manifest-qemux86-make.populate-sysroot)?!
ERROR: Function failed: sstate_task_postfunc
[snip]

Or there would be python's strack trace such as:

[snip]
 *** 0004:    mfile = open(manifest)
     0005:    entries = mfile.readlines()
     0006:    mfile.close()
     0007:
     0008:    for entry in entries:
Exception: IOError: [Errno 2] No such file or directory: xxx
[snip]

[YOCTO #5094]

We can quit earlier to avoid this kind of issue when two versions of the same PN
are going to be built since this isn't supported.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoproviders.py: enhance the runtime debug degbug messgae
Robert Yang [Mon, 2 Sep 2013 13:41:19 +0000 (09:41 -0400)]
providers.py: enhance the runtime debug degbug messgae

The runtime provider debug message is the same as the build time debug
message, make them different would be better.

[YOCTO #5067]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoprserv: Ensure data is committed
Richard Purdie [Sun, 8 Sep 2013 09:45:54 +0000 (09:45 +0000)]
prserv: Ensure data is committed

In exclusive mode, we need to complete the transaction for writes to make
it to the database. Therefore add sync calls to ensure this happens.

Autocommit mode is significantly (100 times) slower so caching the
data is of significant benefit.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agocontrib/vim: merge changes from vim-bitbake repo
Christopher Larson [Fri, 6 Sep 2013 19:25:32 +0000 (12:25 -0700)]
contrib/vim: merge changes from vim-bitbake repo

commit 00ca441614695b4261d8d4f31b7ef0e3e3784282
Merge: 8cc367d bb88c0f
Author: Christopher Larson <kergoth@gmail.com>
Date:   Thu Aug 22 16:42:42 2013 -0700

    Merge pull request #6 from staticshock/multi-line-strings

    Remove "keepend" and "excludenl" directives

commit bb88c0fd4ad2b7b9c8c4c73def2b3cb20c473ac3
Author: Anton Backer <olegov@gmail.com>
Date:   Sat Jul 13 01:24:15 2013 -0400

    Remove "keepend" and "excludenl" directives

    It looks like these were never actually used correctly, and were doing
    more harm than good. "keepend" on bbString, for instance, prevented
    proper nesting of ${@python} in strings. Similarly, a balanced pair of
    { } braces inside a shell function would force the function to terminate
    early if the closing brace was on its own line.

    So far I've seen absolutely no negative consequences from removing
    these, but a bunch of positive consequences.

    Fixes #1

commit 8cc367d01f4c699be5fcc072de59e6f2f14a138b
Merge: c58628c eec6b7f
Author: Christopher Larson <kergoth@gmail.com>
Date:   Thu Aug 22 09:46:46 2013 -0700

    Merge pull request #4 from staticshock/function-names

    Parse function names with nested vars

commit c58628ca517cd25985361fc0d27863521cc28a5d
Merge: dfb0f7c a890982
Author: Christopher Larson <kergoth@gmail.com>
Date:   Thu Aug 22 09:43:40 2013 -0700

    Merge pull request #5 from yoyko/master

    syntax: python expansion (${@...}) inside shell functions

commit a890982b7c33a6e363b12d6cb69e22b4bbc0f317
Author: Jozef Šiška <yoyo@ksp.sk>
Date:   Thu Aug 22 13:20:45 2013 +0200

    syntax: python expansion (${@...}) inside shell functions

Signed-off-by: Jozef Šiška <jsiska@nuvotechnologies.com>
commit eec6b7f6f0472787929f424968f9a0d78ac4af08
Author: Anton Backer <olegov@gmail.com>
Date:   Fri Jul 12 22:16:01 2013 -0400

    Parse function names with nested vars

    For instance, pkg_postinst_${PN}

    Fixes #3

commit dfb0f7c0d51556448cba79b474b8c19b9cded9af
Author: Christopher Larson <chris_larson@mentor.com>
Date:   Fri Jun 1 18:57:13 2012 -0400

    syntax: add ?= flag def

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
commit 589a62a00709ca822a42327e7086008aba2d9933
Author: Christopher Larson <kergoth@gmail.com>
Date:   Fri Dec 9 22:25:47 2011 -0700

    ftplugin: set commentstring

Signed-off-by: Christopher Larson <kergoth@gmail.com>
commit 7ffc80b3fb4ddf68cc5a69bdc63ab03d70c44f87
Author: Chris Larson <chris_larson@mentor.com>
Date:   Thu Jun 2 15:27:48 2011 -0700

    Handle +=/=+ for flags

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoknotty: Reduce refresh of footer
Richard Purdie [Mon, 2 Sep 2013 21:48:29 +0000 (21:48 +0000)]
knotty: Reduce refresh of footer

When displaying larger number of events the client can get caught up in displaying
the footer, then immediately overwriting it. To avoid this, wait for pauses
in the event stream before displaying the footer to give a slightly more
friendly feel to the UI.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake: Ensure ${DATE} and ${TIME} are consistent
Peter Kjellerstedt [Fri, 6 Sep 2013 15:53:20 +0000 (17:53 +0200)]
bitbake: Ensure ${DATE} and ${TIME} are consistent

Due to the worker split the ${DATE} and ${TIME} variables could end up
with different values for different workers.

E.g., a task like do_rootfs that is run within a fakeroot environment
had a slightly different view of the time than another task that was not
fakerooted which made it impossible to correctly refer to the image
generated by do_rootfs from the other task.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agohob: remove custom image from the images list when not needed
Cristiana Voicu [Wed, 4 Sep 2013 15:45:07 +0000 (18:45 +0300)]
hob: remove custom image from the images list when not needed

Selected custom image recipes should be cleared from the combo the moment
you change your selection. The idea is to always perform the selection of
those images in the same way (i.e through the "Select from my image
recipes" option).

[YOCTO #5001]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agohob: limit the description size when a custom image is saved
Cristiana Voicu [Thu, 5 Sep 2013 12:47:52 +0000 (15:47 +0300)]
hob: limit the description size when a custom image is saved

[YOCTO #5003]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agonewbb.vim: Use 'git config' instead of git-config
Khem Raj [Thu, 5 Sep 2013 21:09:20 +0000 (14:09 -0700)]
newbb.vim: Use 'git config' instead of git-config

Newer versions of git do not have the '-' concatenated command

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agodata_smart: Move getVar expand cache handing to fix _remove operations
Richard Purdie [Fri, 6 Sep 2013 20:14:25 +0000 (20:14 +0000)]
data_smart: Move getVar expand cache handing to fix _remove operations

DISTRO_FEATURES_remove = "opengl" wasn't working as expected. The reason
turned out the be the indirect reference to opengl and the fact _remove was
operating on unexpanded data.

This patch rearranges some code to ensure we operate on expanded data
by moving the expand cache handing into getVarFlags instead of getVar.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agohob: remove PACKAGE_INSTALL variable setting from hob
Cristiana Voicu [Tue, 3 Sep 2013 07:28:01 +0000 (10:28 +0300)]
hob: remove PACKAGE_INSTALL variable setting from hob

Before saving the packages for a custom images in a .bb file,
the packages were saved in bitbake memory. Now all the variables
are saved in conf file, so saving PACKAGE_INSTALL is not needed anymore.
Moved were LINGUAS_INSTALL is set, because both conditions are for testing
if a custom image is saved.

[YOCTO #5101]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake-worker: ensure BUILDNAME is available during execution
Paul Eggleton [Mon, 2 Sep 2013 17:26:28 +0000 (18:26 +0100)]
bitbake-worker: ensure BUILDNAME is available during execution

BUILDNAME is set from cooker by default, so since the worker split it
will not be set when executing functions. In OpenEmbedded this results
in /etc/version (which is populated from BUILDNAME) not having any
content. Pass this variable value through to the worker explicitly to
fix the issue.

Fixes [YOCTO #4818].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobitbake/cooker: fix some calls of cookerdata.findConfigFile method
Cristiana Voicu [Mon, 2 Sep 2013 11:18:25 +0000 (14:18 +0300)]
bitbake/cooker: fix some calls of cookerdata.findConfigFile method

Cookerdata.findconfigFile method has a new parameter. Changed some calls.

Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agorunqueue: Fix scenequeue to pass file descriptors, not a float
Richard Purdie [Mon, 2 Sep 2013 08:15:19 +0000 (08:15 +0000)]
runqueue: Fix scenequeue to pass file descriptors, not a float

This was missed off in a previous patch.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoserver/xmlrpc: stop server on client exit
Alexandru DAMIAN [Sat, 31 Aug 2013 22:46:06 +0000 (23:46 +0100)]
server/xmlrpc: stop server on client exit

If the server only expects a single client to connect (i.e.
no bind parameter set, so there is no way for the
clients to get the server port), stop the server after
the first client exits.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoprserv/serv: Settle on two threads for optimal performance
Richard Purdie [Sat, 31 Aug 2013 22:44:42 +0000 (23:44 +0100)]
prserv/serv: Settle on two threads for optimal performance

Using the threading mixin class resulted in large amounts of memory
being used by the PR server for no good reason. Using a receiver thread
and a thread to do the actual database operations on a single connection
gives the same performance with a much saner memory overhead so
switch to this.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agobuild: Fix profile file names
Richard Purdie [Sat, 31 Aug 2013 22:43:37 +0000 (23:43 +0100)]
build: Fix profile file names

Using the basename of the .bb file is not unique, for example xxx-native
and xxx can overwrite each other. If this happens whilst running, you can
get odd backtraces as one file is parsed as another tries to write out
new data.

Avoid issues by using PN for the output filename instead.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoserv/db: Don't use BEGIN/COMMIT
Richard Purdie [Sat, 31 Aug 2013 22:42:13 +0000 (23:42 +0100)]
serv/db: Don't use BEGIN/COMMIT

Since we don't support using multiple servers on the same database file,
don't use the BEGIN/COMMIT syntax and allow writes to the database
to work ~100 times faster with no transaction locking.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoserv/db: Take an excluside lock on the database
Richard Purdie [Sat, 31 Aug 2013 22:42:57 +0000 (23:42 +0100)]
serv/db: Take an excluside lock on the database

We only support one server using the database at a time so take an exclusive
lock and avoid later lock overhead.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoserv/db: Fix looping upon database locked issues
Richard Purdie [Sat, 31 Aug 2013 22:41:35 +0000 (23:41 +0100)]
serv/db: Fix looping upon database locked issues

If the database is locked we will get an immediate error indicating so,
there is no retry timeout. The looping code is therefore useless, the loop
count is near instantly exceeded.

Using a time based retry means we can wait a sensible time, then gracefully
exit.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoserver/process, server/xmlrpc, runqueue: Use select.select() on fds, not time.sleep()
Richard Purdie [Sat, 31 Aug 2013 22:40:55 +0000 (23:40 +0100)]
server/process, server/xmlrpc, runqueue: Use select.select() on fds, not time.sleep()

The existing backend server implementations were inefficient since they
were sleeping for the full length of the timeouts rather than being woken when
there was data ready for them. It was assumed they would wake and perhaps did
when we forked processes directory but that is no longer the case.

This updates both the process and xmlrpc backends to wait using select(). This
does mean we need to pass the file descriptors to wait on from the internals
who know which these file descriptors are but this is a logical improvement.

Tests of a pathaolgical load on the process server of ~420 rapid tasks
executed on a server with BB_NUMBER_THREAD=48  went from a wall clock
measurement of the overall command execution time of 75s to a much more
reasonable 24s.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoprserv: Allow 'table is locked' matching for retry loop
Richard Purdie [Fri, 30 Aug 2013 16:52:20 +0000 (17:52 +0100)]
prserv: Allow 'table is locked' matching for retry loop

Try and avoid errors like "ERROR: database table is locked: PRMAIN_nohist"
by retrying if we see the string "is locked".

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agoserver/xmlrpc: Increase timeout to 60s
Richard Purdie [Fri, 30 Aug 2013 16:41:16 +0000 (17:41 +0100)]
server/xmlrpc: Increase timeout to 60s

This is a better value that the earlier infinite timeout yet still
allows for servers with high loads. It does mean the bitbake process
can hang at exit for the timeout period but that should never happen
and only happened for me in some test cases which wouldn't happen
in normal use.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
6 years agocookerdata: Allow bblayers.conf to be found using BBPATH
Richard Purdie [Thu, 29 Aug 2013 13:27:29 +0000 (14:27 +0100)]
cookerdata: Allow bblayers.conf to be found using BBPATH

It should be possible to run a build anywhere on the filesystem and have
bitbake find the correct build directory if its set somehow. The BBPATH
variable makes perfect sense for this usage. Therefore use any available
value of BBPATH to search for conf/bblayers.conf before walking the parent
directory structure.

This restores the option of being able to run bitbake from anywhere if
the user has set things up to operate in that environment.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>