The order of properties in 'device-list-properties' can hange
arbitrarily and git is not great at picking the contexts in JSON to help
seeing what changed.
The new --dump-device-list-properties produces a stable order of
properties and dumps also the type and default value mainly useful for
comparing two .replies files.
Example output:
$ ./scripts/qemu-replies-tool.py tests/qemucapabilitiesdata/caps_9.0.0_x86_64.replies --dump-device-list-properties
(dev) ICH9-LPC acpi-index uint32 (0)
(dev) ICH9-LPC acpi-pci-hotplug-with-bridge-support bool
(dev) ICH9-LPC acpi_disable_cmd uint8
(dev) ICH9-LPC acpi_enable_cmd uint8
(dev) ICH9-LPC addr int32 (-1)
(dev) ICH9-LPC cpu-hotplug-legacy bool
(dev) ICH9-LPC disable_s3 uint8
(dev) ICH9-LPC disable_s4 uint8
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
The order of entries in 'qom-list-types' sometimes changes arbitrarily.
The --dump-qom-list-types produces a stable order and drops the for
libvirt unneeded 'parent' information.
Sample output:
$ ./scripts/qemu-replies-tool.py tests/qemucapabilitiesdata/caps_9.0.0_x86_64.replies --dump-qom-list-types
(qom) 486-v1-x86_64-cpu
(qom) 486-x86_64-cpu
(qom) AC97
(qom) AMDVI-PCI
(qom) Broadwell-IBRS-x86_64-cpu
(qom) Broadwell-noTSX-IBRS-x86_64-cpu
(qom) Broadwell-noTSX-x86_64-cpu
(qom) Broadwell-v1-x86_64-cpu
(qom) Broadwell-v2-x86_64-cpu
(qom) Broadwell-v3-x86_64-cpu
[...]
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Make the tool useful also for non-testing purposes by adding 'dump'
mode, which will process the data and output information about the qemu
version.
The first 'dump' mode produces all possible valid query strings per
virQEMUQAPISchemaPathGet/virQEMUCapsQMPSchemaQueries. This is useful for
users to look up a query string via 'grep' rather than trying to come up
with it manually.
Additionally the data as represented by qemu changes naming very often
and that makes it un-reviewable to find changes between two qemu builds.
By using the dump mode, which produces results in stable order we can
use it to 'diff' two .replies file without churn.
Sample output '[...]' denotes an arbitrary trim:
$ ./scripts/qemu-replies-tool.py tests/qemucapabilitiesdata/caps_9.0.0_x86_64.replies --dump-qmp-query-strings
[...]
(qmp) blockdev-add
(qmp) blockdev-add/arg-type/auto-read-only
(qmp) blockdev-add/arg-type/auto-read-only/!bool
(qmp) blockdev-add/arg-type/cache
(qmp) blockdev-add/arg-type/cache/direct
(qmp) blockdev-add/arg-type/cache/direct/!bool
(qmp) blockdev-add/arg-type/cache/no-flush
(qmp) blockdev-add/arg-type/cache/no-flush/!bool
(qmp) blockdev-add/arg-type/detect-zeroes
(qmp) blockdev-add/arg-type/detect-zeroes/^off
(qmp) blockdev-add/arg-type/detect-zeroes/^on
(qmp) blockdev-add/arg-type/detect-zeroes/^unmap
[...]
(qmp) blockdev-add/arg-type/driver
(qmp) blockdev-add/arg-type/driver/^blkdebug
(qmp) blockdev-add/arg-type/driver/^blklogwrites
(qmp) blockdev-add/arg-type/driver/^blkreplay
(qmp) blockdev-add/arg-type/driver/^blkverify
(qmp) blockdev-add/arg-type/driver/^bochs
(qmp) blockdev-add/arg-type/driver/^cloop
[...]
(qmp) blockdev-add/arg-type/+blkdebug
(qmp) blockdev-add/arg-type/+blkdebug/align
(qmp) blockdev-add/arg-type/+blkdebug/align/!int
(qmp) blockdev-add/arg-type/+blkdebug/config
(qmp) blockdev-add/arg-type/+blkdebug/config/!str
(qmp) blockdev-add/arg-type/+blkdebug/image
(qmp) blockdev-add/arg-type/+blkdebug/image (recursion)
(qmp) blockdev-add/arg-type/+blkdebug/image/!str
(qmp) blockdev-add/arg-type/+blkdebug/inject-error
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
If the schema itself is extended in qemu we need to have a notification
to add appropriate handling to ensure that we have full coverage of all
fields.
Add validation that only fields that libvirt currently knows about are
present in the schema.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
The tool in the current shape functionally replaces
tests/qemucapabilitiesnumbering.c
It validates that the output '.replies' files conform to how we generate
them from qemu and also allows programmatic modification of the
'.replies' files if re-generation is not feasible any more.
The main advantage is that JSON objects are parsed into native python
types and thus the programatic modification is much more convenient.
The tool will be later extended to also do validation that we properly
handle the whole of QMP schema as well as help in reviewing the
differences in the .replies file after qemu updates.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
The script expects each of the symbols that it looks for to
be in one of three sections, which in nm(1) are described as
follows:
T - The symbol is in the text (code) section.
B - The symbol is in the BSS data section. This section
typically contains zero-initialized or uninitialized
data, although the exact behavior is system dependent.
D - The symbol is in the initialized data section.
When building on alpha, however, some of the symbols show up
in one of two additional sections, specifically:
S - The symbol is in an uninitialized or zero-initialized
data section for small objects.
G - The symbol is in an initialized data section for small
objects.
In other words, S is the same as B and G is the same as D,
except with some optimization for small objects that for some
reason is applied on alpha but not on other architectures.
I have confirmed that, for all the symbols that the script
complained about being missing on alpha, the section is the
expected one, that is, symbols that are reported as B on x86
are reported as S on alpha, and symbols that are reported as
D on x86 are reported as G on alpha.
Note that, while the B section doesn't seem to be used at all
on alpha, at least in our case, the D section still is.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
The generator can produce different code on Linux and macOS:
specifically, on the former we want to use xdr_uint64_t while
the latter needs xdr_u_int64_t instead.
This is clearly a problem for tests that involve comparing the
output produced against some expected output that's stored in
the git repository.
In the long run, we need to find a better way to handle this,
but since 9.10.0 is going to be released very soon and we don't
want it to have a broken test suite on macOS, simply skip the
generator tests on that platform for now.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
macOS XDR library is an oddball using xdr_u_int64_t instead of
xdr_uint64_t which everyone else has.
The code generator already does the right thing, but the test
program previously generated with the Linux rpcgen program
does not compile on macOS due to this.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
The test_demo program currently fails to compile on macOS with
too few arguments to function call, expected 3, have 2
ret = !!proc(&xdr, vorig);
~~~~ ^
Way back in 2013, commit 9fa3a8ab6f handled this situation
for the main library code. Apply the same fix here.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
These are currently the only tests that are not part of any
suite.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Some of the files used by test_demo.c can only be regenerated
when pytest is present, but we have pre-generated copies in the
repository, so overall we just need the C compiler to build and
run that specific test program.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
There are some platforms where 'char' is unsigned, by default
(RPi, s390x to name a few). And because of how test_demo is
written we are experiencing some test cases failing there. For
instance: /xdr/struct-scalar is failing. This is because in the
test (test_struct_scalar()), we have a struct with two chars. One
is initialized to 0xca, the other 0xfe (note that both have the
MSB set). The XDR encoder (xdr_TestStructScalar()) then calls
xdr_char() on both of them. But XDR itself has no notion of
char type, so under the hood, it expands it to int [1] and calls
xdr_int(). And this is where the problem lies. On platforms where
char is signed, the integer expansion results in 0xffffffca, but
on platforms where char is unsigned it results in 0x000000ca. Two
distinct results.
The test then goes and compares the encoded buffer with an
expected one (memcmp(), read from the disk earlier).
This poses no problem for real life use, because when decoding
those chars back, the padding is thrown away.
To avoid tickling this issue, use values that don't have the MSB
set.
1: https://git.linux-nfs.org/?p=steved/libtirpc.git;a=blob;f=src/xdr.c;h=28d1382cc4853ecf1238d792af5016160435d1e0;hb=HEAD#l487
Fixes: 40cbaa8fbe rpcgen: add test case for XDR serialization
Reported-by: Boris Fiuczynski <fiuczy@linux.ibm.com>
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Boris Fiuczynski <fiuczy@linux.ibm.com>
The test_demo program compares whether XDR encoded data match the
expected output as read from a file. But the file path is not
absolute and thus relative to CWD which means the program can run
only from one specific directory.
Do what we do in the rest of our test suite: define 'abs_srcdir'
macro and prefix the path with it.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Jiri Denemark <jdenemar@redhat.com>
flake8 (run on all python scripts as a part of the syntax checks)
version 6.1.0 (on macOS 14) issued many complaints like this on the
new rpcgen python scripts:
[...]libvirt/scripts/rpcgen/rpcgen/lexer.py:57:17: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
This patch changes all [type] == [type] to use "is" instead of "==",
and similarly to use "is not" instead of "!=".
(flake8 5.03, e.g. on Fedora 38, is just fine with using "==" and "!=",
but python on both likes "is" and "is not")
Fixes: commit v9.9.0-24-g8ec79e5e14
Fixes: commit v9.9.0-22-gca3f025011
Fixes: commit v9.9.0-21-g031efb691f
Fixes: commit v9.9.0-20-g8c8b97685b
Signed-off-by: Laine Stump <laine@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
This will eliminate the need to call xdr_free to clear
pointers from data structures.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
This replaces use of 'rpcgen' with our new python impl of
the RPC code generator. Since the new impl generates code
that matches our style/coding rules, and does not contain
long standing bugs, we no longer need to post-process the
output.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
The new program takes the form
rpcgen [--mode source|header|repr] \
[--header include] \
xdr-file output-file
If '--mode' is not given it parses the XDR file but does not
generate anything, which is useful as a syntax check. The
'source' mode gives the '.c' file content, while 'header'
gives the '.h' file content. 'repr' gives a representation
of the abstract syntax tree, mostly useful for debugging
the parser.
If '--header' is given, it is added as a local #include ".."
statement in the output and is valid for either 'header'
or 'source' modes.
Either 'xdr-file' or 'output-file' can be omitted in which
case they default to stdin/stdout respectively.
This rpcgen program will directly include the 'config.h'
header in its output.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Test the serialization done by libtirpc, so that when we later
switch to our own code, we can prove wire compatibility.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
This implements a C code generator that emits code that is
(almost) identical to the classic 'rpcgen' program. The
key differences are:
- Skip inlining of calls for struct fields
- Skip K&R style function prototypes in headers
- Use int64_t instead of quad_t for OS portability
- Saner whitespace / indentation
The tests/demo.c and tests/demo.h files were created using
the traditional 'rpcgen' program, and then editted to cut
out the leading boilerplate, and the differences mentioned
above.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
The visitor API defines an interface for visiting each element
in the XDR protocol spec abstract syntax tree.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
This adds a parser capable of handling the XDR protocol files.
The parsing grammar requirements are detailed in
https://www.rfc-editor.org/rfc/rfc4506#section-6.3
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
This introduces classes needed to form an abstract syntax
tree representing the XDR protocol language.
The syntax requirements are detailed in
https://www.rfc-editor.org/rfc/rfc4506#section-6.3
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
This adds a lexer capable of handling the XDR protocol files.
The lexical rquirements are detailed in
https://www.rfc-editor.org/rfc/rfc4506#section-6.2
pytest is introduced as a build dependancy for testing python
code.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Currently the script will reject any type of contents outside
of a section, but we want to be able to have some useful
comments at the top of each file to help users understand how
they are processed.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
In order to further deduplicate the contents of the various unit
files, we need to be able to merge multiple additional units
into the initial one.
Luckily the merge logic is in no way constrained to working with
just two units, so achieving this is pretty much just a matter
of lifting the existing limitation on the number of arguments
that the script accepts.
As a special case, it's now also possible to call the script
with just the base unit as argument. No merging will be performed
in that case, obviously, but we'll still go through the basic
validation and cleanup steps.
This also fixes a bug in the check for the number of arguments:
sys.argv also contains the name of the script, so we should have
checked that its size was at least 3. The check is now written in
a way that's less prone to misunderstandings.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
We already use templating to generate sockets, which are all
based off libvirtd's. Push the idea further, and extend it to
cover services as well.
This is more challenging, as the various modular daemons each have
their own needs in terms of what system services needs to be
available before they can be started, which other components of
libvirt they depend on, and so on.
In order to make this sort of per-service tweaks possible, we
introduce a Python script that can merge two systemd units
together. The script is aware of the semantics of systemd's unit
definition format, so it can intelligently merge sections
together.
This generic systemd unit merging mechanism will also supersede
the extremely ad-hoc @deps@ variable, which is currently used in
a single scenario.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Fix the syntax-check failures (which can be seen after
python3-flake8-import-order package is installed) with the help
of isort[1]:
289/316 libvirt:syntax-check / flake8 FAIL 5.24s exit status 2
[1]: https://pycqa.github.io/isort/
Signed-off-by: Han Han <hhan@redhat.com>
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Using os.system("cp {0} {1}".format(...)) has two issues, it does not
work on Windows, but more importantly it can cause issues in case one of
the directories has a space in it.
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Wrap the auto-generated pages (API ref and hvsupport.html) in the proper
top level element similarly to what the pages generated from RST have to
remove the extra case when templating our web.
(Best viewed with 'git show -w')
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Common APIs such as virConnectOpen/Close and similar which are used by
the non-hypervisor drivers in libvirt are grouped together with
hypervisor drivers, which makes the table very wide.
Split them out into a separate group and clean up the list of hypervisor
drivers.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Use the proper driver struct member names for the aforementioned APIs so
that the fixup of the versions works properly.
Currently we reported that no of the drivers supported the APIs despite
being only shims above 'open'.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
The only remaining page was 'hvsupport.html' which is generated by
'scripts/hvsupport.py'. The script already has all the data to generate
the table of contents internally so we can remove the whole complicated
template.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Since all messages marked for translation contain permutable format
strings, we can add checks for enforcing them.
The syntax check does not catch all cases as it only checks format
strings between _(" and the first ". In other words messages where \"
appears before the first format string or multi-line messages where the
first format strings is not in the first line will not be checked. On
the other hand, it's run automatically by "meson test".
check-pot.py python script will detect all incorrect format strings, but
it's not as easy to use as it requires libvirt.pot to be regenerated and
this does not happen during a standard build. The following steps are
needed to check messages with check-pot.py:
meson compile libvirt-pot-dep
meson compile libvirt-pot
meson compile libvirt-pot-check
Don't forget to revert changes to libvirt.pot if you run these commands
locally as we don't want each patch series to update libvirt.pot.
Shell scripts (tools/libvirt-guests.sh.in is the only one currently)
need to be exempt from this check as shell's printf function does not
understand the permutable format strings.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
As an additional step before processing the API parse the protocol file
and extract all ACL definitions. This way we can distribute them for any
user of the libvirt API XML files. We will be also able to avoid another
call to gendispatch, which generates all this data into a standalone
XML.
The remote procedure to API name is inspired by what rpcgen does.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
If the user of the 'docBuilder' class provides a dict (key is API name,
value is a tuple of arrays (acls, aclfilters), use the dict to generate
ACL definitions into the function definition.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
It's not trivial to figure out the ACL object name from our
documentation. Add it above the table outlining existing permissions.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Both the object name and permission name in ACL use '-' instead of '_'
separator when referring to them in the docs or even when used inside of
polkit. Unfortunately the generators used for generating our docs don't
honour this in certain cases which would result in broken names in the
API docs (once they will be generated).
Rename both object and permission name to use dash and reflect that in
the anchor names in the documentation.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Certain APIs are allowed also without authentication but the ACL page
didn't outline which. Generate a new column with the information.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Check both that a file is referenced from our pages and also that pages
reference existing images.
The mode for dumping external references now also dumps images.
'--ignore-image' can be used repeatedly to suppress errors for specific
images.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Now that we have the source file name as a custom attribute we can use
it to report which file actually needs to be edited to fix the error:
ERROR: 'docs/uri.rst': broken link to: 'drvqemu.html#exaple'
rather than:
broken link targets:
docs/uri.html broken link: drvqemu.html#exaple
which pointed to file which does not exist in the source directory.
This also allows us to delete all the relative path handling needed to
report at least somewhat user-legible errors before.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Force users to pass the path to the root of the webpage the script
should check. The script lives in a different subdirectory so the
default of the current directory doesn't make much sense.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
When building the top level description from a header file the
'parseTopComment' method of the 'CParser' would include all trailing
lines into the <description> field. This was designed to concatenate
multi-line descriptions, but unfortunately in all cases also included
the Copyright statement which followed.
Explicitly end the scanning of the header on a line which starts with
'Copyright (C)' and truncate the spaces from the end of the last item.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
meson already supports $DESTDIR natively, but in this case
we're using a custom script and so we have to do some extra
work ourselves.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
The G_GNUC_NO_INLINE macro will eventually be marked as
deprecated [1] and we are recommended to use G_NO_INLINE instead.
Do the switch now, rather than waiting for compile time warning
to occur.
1: 15cd0f0461
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
Improve:
KeyError: 'virAdmConnectSetDaemonTimeout'
to
Exception: Missing symbol file entry for 'virAdmConnectSetDaemonTimeout'
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
In many cases we move around or rename internal anchors which may break
links leading to the content.
docutils handle the case of links inside a document, but we are lacking
the same form of checking between documents.
Introduce a script which cross-checks all the anchors and links in HTML
output files and prints problems and use it as a test case for the
'docs' directory.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
