The generator can produce different code on Linux and macOS:
specifically, on the former we want to use xdr_uint64_t while
the latter needs xdr_u_int64_t instead.
This is clearly a problem for tests that involve comparing the
output produced against some expected output that's stored in
the git repository.
In the long run, we need to find a better way to handle this,
but since 9.10.0 is going to be released very soon and we don't
want it to have a broken test suite on macOS, simply skip the
generator tests on that platform for now.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
macOS XDR library is an oddball using xdr_u_int64_t instead of
xdr_uint64_t which everyone else has.
The code generator already does the right thing, but the test
program previously generated with the Linux rpcgen program
does not compile on macOS due to this.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
The test_demo program currently fails to compile on macOS with
too few arguments to function call, expected 3, have 2
ret = !!proc(&xdr, vorig);
~~~~ ^
Way back in 2013, commit 9fa3a8ab6f handled this situation
for the main library code. Apply the same fix here.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
These are currently the only tests that are not part of any
suite.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Some of the files used by test_demo.c can only be regenerated
when pytest is present, but we have pre-generated copies in the
repository, so overall we just need the C compiler to build and
run that specific test program.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
There are some platforms where 'char' is unsigned, by default
(RPi, s390x to name a few). And because of how test_demo is
written we are experiencing some test cases failing there. For
instance: /xdr/struct-scalar is failing. This is because in the
test (test_struct_scalar()), we have a struct with two chars. One
is initialized to 0xca, the other 0xfe (note that both have the
MSB set). The XDR encoder (xdr_TestStructScalar()) then calls
xdr_char() on both of them. But XDR itself has no notion of
char type, so under the hood, it expands it to int [1] and calls
xdr_int(). And this is where the problem lies. On platforms where
char is signed, the integer expansion results in 0xffffffca, but
on platforms where char is unsigned it results in 0x000000ca. Two
distinct results.
The test then goes and compares the encoded buffer with an
expected one (memcmp(), read from the disk earlier).
This poses no problem for real life use, because when decoding
those chars back, the padding is thrown away.
To avoid tickling this issue, use values that don't have the MSB
set.
1: https://git.linux-nfs.org/?p=steved/libtirpc.git;a=blob;f=src/xdr.c;h=28d1382cc4853ecf1238d792af5016160435d1e0;hb=HEAD#l487
Fixes: 40cbaa8fbe rpcgen: add test case for XDR serialization
Reported-by: Boris Fiuczynski <fiuczy@linux.ibm.com>
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Boris Fiuczynski <fiuczy@linux.ibm.com>
The test_demo program compares whether XDR encoded data match the
expected output as read from a file. But the file path is not
absolute and thus relative to CWD which means the program can run
only from one specific directory.
Do what we do in the rest of our test suite: define 'abs_srcdir'
macro and prefix the path with it.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Jiri Denemark <jdenemar@redhat.com>
flake8 (run on all python scripts as a part of the syntax checks)
version 6.1.0 (on macOS 14) issued many complaints like this on the
new rpcgen python scripts:
[...]libvirt/scripts/rpcgen/rpcgen/lexer.py:57:17: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
This patch changes all [type] == [type] to use "is" instead of "==",
and similarly to use "is not" instead of "!=".
(flake8 5.03, e.g. on Fedora 38, is just fine with using "==" and "!=",
but python on both likes "is" and "is not")
Fixes: commit v9.9.0-24-g8ec79e5e14
Fixes: commit v9.9.0-22-gca3f025011
Fixes: commit v9.9.0-21-g031efb691f
Fixes: commit v9.9.0-20-g8c8b97685b
Signed-off-by: Laine Stump <laine@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
This will eliminate the need to call xdr_free to clear
pointers from data structures.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
This replaces use of 'rpcgen' with our new python impl of
the RPC code generator. Since the new impl generates code
that matches our style/coding rules, and does not contain
long standing bugs, we no longer need to post-process the
output.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
The new program takes the form
rpcgen [--mode source|header|repr] \
[--header include] \
xdr-file output-file
If '--mode' is not given it parses the XDR file but does not
generate anything, which is useful as a syntax check. The
'source' mode gives the '.c' file content, while 'header'
gives the '.h' file content. 'repr' gives a representation
of the abstract syntax tree, mostly useful for debugging
the parser.
If '--header' is given, it is added as a local #include ".."
statement in the output and is valid for either 'header'
or 'source' modes.
Either 'xdr-file' or 'output-file' can be omitted in which
case they default to stdin/stdout respectively.
This rpcgen program will directly include the 'config.h'
header in its output.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Test the serialization done by libtirpc, so that when we later
switch to our own code, we can prove wire compatibility.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
This implements a C code generator that emits code that is
(almost) identical to the classic 'rpcgen' program. The
key differences are:
- Skip inlining of calls for struct fields
- Skip K&R style function prototypes in headers
- Use int64_t instead of quad_t for OS portability
- Saner whitespace / indentation
The tests/demo.c and tests/demo.h files were created using
the traditional 'rpcgen' program, and then editted to cut
out the leading boilerplate, and the differences mentioned
above.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
The visitor API defines an interface for visiting each element
in the XDR protocol spec abstract syntax tree.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
This adds a parser capable of handling the XDR protocol files.
The parsing grammar requirements are detailed in
https://www.rfc-editor.org/rfc/rfc4506#section-6.3
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
This introduces classes needed to form an abstract syntax
tree representing the XDR protocol language.
The syntax requirements are detailed in
https://www.rfc-editor.org/rfc/rfc4506#section-6.3
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
This adds a lexer capable of handling the XDR protocol files.
The lexical rquirements are detailed in
https://www.rfc-editor.org/rfc/rfc4506#section-6.2
pytest is introduced as a build dependancy for testing python
code.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Currently the script will reject any type of contents outside
of a section, but we want to be able to have some useful
comments at the top of each file to help users understand how
they are processed.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
In order to further deduplicate the contents of the various unit
files, we need to be able to merge multiple additional units
into the initial one.
Luckily the merge logic is in no way constrained to working with
just two units, so achieving this is pretty much just a matter
of lifting the existing limitation on the number of arguments
that the script accepts.
As a special case, it's now also possible to call the script
with just the base unit as argument. No merging will be performed
in that case, obviously, but we'll still go through the basic
validation and cleanup steps.
This also fixes a bug in the check for the number of arguments:
sys.argv also contains the name of the script, so we should have
checked that its size was at least 3. The check is now written in
a way that's less prone to misunderstandings.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
We already use templating to generate sockets, which are all
based off libvirtd's. Push the idea further, and extend it to
cover services as well.
This is more challenging, as the various modular daemons each have
their own needs in terms of what system services needs to be
available before they can be started, which other components of
libvirt they depend on, and so on.
In order to make this sort of per-service tweaks possible, we
introduce a Python script that can merge two systemd units
together. The script is aware of the semantics of systemd's unit
definition format, so it can intelligently merge sections
together.
This generic systemd unit merging mechanism will also supersede
the extremely ad-hoc @deps@ variable, which is currently used in
a single scenario.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Fix the syntax-check failures (which can be seen after
python3-flake8-import-order package is installed) with the help
of isort[1]:
289/316 libvirt:syntax-check / flake8 FAIL 5.24s exit status 2
[1]: https://pycqa.github.io/isort/
Signed-off-by: Han Han <hhan@redhat.com>
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Using os.system("cp {0} {1}".format(...)) has two issues, it does not
work on Windows, but more importantly it can cause issues in case one of
the directories has a space in it.
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Wrap the auto-generated pages (API ref and hvsupport.html) in the proper
top level element similarly to what the pages generated from RST have to
remove the extra case when templating our web.
(Best viewed with 'git show -w')
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Common APIs such as virConnectOpen/Close and similar which are used by
the non-hypervisor drivers in libvirt are grouped together with
hypervisor drivers, which makes the table very wide.
Split them out into a separate group and clean up the list of hypervisor
drivers.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Use the proper driver struct member names for the aforementioned APIs so
that the fixup of the versions works properly.
Currently we reported that no of the drivers supported the APIs despite
being only shims above 'open'.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
The only remaining page was 'hvsupport.html' which is generated by
'scripts/hvsupport.py'. The script already has all the data to generate
the table of contents internally so we can remove the whole complicated
template.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Since all messages marked for translation contain permutable format
strings, we can add checks for enforcing them.
The syntax check does not catch all cases as it only checks format
strings between _(" and the first ". In other words messages where \"
appears before the first format string or multi-line messages where the
first format strings is not in the first line will not be checked. On
the other hand, it's run automatically by "meson test".
check-pot.py python script will detect all incorrect format strings, but
it's not as easy to use as it requires libvirt.pot to be regenerated and
this does not happen during a standard build. The following steps are
needed to check messages with check-pot.py:
meson compile libvirt-pot-dep
meson compile libvirt-pot
meson compile libvirt-pot-check
Don't forget to revert changes to libvirt.pot if you run these commands
locally as we don't want each patch series to update libvirt.pot.
Shell scripts (tools/libvirt-guests.sh.in is the only one currently)
need to be exempt from this check as shell's printf function does not
understand the permutable format strings.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
As an additional step before processing the API parse the protocol file
and extract all ACL definitions. This way we can distribute them for any
user of the libvirt API XML files. We will be also able to avoid another
call to gendispatch, which generates all this data into a standalone
XML.
The remote procedure to API name is inspired by what rpcgen does.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
If the user of the 'docBuilder' class provides a dict (key is API name,
value is a tuple of arrays (acls, aclfilters), use the dict to generate
ACL definitions into the function definition.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
It's not trivial to figure out the ACL object name from our
documentation. Add it above the table outlining existing permissions.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Both the object name and permission name in ACL use '-' instead of '_'
separator when referring to them in the docs or even when used inside of
polkit. Unfortunately the generators used for generating our docs don't
honour this in certain cases which would result in broken names in the
API docs (once they will be generated).
Rename both object and permission name to use dash and reflect that in
the anchor names in the documentation.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Certain APIs are allowed also without authentication but the ACL page
didn't outline which. Generate a new column with the information.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Check both that a file is referenced from our pages and also that pages
reference existing images.
The mode for dumping external references now also dumps images.
'--ignore-image' can be used repeatedly to suppress errors for specific
images.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Now that we have the source file name as a custom attribute we can use
it to report which file actually needs to be edited to fix the error:
ERROR: 'docs/uri.rst': broken link to: 'drvqemu.html#exaple'
rather than:
broken link targets:
docs/uri.html broken link: drvqemu.html#exaple
which pointed to file which does not exist in the source directory.
This also allows us to delete all the relative path handling needed to
report at least somewhat user-legible errors before.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Force users to pass the path to the root of the webpage the script
should check. The script lives in a different subdirectory so the
default of the current directory doesn't make much sense.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
When building the top level description from a header file the
'parseTopComment' method of the 'CParser' would include all trailing
lines into the <description> field. This was designed to concatenate
multi-line descriptions, but unfortunately in all cases also included
the Copyright statement which followed.
Explicitly end the scanning of the header on a line which starts with
'Copyright (C)' and truncate the spaces from the end of the last item.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
meson already supports $DESTDIR natively, but in this case
we're using a custom script and so we have to do some extra
work ourselves.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
The G_GNUC_NO_INLINE macro will eventually be marked as
deprecated [1] and we are recommended to use G_NO_INLINE instead.
Do the switch now, rather than waiting for compile time warning
to occur.
1: 15cd0f0461
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
Improve:
KeyError: 'virAdmConnectSetDaemonTimeout'
to
Exception: Missing symbol file entry for 'virAdmConnectSetDaemonTimeout'
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
In many cases we move around or rename internal anchors which may break
links leading to the content.
docutils handle the case of links inside a document, but we are lacking
the same form of checking between documents.
Introduce a script which cross-checks all the anchors and links in HTML
output files and prints problems and use it as a test case for the
'docs' directory.
Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
The parameters of self.warning is inconsistent with its definition, So
fix it.
Signed-off-by: luzhipeng <luzhipeng@cestc.cn>
Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
This makes it mandatory to *not* add 'v' to version numbers.
Signed-off-by: Victor Toso <victortoso@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
scripts/apibuild.py did not consider exporting external variable's
comments into the XML API. This commits fixes that.
Noe that the way that CParser is designed, it is currently possible to
lose a parsed comment when parsing other fields as self.comment in
several places. I've added a comment to highlight this.
Signed-off-by: Victor Toso <victortoso@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
So we can use for comments that are being hold in helper variables.
Signed-off-by: Victor Toso <victortoso@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
This patch adds 'version' parameter to generated XML API for functions
and functypes.
The 'version' metadata has been added with e0e0bf6628 by parsing .syms
files. This commit does not override that but it will warn if there is
not 'Since' metadata with new additions.
There is not clear benefit for keeping both. For now, I've added a
warning in case there is a mismatch between the version provided by
.syms and docstring.
Signed-off-by: Victor Toso <victortoso@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
This patch adds 'version' parameter to the generated XML API for
macros
It'll require, for new additions, to add a comment with the version
that the macro was added. An example bellow of code diff and
the change in the generated XML.
Note that the Since tag is removed from the comment as there is a
proper field for it in the XML.
Signed-off-by: Victor Toso <victortoso@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
