Repository navigation
Conversation
This is the first step towards having the workflow definition inside plugins
rather than in executables. This will allow accumulating plugins which are needed
to instantiate a topology and do the option parsing / topology building only once,
simplifying the current case.
The end goal is to allow the driver to preload certain common services (e.g. ROOT)
and share it among the different tasks (which at the moment it's not allowed because different
tasks are in different executables). Moreover this will allow us to coalesce strictly coupled
dataprocessors and reduce the number of running processes.
For now the plugins are embedded in the executables and behave exactly like before:
runDataProcessing.h wraps what used to be mainNoCatch in a WorkflowPlugin of kind
DplPluginKind::Workflow, which returns the whole WorkflowDefinitionContext (options,
policies, config context and specs), and callMain picks it by plugin spec
("<library>:<plugin>", with an empty library for the embedded one) before handing
it to doMain. The embedded plugin is looked up through the callback passed by main,
not with dlsym on the executable, which would need exported symbols.
--workflow-plugin <library>:<plugin>[,<library>:<plugin>...] replaces the workflow plugin embedded in the executable with the given ones, which are merged in a single workflow. An empty library means the executable itself, so :InternalCustomWorkflow adds the executable's own workflow. A plugin now only provides two steps, customize and defineWorkflow, and defineWorkflow() in the framework does the rest once for all of them: - the workflow options of all plugins are merged and parsed once, into a single ConfigContext and service registry, shared by all plugins, so that e.g. what analysis tasks request lands in the same registry. An option declared by more than one plugin is shared, provided they agree on its type and default; otherwise defining the workflow fails. The options required by DPL are added once, after those of the plugins. - the data processors are concatenated in plugin order. A data processor name defined by more than one plugin is an error. Within a single plugin nothing is checked, as before. - the policies customised by each plugin are concatenated in plugin order, followed by the default policies once, so that the catch-all defaults of one plugin do not hide the policies of the next one. For a single plugin, which is what every executable does by default, the sequence of calls is the same as before, so behaviour, help and dumps are unchanged. --workflow-plugin is a hidden driver option, it is forwarded to the children, which need it to define the same workflow, and accepted by the devices. Tested by test_WorkflowDefinition (merging, conflicting options, duplicate data processors) and test_Framework_test_WorkflowPlugins, which runs a producer and a consumer from two DSO plugins and checks that both see the same value of their shared option.
DataProcessorInfo and DeviceExecution get the plugin which defines the data processor, as <library>:<plugin>, and the workflow JSON gets a "plugin" key in the metadata of the data processor. This is what a driver needs to load the plugins of a workflow itself. Plugins embedded in the executable, i.e. with an empty library, are not recorded: their plugin stays empty and the key is not written, so the dumps of all existing executables are unchanged. Internal data processors added by DPL have no plugin either. Data processors imported from an upstream workflow keep the plugin they had there. Tested by TestDataProcessorInfoPlugin, a round trip of a workflow with and without plugins, and test_Framework_test_WorkflowPluginsDump, which dumps the workflow of the two test plugins.
A workflow opts in with WORKFLOW_PLUGIN, on o2_add_dpl_workflow or on o2_add_executable. Its sources are then compiled once, into the DSO libO2<Name>Plugin holding the workflow plugin <Name>, the CamelCase of the executable name without o2-: o2-testworkflows-diamond-workflow gives O2TestworkflowsDiamondWorkflowPlugin:TestworkflowsDiamondWorkflow. The executable keeps its name and options, and only has a main() which calls dpl_workflow_main() of the plugin it links. That one runs the plugin as the one embedded in the executable, exactly as main() does for any other workflow, so its behaviour and dumps are the same. The plugin is installed in lib with the executable. WORKFLOW_PLUGIN, rather than PLUGIN, because DPL has several kinds of plugins and this one is DplPluginKind::Workflow, loaded with --workflow-plugin; the same name works on both functions. The existing o2-dpl-run, whose embedded workflow is empty, is the generic loader: o2-dpl-run --workflow-plugin A:X,B:Y runs the workflows of the plugins, and its children are again o2-dpl-run. Workflows which do not opt in are built exactly as before. The diamond workflow, the simple source, processor and sink, and the two test plugins of test_WorkflowPlugins opt in. test_Framework_test_WorkflowPluginDump checks that the plugin of the diamond workflow, run by o2-dpl-run, defines the same workflow as its executable, apart from the executable, plugin and command line.
With DPL_FORK_WITHOUT_EXEC=1 in the environment, the driver only forks the data processors it defined itself, i.e. those not imported from an upstream workflow, including the internal ones, and the child runs the device straight away, from the topology, libraries and services the driver has already set up. Data processors imported from an upstream workflow are still exec'd, since only their executable can define them. By default nothing changes: the exec path is as before, and the new code is only reached when noExec is set. The forked child is made to look like one which did exec: - only the driver's end of the pipes of the devices forked so far, and of its own, are closed, rather than every descriptor: what the driver opened, e.g. libuv's signal lock pipe, recreated at fork, is still in use in the child. The pipes of the next devices do not exist yet. - SIGUSR1/SIGUSR2 are ignored as on the exec path, and the handlers the driver installed for SIGINT, SIGTERM and SIGCHLD are reset, as exec would do. - it gets a loop of its own, the driver's listening socket is closed, and it reads to the end the workflow the driver streams on its stdin, so the driver neither blocks on nor gets SIGPIPE from that pipe. - stdio is flushed before forking, so nothing is printed twice. - on macOS its os_log handles are replaced by OS_LOG_DISABLED: after a fork without exec os_signpost_enabled() reports every one of them as enabled, so every signpost argument would be evaluated. --signposts still works, since it does not go through os_log. This was seen as a crash in test_SimpleWildcard: the data_relayer signpost in saveInSlot dereferences a null input channel for the second wildcard route. That is an upstream bug, the same crash happens in exec mode with --signposts data_relayer. - getIdString() returns the device id, so that the workflow termination hook knows it runs in a device, as test_RootTreeWriterWorkflow needs. - it only redoes the steps the driver did not perform before forking: MERGE_CONFIGS, IMPORT_CURRENT_WORKFLOW and MATERIALISE_WORKFLOW (from 6903032bdeac).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With DPL_FORK_WITHOUT_EXEC=1 in the environment, the driver only forks
the data processors it defined itself, i.e. those not imported from an
upstream workflow, including the internal ones, and the child runs the device
straight away, from the topology, libraries and services the driver has
already set up. Data processors imported from an upstream workflow are
still exec'd, since only their executable can define them. By default
nothing changes: the exec path is as before, and the new code is only
reached when noExec is set.
The forked child is made to look like one which did exec:
of its own, are closed, rather than every descriptor: what the driver
opened, e.g. libuv's signal lock pipe, recreated at fork, is still in
use in the child. The pipes of the next devices do not exist yet.
driver installed for SIGINT, SIGTERM and SIGCHLD are reset, as exec
would do.
and it reads to the end the workflow the driver streams on its stdin,
so the driver neither blocks on nor gets SIGPIPE from that pipe.
fork without exec os_signpost_enabled() reports every one of them as
enabled, so every signpost argument would be evaluated. --signposts
still works, since it does not go through os_log. This was seen as a
crash in test_SimpleWildcard: the data_relayer signpost in saveInSlot
dereferences a null input channel for the second wildcard route. That
is an upstream bug, the same crash happens in exec mode with
--signposts data_relayer.
hook knows it runs in a device, as test_RootTreeWriterWorkflow needs.
MERGE_CONFIGS, IMPORT_CURRENT_WORKFLOW and MATERIALISE_WORKFLOW (from
6903032bdeac).