Changed the API to make the shutdown behavior of workers more explicit
@@deriving bin_iofor managed workers.
spawn_in_foregroundfunction that returns a
Process.talong with the worker.
Also use this opportunity to clean up the handling of file descriptors in the
spawncase as well.
Add a couple features to Rpc_parallel to make debugging connection issues
Add some extra security around making Rpc calls to workers.
Because we do not expose the port of a worker, unless you do really
hacky things, you are going to go through Rpc_parallel when running
Rpcs. When Rpc_parallel connects to a worker, it initializes a
connection state (that includes the worker state). This
initialization would raise if the worker did not have a server
listening on the port that the client was talking to. Add some
security by enforcing unification of worker_ids instead of ports
(which will be reused by the OS).
Make an Rpc_parallel test case deterministic
Adding the mandatory arguments
Map_reduceso it is easier to debug your workers.
spawn_exnin favor of only exposing
spawn_config_exn. If you want to spawn a single worker, make a config of one worker.
Cleans up the implementation-side interface for aborting
Deferred.tthat got passed to
gone. The situations where it would have been determined now close the reading
end of the user-supplied pipe instead.
Previously, when an RPC dispatcher decided to abort a query, the RPC
implementation would get its
Deferred.tfilled in, but would remain
free to write some final elements to the pipe.
This is a little bit more flexible than the new interface, but it's also
problematic in that the implementer could easily just not pay attention to
aborted. (They're not obligated to pay attention to when the pipe is closed,
either, but at least they can't keep writing to it.) We don't think the extra
flexibility was used at all.
In the future, we may also simplify the client side to remove the
function on the dispatch side (at least when not using
dispatch_iter). For the
time being it remains, but closing the received pipe is the preferred way of
aborting the query.
There are a couple of known ways this might have changed behavior from before.
Both of these appear not to cause problems in the jane repo.
In the past, an implementation could write to a pipe as long as the client
didn't disconnect, even if it closed its pipe. Now writes will raise after
a client closes its pipe (or calls
abort), since the implementor's pipe will
also be closed in this case. Doing this was already unsafe, though, since the
pipe was closed if the RPC connection was closed.
abortedwas only determined if a client aborted the query or the connection
was closed. The new alternative,
Pipe.closedcalled on the returned pipe,
will also be determined if the implementation closes the pipe itself. This is
unlikely to cause serious issues but could potentially cause some confusing
Deal with an fd leak in Managed worker connections
There was a race condition where
spawnwould return a
Worker.teven though the worker had not daemonized yet.
This manifested itself in a spurious test case failure.
This also allowed for running the worker initialization code before we daemonize.
Reuse the master rpc settings when the worker connects using the
Make all global state and rpc servers lazy.
Toplevel hashtables are initialized with size 1 to reduce overhead due to linking with
The master rpc server is only started upon the first call to spawn
Rpc_parallelbug that caused a Master to Worker call to never return.
Add new functions
spawn_and_connection_exnwhich return back the worker and a connection to the worker.
Also, move all the example code off of the managed module
serveout of the
Connectionmodule. It shouldn't be in there because it does nothing with the
Remove some unnecessary functions defined on
Connection.t's. E.g. no longer expose
close_reasonbecause all the exception handling will be done with the registered exn handlers.
Some cleanup with how we are managing worker ids
create a record type for
Switched to PPX.
connection_timeoutargument in rpc_parallel. This argument
Rpc_parallel_core.Parallel, but it is not exposed in
Allow custom handling of missed async_rpc heartbeats.
Give a better error message when redirecting output on a remote box to a file
path that does not exist.
remove unncessary chmod 700 call on the remote executable
Give a clear error message for the common mistake of not making the
Parallel.Make_worker()functor application top-level
Make errors/exceptions in
Make stderr and stdout redirection mandatory in order to encourage logging stderr
Clean up the use of monitors across
Fix bug with exceptions that are sent directly to
Add the ability to explicitly initialize as a master and use some subcommand for the
worker. This would allow writing programs with complex command structures that don't have
to invoke a bunch of
Rpc_parallellogic and start RPC servers for every command.
Add the ability to get log messages from a worker sent back to the master.
In fact, any worker can register for the log messages of any other workers.
Fixed a file-descriptor leak
There was a file descriptor leak when killing workers. Their stdin,
stdout, and stderr remain open. We now close them after the worker process
Parallel.State.getfunction, to check whether
Map_reducemodule, which is an easy-to-use parallel map/reduce library.
It can be used to map/fold over a list while utilizing multiple cores on multiple machines.
Also added support for workers to keep their own inner state.
Fixed bug in which zombie process was created per spawned worker.
Also fixed shutdown on remote workers
Made it possible for workers to spawn other workers, i.e. act as masters.
Made the connection timeout configurable and bumped the default to 10s.
Follow changes in Async RPC