I needed to solve the task of having a minion execute a job on another minion and this is how I accomplished the task. This stemmed from having a minion that runs a web applications which just needed to execute an execution module that already existed, but needed to run on a different minion.
My solution depends on Salt events and reactors to orchestrate the request from the web server minion to the worker minion and have the worker minion give a response of the work it completed back to the web server minion.
Salt version: salt 2016.3.4
Machines:
| Host | Salt Role | Use Case | 
| saltmast16 | Salt Master | Orchestrator | 
| saltmina16 | Salt Minion | Where execution module will run | 
| saltminb16 | Salt Minion | Web server that initiates work | 
/srv/salt/_modules/rhtest.py
The event.fire_master call sends the com/millamilla/test/event/finish event to the master
| saltminb16
 Web Server  | 
Web Server Files/ Code | salmast16
 Master  | 
Master Files/ Code | saltmina16
 Worker  | 
Worker Files/ Code | |
| Send an event from the web server to the master. | Send event com/millamilla/test/event/start  | 
Python: send_event.py 
import salt.client
import uuid
caller = salt.client.Caller()
event_uuid = str(uuid.uuid1())
caller.sminion.functions['event.send'](
    'com/millamilla/test/event/start',
    {
      'target': "saltmina16",
      'site': "moodle",
      'user': "hedrickbt",
      'uuid': event_uuid,
    }
)
 | 
Receive event com/millamilla/test/event/start  | 
 /etc/salt/master.d/reactor.conf
reactor:
  # ...
  - 'com/millamilla/test/event/start':        
    - salt://reactor/rhtesteventstart.sls   
  # ...
#...  | 
||
| >>>>>>>>>>>>>>>>>>>> | >>>>>>>>>>>>>>>>>>>>>>>>>>>> | |||||
| Master handles event and causes execuction module to run on worker minion | Handle com/millamilla/test/event/start Event. Call execution module on worker. |  /srv/salt/reactor/rhtesteventstart.sls
run_it:
  local.rhtest.rhfunc:
    - tgt: {{ data['data']['target'] }}
    - arg:
      - {{ data['data']['site'] }}
      - {{ data['data']['user'] }}
      - {{ data['id'] }}
      - {{ data['data']['uuid'] }}
  | 
Execute module | Created on master: /srv/salt/_modules/rhtest.py and sync’d to minions from master via: salt ‘*’ saltutil.sync_all import salt
__outputter__ = {
  'rhfunc': 'yaml'
}
def rhfunc(site, user, requesting_host, event_uuid):
  result = {
    "site":site,
    "user":user,
    "requesting_host":requesting_host,
    "uuid":event_uuid
  }
  __salt__['event.fire_master'](
    {
      "site":site,
      "user":user,
      "requesting_host":requesting_host,
      "uuid":event_uuid,
    },
    'com/millamilla/test/event/finish',
  )
  return result
 | 
||
| >>>>>>>>>>>>>>>>>>> | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | |||||
| After worker minion executes modules send event back to master | Receive event com/millamilla/test/event/finish  | 
/etc/salt/master.d/reactor.conf
reactor:
  # ...
  - 'com/millamilla/test/event/finish':
    - salt://reactor/rhtesteventfinish.sls 
  # ...
 | 
||||
| <<<<<<<<<<<<<<<<<<< | <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< | |||||
| Forward event Master received to the web minion. | Receive/Handle Event
 com/millamilla/test/event/finish  | 
Python: receive_event.py
 This needs to run just after sending the message so it can be listening. You will want to add code that has a timeout in the for…print loop. Otherwise, it could run forever. You will also want to add code to look at the data[‘uuid’] to make sure it is the one you sent out. Typically, you wouldn’t have a separate send/receive program, but you can. I plant to create a single script that generates the uuid, sends the request, and waits for a message to return within a timeout period. If the timeout is exceeded an error would be thrown. import salt.config
import salt.utils.event
import socket
import os
# Import 3rd-party libs
import salt.ext.six as six
my_hostname = socket.gethostname()
#opts = salt.config.client_config('/etc/salt/minion')
opts = {}
opts['node'] = 'minion'
opts['sock_dir'] = '/var/run/salt'
opts['sock_dir'] = os.path.join(opts['sock_dir'], opts['node'])
opts['id'] = my_hostname
opts['transport'] = 'zeromq'
event = salt.utils.event.get_event(
        opts['node'],
        sock_dir=opts['sock_dir'],
        transport=opts['transport'],
        opts=opts)
for data in event.iter_events(tag='com/millamilla/test/event/finish'):
    print(data)
 | 
Handle Event
 com/millamilla/test/event/finish  | 
 /srv/salt/reactor/rhtesteventfinish.sls
notify_it:
  local.event.fire:
  - tgt: {{ data['data']['requesting_host'] }}
  - arg:
    - {{ data['data'] }}
    - com/millamilla/test/event/finish
 | 
||
| <<<<<<<<<<<<<<<<<<< | <<<<<<<<<<<<<<<<<<< | |||||