Go to www.awrcorp.com
Back to search page Click to download printable version of this guide.

Chapter 11. EM: Job Scheduler

The Job Scheduler is a separate program that configures and schedules AXIEM and AnalystTM EM simulations in the NI AWR Design Environment suite. When you run a local or remote simulation, the Job Scheduler determines when to start it based on licenses available and the number of processors to use for the simulation.

11.1. Running Asynchronous Simulations

The Job Scheduler, along with data sets (see Data Sets for more information), allows simulations it controls to run asynchronously. Asynchronous simulation allows you to do the following during a simulation:

  • Continue to work on other areas of your design, including the EM structure that is simulating.

  • Edit the same EM structure, and when you simulate, the new version also simulates.

  • Open and work in a different project. The EM simulations from the previous project continue to simulate. Data is written to data sets and the data is available after the simulation is complete.

The status of jobs controlled by the Job Scheduler displays in the Simulation dialog box. For example, in a project with one AXIEM EM stand-alone structure, when it is first simulated, the Simulation dialog box displays as shown in the following figure.

While this simulation is running, you can edit the same EM document and run the simulation again. The Simulation dialog box now displays as shown in the following figure.

The first simulation is shown as running and the second simulation is shown as scheduled. When the first job finishes, the second job starts simulating, as shown in the following figure.

Note the following in the Status dialog box:

  • For each job, the name of the EM document, the simulator type, and the name of the data set being written to are listed next to the job number. For example, "Rat Race:Axiem:EM_Sim_DS1" indicates that the document name is Rat Race, the simulator is AXIEM, and the data set being written to is EM_Sim_DS1.

  • You can select individual jobs. The selected job displays its simulation status in the Output log section of the dialog box.

  • You can select multiple jobs by pressing the Shift or Ctrl key while selecting the individual jobs.

  • You can pause one or more simulation jobs by clicking the Pause button. Click the Resume button to continue the simulation.

  • You can cancel one or more simulation jobs by clicking the Cancel button.

11.1.1. Extraction Issues

Using extraction is a special case that currently does not run asynchronously because another simulation (a measurement pointing at the schematic that uses the EXTRACT block) needs to occur after the current EM simulation is complete. The following figure shows an example of the Simulation dialog box when simulating with extraction.

In this case, the "Wilkinson" schematic contains the EXTRACT block and is the top level simulation listed. The second job in the list shows that the schematic simulation is dependent on this simulation finishing. Also notice that the Type is set to "Sync", so the environment is locked until this simulation completes.

To make extraction simulations behave asynchronously:

  1. Right-click the EXTRACT block in the schematic, or the schematic in the Project Browser, and choose Add Extraction to prepare the EM structure for simulation.

  2. Right-click the EM document created in the previous step and choose Simulate to run the EM simulation asynchronously.

  3. Simulate the project to make the schematic simulate using the extracted EM results.

11.2. Setting Job Scheduler Options

The Job Scheduler has its own set of options. To set project level options, choose Options > Default EM Options and click the Job Scheduler tab on the EM Options dialog box. To set Job Scheduler options for each EM structure, right-click the EM structure node in the Project Browser and choose Options, then click the Job Scheduler tab on the Options dialog box.

Some of the main Job Scheduler settings are:

  • Max Threads per Job: Controls the maximum number of execution threads to use for the job. A value of 0 means to use all available processors, up to a maximum of 8. A negative value means to use the maximum on the machine minus this number. A positive value means to use that maximum number. When using positive numbers you might not get optimal use of the simulator. For example, if you set the value to 1 and you have 4 processors, the simulation runs approximately 4x slower than it would if set to 0. Additionally, if used on a different computer with more cores than the computer setup, this value could limit the simulation speed.

  • Max Parallel Jobs per Computer: Controls the maximum number of parallel NI AWR Design Environment jobs that run per computer. Note that this limit excludes the NI AWR Design Environment client process that launched the jobs (so if you are running four parallel jobs on the computer on which you launched the simulation, you will see five NI AWR Design Environment processes running at the same time). If there are multiple computers in a remote simulation queue, this limit applies to the number of parallel jobs on each computer.

  • Job Process Priority: Controls the priority of execution in the Job Scheduler.

  • Remote Execution: Controls job assignments to remote computers (in combination with the LocalComputePerformance option set via the Job Scheduler Admin dialog box). The default is Normal, so jobs are never rejected by remote computers. If set to High, a job is only assigned to a remote computer with Server Performance set as High, and rejected by computers with this option set to Normal. If no remote computers satisfy the Remote Performance Preference of the job, the job is rejected by the Scheduler node.

  • Max Computers for Distributed Jobs: Controls the maximum number of computers used to run the distributed simulation. A value of 0 means to use all available computers.

  • Remote Memory Preference: Controls job assignments to remote computers (in combination with the LocalMemoryCapacity option set via the Job Scheduler Admin dialog box). The default is Normal, so jobs are never rejected by remote computers. If set to High, a job is only assigned to a remote computer with Server Memory Capacity set as High, and rejected by computers with this option set to Normal. If no remote computers satisfy the Remote Memory Preference of the job, the job is rejected by the scheduler node.

11.3. Using the Job Monitor

The Job Scheduler can display job status in the Simulation dialog box while the NI AWR Design Environment program is running. While the program is closed you can use the Job Monitor to view the job status.

To launch the Job Monitor, choose Tools > Job Monitor or open it from the Start menu AWRDE program folder.

The Job Monitor displays as shown in the following figure.

The Job Monitor normally displays all jobs running from all users on remote computers, but can also display only local jobs by changing the selection in the drop-down box. Select any job in the list and use the dialog box buttons to cancel, pause, or resume jobs. Click the Details button to display additional job information that can help diagnose any problems that might occur. To remove completed jobs from the list, clear the Show Ended Jobs check box.

When you submit a job to the scheduler node, the Job Status is "Pending". Once all input files are transferred from the user computer to the scheduler node, the status becomes "Scheduled". The status updates to "Running" when the simulation starts, and to "Ended" once the simulation is complete. After the status changes to "Ended" on the remote machine, there may be a delay before it changes from "Running" to "Finished" in the user computer Simulation window, due to data transfer delays.

If the scheduler node on a user computer changes while the Job Monitor is open, you must restart the Job Monitor to display jobs on the new scheduler node.

11.4. Utilizing Remote Computing

Remote computing allows you to set up an EM simulation (AXIEM or Analyst) on a local computer, but simulate the structure on a different computer. By setting up dedicated, remote computers for resource intensive simulations, you can use local computers for other work or smaller simulations.

Licensing: No additional licenses are required for remote computing. You must have a front-end license (XXX-001) to submit remote simulation jobs, and the remote computers must have TOK licenses to simulate. Remote scheduler nodes that function solely as queue managers (not enabled for simulations) do not need a license.

Version Compatibility: Multiple versions of NI AWR Design Environment software can be installed on either the user or remote computers. However, in order to simulate, an exact version match must be found between the NI AWR Design Environment version used to submit an EM simulation job on the user computer, and the simulator version (AXIEM or Analyst) on the remote computer.

On remote computers, the AWR_JobScheduler service manages the simulation queue. You must install this service from the latest version of NI AWR Design Environment software installed on the computer. The service supports jobs submitted from lower versions of NI AWR Design Environment software, but not from higher versions. For example, a remote computer running a v14 Job Scheduler service can simulate jobs submitted from user computers running v13. But a v13 Job Scheduler service cannot run jobs submitted from a v14 user computer.

11.4.1. Remote Server Configurations

Each remote simulation is referred to as a job. The following are the various computers in a remote simulation set up:

  • User computer - This is the designer's local (client) computer that is used to set up the job.

  • Scheduler node - This is the remote (server) computer that the designer submits jobs to. The scheduler node maintains the job queue and distributes jobs to available compute nodes.

  • Compute node - This is the remote (server) computer that runs the simulation. Compute nodes do not maintain a queue, and can only accept one job at a time. Note that the scheduler node can also function as a compute node.

The following sections describe the supported server configurations. You can set up multiple scheduler nodes (with connected compute nodes) on a network, but user computers can only connect to one scheduler node at a time.

One Remote Computer

In this configuration, only one remote computer is available. It is set up as both a scheduler node and a compute node. It maintains a job queue, and also run the jobs in order of submission.

Multiple Remote Computers

In this configuration, several remote computers are available. One computer is set up as the scheduler node, and the remaining computers are set up as compute nodes. User computers submit jobs to the scheduler node. The scheduler node maintains a job queue and then distributes the jobs in the order of submission to available compute nodes. The scheduler node can be set up to also function as a compute node and run jobs, or you can disable simulations on it and designate it as strictly a queue manager.

Job Distribution

Each EM simulation counts as one job. For swept EM documents, each sweep counts as one job. If multiple compute nodes are available (when using multiple remote computers), jobs from one project might be distributed among several computers and simulated in parallel. A single job is simulated on one computer, and not spread among several computers.

11.4.2. Installing and Setting Up Remote Computing

The following steps provide an overview of the installation and set up required for remote computing. The sections that follow provide the details for these steps.

  1. On each remote computer, run a command line installation of NI AWR Design Environment software to set up the service.

  2. On each remote computer, configure the Job Server settings.

  3. For a configuration with multiple remote machines, add compute nodes to the scheduler node's Remote Hosts table.

  4. After remote computer setup, add the scheduler node to the user computer's remote host table.

User Computer Installation and Set Up

For user computers, follow the standard NI AWR Design Environment installation procedure. No additional installation procedure is required to enable remote computing functionality.

To simulate remotely, you must specify a scheduler node on the user computer. To add a new scheduler node by name:

  1. Choose Tools > Job Scheduler Admin to display the Job Scheduler Admin dialog box, or open it from the Start menu AWRDE program folder.

  2. Click the Create a new host entry button above the Remote Hosts table to add a new entry. The cursor moves to the Entry Name column in edit mode.

  3. Type the scheduler node name (“Sched1” in the following figure) and click Enter to add that host to the list. The host (scheduler node) must be a different computer. You cannot add the same computer as a host to itself.

  4. Click the Verify Hosts button to check the connection between the user computer and the scheduler node. A Verify Hosts window displays with a checkmark next to the remote computer name if verification is successful. Click OK to close this window.

    The Job Scheduler Admin dialog box now shows the scheduler node as "Verified". The Version column lists the version of the AWR_JobScheduler service running on the scheduler node, and the Queue Type column switches from auto-detect to "Scheduler".

    NOTE: Do not add a compute node to a user computer remote host table because a user computer should not connect to a compute node. The Queue Type for a user computer remote host entry should always be "Scheduler" and not "Compute".

  5. Click the Save button to save the current host list and then click Close to close the Job Scheduler Admin dialog box.

The user computer can now submit the job to the scheduler node. See “Using Remote Simulation” for details.

Connecting to Remote Servers

The Remote Hosts table can list multiple scheduler nodes, but a user computer can only connect to one scheduler node at a time. To disconnect from a scheduler node, clear the check box next to its name in the Entry Name column of the Remote Hosts table, then click the Save button. You do not need to delete the entry. To reconnect to a scheduler node, select the check box next to its name in the Entry Name column and click the Save button.

Remote Computer Installation and Set Up

The following sections apply to scheduler and compute node computers only, not to user computers. For scheduler and compute nodes, you must install the NI AWR Design Environment software from a command shell with Administrator privileges.

New Installations

The following command line syntax installs the NI AWR Design Environment software with the Job Scheduler service to default location and writes the installation log to awrde_install.log. See the Installation Guide for additional command line arguments.

awrde_<version>.exe /ComponentArgs "AWRDE":"JOBSERVICE=YES" /Log /LogFile .\awrde_install.log

After running the installers, you need to specify the license file location by either running the NI AWR Design Environment program, or by running LicLocSetup.exe from the NI AWR Design Environment installation directory. You also need to restart the Job Scheduler service in order to configure the Job Scheduler for the newly set license. See “Using the AWR_JobScheduler Service” for details on managing the service. NOTE: Licensing changes always require a restart of the Job Scheduler service.

After installation, open the Windows Task Manager and click the Services tab to verify that the AWR_JobScheduler service is listed with its Status as "Running". For further details and service options, click the Services button at the bottom of the window to display the Services window. After the Job Scheduler is registered as a service, it launches whenever Windows starts and serves jobs on that computer until the computer shuts down or you stop the service. See “Using the AWR_JobScheduler Service” for information on stopping and removing the Job Scheduler service.

NOTE: Running the NI AWR Design Environment installer again switches the version of the Job Scheduler service to that of the most recent installation. When installing multiple versions of NI AWR Design Environment software on a remote computer, you should install from the lowest to the highest version number to ensure that the highest version of the Job Scheduler service is installed. See “Manually Setting the Service Version” for information on manually changing the service version.

Manual Upgrade

Before upgrading to a newer version of the NI AWR Design Environment program you need to stop the AWR_JobScheduler service. See “Using the AWR_JobScheduler Service” for details. After stopping the service, the upgrade procedure is the same as for new installations. The Job Scheduler service automatically migrates to the newer version of NI AWR Design Environment software when you run the installer.

Remote Install and Uninstall

Compute nodes can be remotely updated from the scheduler node if all nodes are running Job Scheduler service version 13.02 or higher. Additional scripts are required for remote upgrades, and are available from the Knowledge Base. See Compute Node Remote Update for details.

Major Version Upgrades

Job Scheduler service settings migrate when upgrading to a new major version, such as upgrading from V13 to V14. If you are migrating from V12 or earlier, you must reconfigure all remote computers following the “Scheduler Node Set Up” and “Compute Node Setup” procedures.

Licensing or .ini File Changes

Whenever there is a licensing change or changes to the mwoffice.ini or redirect.ini files, you must restart the AWR_JobScheduler service on all remote scheduler and compute nodes. See “Using the AWR_JobScheduler Service” for details.

Compute Node Setup

After the AWR_JobScheduler service is installed on compute node computers, setup is complete. No other steps are needed on these computers, unless you want advanced options. See “Advanced Remote Computer Settings” for information on advanced options.

Scheduler Node Set Up

After the AWR_JobScheduler service is installed, follow these steps:

  1. Choose Tools > Job Scheduler Admin to open the Job Scheduler Admin dialog box or open it from the Start menu AWRDE program folder.

  2. Set Editing to "All User (Service) Settings".

  3. Under Job Server Settings, change QueueTypeOverride to Scheduler.

  4. If the scheduler node is the only remote computer and is also functioning as the compute node, then click Save and Close to complete scheduler node setup.

  5. If there are additional compute nodes, you must add them to the Remote Hosts table of the scheduler node. The process is the same as adding a new scheduler node for a user computer.

    1. Click the Create a new host entry button to add the compute nodes. The host (compute nodes) must be a different computer. You cannot add the same computer as a host to itself.

    2. Select the check box in the Entry Name column to enable the compute nodes.

    3. Click the Verify Hosts button to check connectivity between the scheduler node and compute nodes.

    4. After adding all the compute nodes, click the Save and then Close buttons to complete scheduler node setup for multiple remote computers.

Advanced Remote Computer Settings

The scheduler node by default also functions as an additional compute node, and runs jobs. You can change this functionality in the Job Scheduler Admin dialog box in the Job Server Settings table:

  • LocalPriority = RemoteFirst makes the scheduler node less desirable in terms of job assignment. Jobs are assigned to other compute nodes first, and run on the scheduler node only if no other compute node is available to run a job.

  • NoLocalJobs = True disables simulation on the scheduler node. The scheduler node only functions as the queue manager, and assigns all jobs to compute nodes.

See “Job Scheduler Admin Dialog Box: General Settings Tab ” for more information on Job Server settings.

In the Job Server Settings table you can use the LocalComputePerformance and LocalMemoryCapacity options to differentiate between remote computer capabilities. These settings, along with corresponding job preferences, are used to control job assignment. The scheduler node only assigns jobs to compute nodes that satisfy these performance and memory preferences of the job. If no computer can satisfy the requirements, the job is rejected by the scheduler node and the job cannot run remotely. See “Job Scheduler Admin Dialog Box: General Settings Tab ” for Job Server Setting details, and “Setting Job Scheduler Options” for details on setting job remote preferences on a user computer.

11.4.3. Using the AWR_JobScheduler Service

You may occasionally need to stop the AWR_JobScheduler Service for upgrades, scheduled maintenance, or other. You should stop and restart all remote computers together. Do not restart a compute node without restarting the scheduler node. It is important to stop, and then restart in the following order, using the shutdown command:

  1. Issue the shutdown command on the scheduler node first to prevent any new jobs from starting. The scheduler node waits for any current jobs in the queue to finish before stopping.

  2. Issue the shutdown command on each compute node. Each compute node completes the current job before stopping.

  3. Restart the service on compute nodes.

  4. Restart the service on scheduler node.

Stopping the Service

To prevent data loss, you must stop the AWR_JobScheduler service before uninstalling the NI AWR Design Environment program or installing a newer version of the NI AWR Design Environment software. To stop the service, run the following command from the NI AWR Design Environment installation directory with Administrator privileges:

awr_jobschedulercontrol –controller "local service" –command shutdown

With this command, the Job Scheduler service waits for the currently running job(s) to complete, saves the job queue, and then shuts down. A restart of the service reloads the queue and starts running the next job(s) in the queue (if any). Alternatively, you can immediately stop the AWR_JobScheduler service from the Services window or command shell, although this results in loss of data for any currently running job(s). To access this window, open the Windows Task Manager and click the Services tab, then click the Services button at the bottom of the window. The command to immediately stop the service is:

net stop AWR_JobScheduler

Starting the Service

After installation, the AWR_JobScheduler service automatically restarts when Windows starts, or anytime an NI AWR Design Environment installer is run. You can also restart the service from the Services window or command shell. To access this window, open the Windows® Task Manager and click the Services tab, then click the Services button at the bottom of the window. The command to start the service is:

net start AWR_JobScheduler

Manually Setting the Service Version

After the AWR_JobScheduler service is installed, any subsequent NI AWR Design Environment installations change the version of the service to the version of the installation. The service version also changes when using the installer to repair an installation, and when uninstalling the NI AWR Design Environment suite without the command line option to remove the service. Problems can arise if the version of the service is unintentionally switched to a lower version. Re-running the installer in the highest version of the NI AWR Design Environment software resets the version, or you can manually reset the version.

Before switching the version of the AWR_JobScheduler service, first stop the service. From a command shell with Administrator privileges, change directories to the desired NI AWR Design Environment installation directory and execute the following two commands separately:

AWR_JobSchedulerService /UnregServer

AWR_JobSchedulerService /Service

You can now restart the service.

Uninstalling the Service

Before removing the AWR_JobScheduler service, stop the service to prevent loss of data. The following commands remove the AWR_JobScheduler service, converting the computer from a remote simulation computer to a user computer. For scheduler and compute nodes, you must uninstall the NI AWR Design Environment software from a command shell with Administrator privileges. For NI AWR Design Environment installation removal, use the following command line syntax:

awrde_<version>.exe /ComponentArgs "AWRDE":"JOBSERVICEOFF=YES" /LOG /LogFile .\awrde_uninstall.log

11.4.4. Using Remote Simulation

Remote simulation is specified via the Remote Execution option. You can set this option for each EM structure, or set it globally on the Options dialog box Job Scheduler tab by selecting the Use project defaults check box. By default, Remote Execution is set to Local Only, so the EM structure simulates on the user computer. To change this option to use remote computers:

  1. On the user computer, right-click the EM structure in the Project Browser, and choose Options.

  2. In the Options dialog box, click the Job Scheduler tab.

  3. Set Remote Execution to Prefer Local, Remote Only, Prefer Remote or Any Available. Remote Only jobs only simulate on the remote computers.

  4. Choose Simulate > Analyze to simulate. The job simulates locally on the user computer or simulates remotely, depending on Remote Execution settings and computer availability. The scheduler node maintains a queue, so if all computers are busy the job is queued until a computer that satisfies the job requirements is available.

11.4.5. Shutting Down Computers

For the proper procedure to stop Job Scheduler service before shutting down or restarting remote computers, see “Using the AWR_JobScheduler Service”. Shutting down without stopping the service may result in a loss of data.

After a job is submitted to the scheduler node, you can shut down the user computer, including sleep and hibernate modes. The job continues to run on the remote computers, and the results are returned to the user computer when it is back online. You should transfer from user computer to the scheduler node all necessary files to run the job before shut down, or the simulation cannot proceed until the user computer is back online to complete file transfer. You can use the Job Monitor to verify file transfer status. After the job status changes from "Pending" to "Scheduled", all files have completed transfer and shut down may proceed.

11.4.6. Troubleshooting Remote Computing

This section provides solutions for potential problems that may occur.

Firewall Ports

The following ports are associated with file sharing and server message block (SMB) communications, and need to be open for remote simulation:

  • Microsoft file sharing SMB: User Datagram Protocol (UDP) ports from 135 through 139 and Transmission Control Protocol (TCP) ports from 135 through 139.

  • Direct-hosted SMB traffic without a network basic input/output system (NetBIOS): port 445 (TCP and UPD)

Verification Failure

User Computer Cannot Verify Scheduler Node

User computers cannot connect to scheduler nodes on a different network domain. Currently there is no way to transfer usernames and passwords from within the NI AWR Design Environment suite or the Job Scheduler interface. One work-around is to first create a shared folder on the scheduler node and then mount the shared folder on the user computer. Before trying to verify the scheduler node:

  1. On the user computer, open Windows Explorer. If the menu toolbar is not displayed, press the Alt key.

  2. Choose Tools > Map network drive. Map a folder on the scheduler node to a drive on the user computer using the username/password required for the scheduler node. Once mapped, verify that you can access the mapped folder.

Scheduler Node Can't Verify Compute Node

If the scheduler node and compute node are set up on a network without a domain controller, the scheduler node will not be able to verify the compute node. Without the domain, there is no concept of network user accounts/passwords. The AWR_JobScheduler service running under SYSTEM on the scheduler node is not recognized by SYSTEM on the compute node.

A solution is to move the service from SYSTEM to a user account on all remote machines. On a Windows 10 machine, the steps are as follows:

  • Stop the AWR_JobScheduler service if it is currently running.

  • Open the Services Window by starting the Task Manager and clicking the Services tab, then click the Services button at the bottom of the window. You can also locate this window by searching for "Services" from the Start menu.

  • In the list of services, right-click on AWR_JobScheduler and choose Properties.

  • In the AWR_JobScheduler Properties window, click the Log On tab.

  • Select This account and enter the account name and password.

  • Click OK until you exit the windows.

  • Restart the service.

Now, from the scheduler node, try to verify the compute node. If verification still fails, mount a shared folder from the compute node to the scheduler computer using the account and password of the user under which the service is running. The procedure is the same as that detailed in “User Computer Cannot Verify Scheduler Node”.

NOTE: When the AWR_JobScheduler service is running under a user account instead of SYSTEM, all Job Scheduler configuration files and log files are located in the Appdatauser folder of the user account instead of in the \Appdatacommon folder.

Log Files

It may be helpful to review the Job Scheduler log files on user and remote computers when there are problems. There is a script that automatically archives the log files and emails them to NI AWR Support. To run this script, choose Scripts > Configuration > Send_Logs.

On the user computer, logs are stored in the Appdatauser\Logs directory. To find the log files on the user computer:

  1. Choose Help > Show Files/Directories.

  2. In the Directories window, select the Logs folder.

  3. Save the logs by copying AWR_JobScheduler_x64_log.txt and MWOffice_ipclog.xml, or archiving the entire Logs folder into a .zip file. Note that restarting the NI AWR Design Environment program clears out MWOffice_ipclog.xml, so you should save these logs before restarting.

On remote computers, logs are stored in the \Appdatacommon\logs directory. To find the log files on the server computer:

  1. Choose Help > Show Files/Directories.

  2. In the Directories window, select the Appdatacommon folder and then the Logs subfolder.

  3. Save the log by copying AWR_JobScheduler_x64_log.txt or archiving the entire Logs folder to a .zip file.

Please send email to awr.support@ni.com if you would like to provide feedback on this article. Please make sure to include the article link in the email.

Legal and Trademark Notice