JobScheduler in a Nutshell

Software- und Organisations-Service Open Source JobScheduler JobScheduler in a Nutshell Product Overview Software- und Organisations-Service GmbH www.sos-berlin.com 2 Open Source JobScheduler Introduction to JobScheduler The JobScheduler is a workload automation product that is used to launch scheduling objects, such as jobs, job chains and orders, when time, file or calendar events occur. The JobScheduler provides solutions from simple to complex scheduling scenarios and is available with open source and commercial licenses. The JobScheduler has been under continual development since 2005 and has a mature status. The JobScheduler was recognized by Gartner Inc. with a placement in their Magic Quadrant for workload automation. 3 JobScheduler worldwide Open Source Users and Commercial Customers About us SOS GmbH was founded in 1983, we specialized in office automation, development of heterogeneous software programs and the provision of services and support. In 2005 SOS GmbH introduced Dual Licensing (Open Source & commercial licensing) for JobScheduler and in 2011 for YADE Managed File Transfer. Today SOS GmbH continues with the ongoing development of its open source products, the delivery of services and support to their customers

worldwide. Open source users in more than 100 countries 191 commercial customers in 27 countries 4 Contents Topics Architecture Components Platforms Security Use Cases Cross-Platform Scheduling File Watching File Transfer Features Interfaces Key Features High Availability Agent Cluster Master Cluster 5 Architecture JobScheduler Components More Information JobScheduler Master JobScheduler Universa l Agent JobScheduler Master processes objects, such as jobs, job chains, orders and schedules Job Scheduler Universal Agent execute jobs on any machines in the network User Interfaces User Interfaces: JOC Cockpit JOE Graphical Editor

JOC Cockpit: web-based interface for operating and monitoring JobScheduler in near real-time JOE: graphical editor for JobScheduler objects such as jobs, job chains, orders and schedules 6 JOC Cockpit / Web Service / Master / Agent Architecture: Components JOC Cockpit The JOC Cockpit is a user interface for job control with browsers Users access the Master using a Web Service that performs authentication and authorization optionally against an LDAP directory Interfaces The PowerShell Command Line Interface and External Applications use the same Web Service for access to a JobScheduler Master Authorization is available for individual requests to the JobScheduler Master Master / Agent The JobScheduler Master executes tasks and orchestrates Agents Agents are deployed on top of existing servers running the programs and scripts that should be scheduled User Interface for job control PowerShell CLI LDAP Directory Programming Interface Scripting Interface

Authentication and Authorization JobScheduler Master Agents JOC Cockpit User Interface External Applications JobScheduler Web Service JobScheduler Master JobScheduler Master Agents Agents Database Access Database Access Database Database Access 7 Platforms and Databases Architecture: Platforms More Information Supported Platforms and Databases Supported Platform s Supported Databas es JOC Cockpit & Web Service Windows and Linux

JobScheduler Master Windows and Linux JobScheduler Universal Agent Windows and Linux Solaris, AIX, HP-UX, Mac OS and any other platform that supports a Java Virtual Machine Databases Oracle, SQL Server MariaDB, MySQL and PostgreSQL 8 Supported Platforms Architecture: Platforms Cockpit / Web Service The JOC Cockpit and REST Web Service are available for Windows JobScheduler JOC Cockpit Windows JobScheduler Web Service Windows JobScheduler JOC Cockpit Linux and Linux JobScheduler Web Service Linux Database access Master / Agent JobScheduler Master is available for Windows and Linux JobScheduler Agents are available for any platform that supports JobScheduler

Master Windows Enables job execution: on JobScheduler Master instances on JobScheduler Agents for any platform JobScheduler Master Linux Database Works with supported databases including: Oracle SQL Server MariaDB/MySQL PostgreSQL a Java Virtual Machine Database The JobScheduler REST Web Service and Master use a database on any platform Jobs Jobs can be executed locally on the Master Jobs can be executed JobScheduler Agent Windows JobScheduler Agent HP-UX JobScheduler Agent Linux JobScheduler Agent Mac OS JobScheduler

Agent Solaris JobScheduler Agent Docker JobScheduler Agent AIX JobScheduler Agent Raspberry Pi Works with any platform that supports a Java Virtual Machine JobScheduler Agent ... any platform 9 JOC Cockpit / Web Service / Master / Agent Architecture: Security Users Network Zone with restricted user access Limited access that requires authentication Any connection to a Master is authenticated by the Web Service that can be configured to use LDAP Use of HTTPS for connections can be enforced Network Zone without user access Master and Agent instances are operated in this zone without direct user access The Master instances are accessed exclusively by the Web Service Agent instances are exclusively accessed by Master instances Network Zone with limited

user access HTTP / HTTPS JOC Cockpit User Interface HTTP / HTTPS PowerShell CLI LDAP Directory LDAP proprietary protocol External Applications Database proprietary protocol JobScheduler Web Service Network Zone without user access HTTP / HTTPS JobScheduler Master Agents HTTP / HTTPS HTTP / HTTPS HTTP / HTTPS JobScheduler Master JobScheduler Master Agents Agents Database proprietary protocol

Database proprietary protocol Database Database proprietary protocol 10 ld Wor l a e s R Case e s U Samples of Real World Use Cases Use Cases Cross-Platform Scheduling Processing with Agents across Platforms Agentless Scheduling across Platforms File Watching Watching incoming files on a Master Remote File Watching with Agents File Transfer Server-2-Server file transfer with YADE Push Files to Internet via Jump Host Pull Files from Internet via Jump Host 11 ld Wor l a e s R Case e s U Use Case The customer operates

servers with Windows, Linux, Solaris and AIX Jobs with dependencies have to be executed on the different servers Solution Remote execution with Universal Agents A JobScheduler Master for Windows optionally executes jobs locally and orchestrates four Agents Job dependency is configured with Master who contacts Agents to execute jobs Benefits JobScheduler covers all required platforms Job dependencies for multi-platform execution can be configured Single point of configuration & operation Zero configuration for Universal Agents and easy deployment to large clusters Processing with Agents across Platforms Use Case: Cross-Platform Scheduling Master contacts Agents to execute jobs JobScheduler Master Windows Master collects log output and execution results Sequence of jobs configured with the JobScheduler Master Job Chain Job Job 1 1 Job Chain

Job Job 2 2 JobScheduler Agent Linux Job Job 3 3 JobScheduler Agent Windows Job Job 4 4 JobScheduler Agent Solaris JobScheduler Agent AIX 12 Case Use ario Scen Use Case The customer operates servers with Windows, Linux, Solaris and AIX Jobs with dependencies have to be executed on the different servers Solution A JobScheduler Master for Windows executes jobs locally and orchestrates execution on three servers by SSH The job dependency is configured with the

Master who contacts Agents to execute jobs Benefits No software installation is required for job execution on existing SSH servers Job dependencies for multi-platform execution can be configured Single point of configuration and operation Agentless Scheduling across Platforms Use Case: Cross-Platform Scheduling JobScheduler Master Windows Master executes jobs by SSH Job Job 2 2 Master collects log output and execution results Job executed with Master Sequence of jobs configured with the JobScheduler Master Job Chain Job Job 1 1 Job Chain Job Job 3 3 SSH Server Linux Job Job 4

4 SSH Server Solaris SSH Server AIX 13 ld Wor l a e s R Case e s U Use Case Files have to be watched on different plat-forms, e.g. on Solaris Job starts shall be caused on Windows and Linux after arrival of a trigger file on Solaris Solution Agents watch for incoming files and send signals to the Master once a file arrives The Master is configured to know which workflows are to follow Benefits Remote file watching with Master & Agents is platform independent All configuration for the workflows are stored on the Master, the Agents execute jobs with zero configuration Remote file watching can be applied to any number of platforms and servers Watching incoming files on a Master

Use Case: File Watching Master receives Agent signal to start job chain JobScheduler Master Linux Master contacts Agents to execute jobs in a job chain Job Job 1 1 Agent signals arrival of incoming trigger files Master collects log output and execution results JobScheduler Agent Windows Job Job 2 2 JobScheduler Agent Solaris Watching for incoming files JobScheduler Agent Linux 14 Case Use ario Scen Use Case Files have to be watched on Solaris and sent to a

Linux server. After transfer the files are processed on Linux and a successor job on Windows is executed Solution Universal Agent watches for incoming files and sends signal to the Master once a file arrives The files are directly sent to a Linux server with a file transfer job on Solaris The Master is configured to know which workflows are to follow Benefits Remote file watching with Master and Agents is platform independent Configuration for the workflows are only required on the Master, the Universal Agents execute jobs with zero configuration Remote File Watching with Agents Use Case: File Watching Master receives Agent signal to continue job chain Agent signals arrival and transfer of incoming files Job Job 1 1 YADE Client YADE Client JITL JITL Job Job JobScheduler Agent Solaris Watching for

incoming files JobScheduler Master Linux Master contacts Agents to execute jobs in a job chain Job Job 2 2 Master collects log output and execution results JobScheduler Agent Linux Files to be send Job Job 3 3 JobScheduler Agent Windows 15 Managed File Transfer Use Case: File Transfer More Information JobScheduler Add-on YADE Managed FileTransfer Managed File Transfer JobScheduler provides different methods for managing the transfer of files JobScheduler Add-on YADE: JobScheduler component for Managed File Transfer YADE JITL job templates are pre-installed in the JobScheduler and can be called directly from the JOC Cockpit YADE Managed File Transfer: YADE Command Line Client is operated independently from

JobScheduler (standalone) Command Line Interface YADE Client CLI is executed by shell jobs YADE Managed File Transfer with JobScheduler YADE includes the components YADE JITL Jobs: job interface integrated with JobScheduler YADE Background Service: collects transfer protocols and history and adds them to the database YADE History Viewer: GUI to view history and transfer logs 16 ld Wor l a e s R Case e s U Server-to-Server File Transfer with YADE Use Case: File Transfer Use Case Files shall be transferred by using different transfer protocols for data sources and targets JobScheduler Master Solution A virtual file system approach is used by YADE. This allows to copy and move files between sources and targets with any given protocol Benefits Any combination of file transfer protocols is possible, e.g. FTP, FTPS, SFTP, HTTP, WebDAV No touchdown of files on JobScheduler

Master Transfers are performed in memory, resulting in very economical use of resources YADE YADE JITL JITL Job Job Push files with SFTP Pull files with HTTP Pull files with FTP Server A FTP Data Source Server C HTTP Data Source Push files with WebDAV Server D WebDAV Data Target Server B SFTP Data Target 17 ld Wor l a e s R

Case e s U Use Case Servers in the intranet have no internet access. Connection are only allowed from the intranet to the DMZ. From the DMZ transfers to the internet are allowed Solution YADE Client in the Intranet contacts YADE for DMZ to initiate a secure transfer of files from the intranet to the internet. YADE Client in intranet sends files to YADE in DMZ for temporary storing In a final step the YADE Client in DMZ pushes the files to the internet Benefits Files are stored temporarily in the DMZ and are removed after transfer No credential data are stored in the DMZ All configuration items are stored in the intranet Push Files to Internet via Jump Host Use Case: File Transfer DMZ Jump Host Intranet Source Host JobScheduler Master YADE YADE Client Client for for DMZ DMZ

1 1 YADE YADE Client Client JITL JITL Job Job Files to be send Internet Target Host Push Files 2 2 Push Files WebDAV, FTP, FTPS, SFTP Server Files are temporarily stored and removed after transfer Files have arrived 18 ld Wor l a e s R Case e s U Use Case Servers in the intranet have no internet access Connection are only

allowed from the intranet to the DMZ. From the DMZ transfers to the internet are allowed Solution The host in the DMZ acts as jump host for YADE YADE Client in intranet contacts YADE in DMZ to start a transfer and to pull files from the internet to the DMZ YADE Client in intranet pulls the files from the YADE in DMZ to the final destination in the intranet Benefits Files are stored temporarily in the DMZ and are removed after transfer No credential data are stored in the DMZ All configuration items are stored in the intranet Pull Files from Internet via Jump Host Use Case: File Transfer DMZ Jump Host Intranet Target Host JobScheduler Master Internet Source Host YADE YADE Client Client for for DMZ DMZ 1 1 Contact YADE DMZ

2 2 Initiate file transfer HTTP, WebDAV, FTP, FTPS, SFTP Server YADE YADE Client Client JITL JITL Job Job 3 3 4 4 Pull files from Internet Pull files from DMZ Files are temporarily stored and removed after transfer Files for transfer 19 Interfaces & Key Features Features Interfaces User Interfaces Programming Interfaces Key Features

Command Line Operation Start Times Logging and Scripting Jobs, Job Chains and Orders Error Handling and Event Handling Central Configuration Directory Monitoring and File Watching more 20 JOC Cockpit and JOE Object Editor Interfaces and Key Features: User Interfaces More Information JobScheduler Operations Center: JOC Cockpit JOE Object Editor JobScheduler Operations Center: JOC Cockpit Web-based GUI provides overview for all relevant information on objects, starting and monitoring jobs, job chains and orders Complete overview for the JobScheduler Master status including Cluster, healthy and unhealthy Agents Automated updates in near real-time JOE Object Editor Manage JobScheduler Objects 21 JOC Cockpit - Dashboard Interfaces and Key Features: User Interfaces Dashboard The Dashboard offers a comprehensive overview of most relevant information in the form of widgets Information in the Dashboard is updated automatically in near real-time The Dashboard shows the JobScheduler Master status including cluster information

The Agent overview shows healthy and unhealthy Agent Clusters The Dashboard is a starting point to navigate to objects of interest, e.g. failed orders, suspended orders, late orders etc. 22 JOE Object Editor - Job Management Interfaces and Key Features: User Interfaces JOE Job Editor JOE is used to create, configure and manage JobScheduler objects such as jobs, job chains, orders, schedules, process classes and locks JOE reads and writes XML files for objects to so-called hot folders where they are loaded by JobScheduler JOE can be operated directly in the scheduling environment or on any computer that supports a Java Virtual Machine 23 REST Web Service Interface Interfaces and Key Features: Programming Interfaces JobScheduler Programming Interfaces JobScheduler REST Web Service Permissions JobScheduler REST Web Service (XML) The JobScheduler comes with a powerful REST Web Service interface for access by external applications This interface is used for tasks such as adding orders, adding events

and retrieving job and order status information Access to the REST Web Service is restricted by permissions JOC Cockpit JOC Cockpit REST Web Service Permissions JOC Cockpit REST Web Service (JSON) The JOC Cockpit comes with a REST Web Service interface for JSON This interface is used for access to the job inventory and the job history as well as individual objects such as jobs, job chains and orders and provides current status information Access to the REST Web Service is restricted by permissions 24 Job Application Programming Interface Interfaces and Key Features: Programming Interfaces More Information API Interface Reference Documentation Areas of Operation The JobScheduler comes with an extensive set of objects and methods for implementation with jobs and monitors. Such implementations often make use of the API Interface to check and manipulate JobScheduler objects, e.g. to add orders, to modify the order and job status The API is available for jobs and monitor scripts that implement conditions for job execution Language Support The API is available for scripted jobs, e.g. in the languages JavaScript, PowerShell, VBScript The API can be used from Java jobs 25 Command Line Interface Interfaces and Key Features: Programming Interfaces More Information Command Line Operation PowerShell Command Line Interface Shell Command Line Operation The JobScheduler can be operated from the command line, allowing a

wide range of operations to be carried out by an external application: checking the JobScheduler status controlling the status of jobs, job chains and orders adding orders to job chains adding events for event handling PowerShell Command Line Interface The JobScheduler Command Line Interface (JCLI) can be used to control JobScheduler instances (start, stop, status) and job-related objects such as jobs, job chains, orders, tasks Areas of Operation include to work as a replacement for command scripts, e.g. to start JobScheduler and to raise events provide bulk operations on jobs, job chains and orders schedule jobs and orders manage Agents, e.g. to check the Agent status 26 Key Features (1/4) Interfaces and Key Features More Information Start Times Logging Scripting Start Times Time events are one way of starting jobs, job chains and orders, with Start Times being set for a predefined point in time such as time of day, weekday, day of month etc. Logging The JobScheduler creates a number of logs to provide specific information about jobs, job chains, orders, tasks and JobScheduler installation and operation Scripting Scripting interface that allows execution of scripts in languages such as JavaScript, etc. 27 Key Features (2/4) Interfaces and Key Features More Information Jobs, Job Chains and O rders Error Handling Event Handling

Jobs, Job Chains and Orders The JobScheduler's unique job and order concept includes the organization of jobs into job chains and the use of dependencies Jobs: basic unit for the processing of executable files Job Chains: assembly line on which job nodes are passed sequentially Orders: represent triggers that will cause a job chain to be started Error Handling A number of methods for error handling are available, e.g. stop a job, suspend an order, setback an order and more Event Handling Mechanism for implementing complex dependencies between jobs or between jobs and external events 28 Key Features (3/4) Interfaces and Key Features More Information Central Configuration Directory Monitoring an d File Watching Resource Contention M anager Central Configuration Central configuration allows the efficient distribution of configuration files from a central source to distributed JobScheduler instances by use of a Supervisor JobScheduler Directory Monitoring and File Watching Two methods to start jobs and job chains automatically based on the arrival of incoming files Resource Contention Manager Process classes and locks are used to manage the use of resources such as databases or printers 29 Key Features (4/4) Interfaces and Key Features More Information Notifications Monitoring Localization

Notifications JobScheduler comes with its own mail client which it can use to send notifying e-mails in the event of, for example, jobs ending in error Monitoring JobScheduler can be monitored by System Monitors. Such products include e.g. HP OpenView, Microsoft SCOM, Nagios, op5, Opsview, Zabbix etc. Localization Language files are provided for the installation of JobScheduler and for most of its operating interfaces. The default language is English Installation Languages: English and German Interface Languages: English, French, German, Japanese 30 Implementation for High Availability and Fault Tolerance High Availability: Overview More Information Cluster Operation Cluster Operation Master and Agents are available for Active and Passive Clusters Fault Tolerance Master / Agent Resilien ce Master / Agent Redund ancy Fault Tolerance Resilience and Redundancy provide high availability of JobScheduler for a number of outage scenarios Recovery Strategies Master / Agent Resilience measures for operational robustness, such as Master/Agent Reconciliation, Master Service Recovery, Database Service Recovery Master / Agent Redundancy includes a number of architecture decisions for Master Clusters and Agent Clusters Recovery Strategies provide an overview of means how to restore the scheduling service 31

Agent Cluster: Fixed Priority and Round-Robin Scheduling High Availability: Agent Cluster Master/Agent Platforms JobScheduler Master is available for Windows and Linux JobScheduler Agents are available for any platform that supports a Java Virtual Machine Agent Cluster Agents can be configured to work in a Cluster Fixed Priority Scheduling JobScheduler Master selects the first available Agent from a Cluster for job execution Should an Agent not be available then the next available Agent is used Round-Robin Scheduling JobScheduler Master switches the Agent used for each job execution Should an Agent not be available the next available Agent is used access database Database JobScheduler Master Windows JobScheduler Master Linux Works with the JobScheduler Master platforms: Windows Linux Enables job execution on JobScheduler Master instances on JobScheduler Agents for any platform

JobScheduler Agent ... any platform JobScheduler Agent ... any platform JobScheduler Agent ... any platform Works with any platform that supports a Java Virtual Machine JobScheduler Agent ... any platform JobScheduler Agent ... any platform Works with all supported databases: Oracle SQL Server DB2 MariaDB MySQL PostgreSQL JobScheduler Agent ... any platform Works with any platform that supports a Java Virtual Machine 32 JobScheduler Master Passive Cluster High Availability: Master Cluster Passive Cluster Primary and Backup

JobScheduler Master use the same database Primary JobScheduler Master is monitored by its failover instance Failover instance operates in stand-by mode All connections to servers use the SSH protocol SSH Connections access configuration access configuration access database Database Primary JobScheduler JS1 JITL Jobs Requires a JVM per task Memory resources SSH Client No pre- and postprocessing No substitution of parameters in script files Script files have to be provided on the target system Share or Supervisor JobScheduler passive stand-by Backup JobScheduler JS2 connect via ssh connect via ssh Agent

JobScheduler SVR1 Agent JobScheduler SVR2 Agent JobScheduler SVR3 33 JobScheduler Master Active Cluster High Availability: Master Cluster Master/Agent Active Cluster JobScheduler Cluster members use the same database JobSchedulers Cluster members share the workload of jobs All instances operate in active mode All Cluster instances use Agents to execute jobs on remote servers Job Execution Jobs are executed locally with the JobScheduler Agent. No central resources required for job execution Pre-/post-processing Use of JITL Jobs or script files with parameter substitution access configuration Share or Supervisor JobScheduler access configuration access configuration access database

Database Active Cluster JobScheduler JS1 connect via JobScheduler protocol Agent JobScheduler SVR1 Active Cluster JobScheduler JS2 connect via JobScheduler protocol Agent JobScheduler SVR2 Active Cluster JobScheduler JS3 connect via JobScheduler protocol Agent JobScheduler SVR3 34 JobScheduler Supervisor for Passive and Active Cluster High Availability: Supervisor JobScheduler Master/Agent Active Cluster JobScheduler Cluster members use the same database JobScheduler Cluster members share the workload of jobs All instances operate in active mode

All Cluster instances use Agents to execute jobs on remote servers Supervisor JobScheduler Distribute configuration to Cluster JobScheduler instances distribute configuration Supervisor JobScheduler JS4 Share or File System distribute configuration distribute configuration access database Database Active Cluster JobScheduler JS1 connect via JobScheduler protocol Agent JobScheduler SVR1 Active Cluster JobScheduler JS2 connect via JobScheduler protocol Agent JobScheduler SVR2 Active Cluster JobScheduler

JS3 connect via JobScheduler protocol Agent JobScheduler SVR3 35 Open Source JobScheduler JobScheduler in a Nutshell Software- und OrganisationsService GmbH Giesebrechtstr. 15 D-10629 Berlin Germany [email protected] www.sos-berlin.com Questions? Comments? Feedback?

Recently Viewed Presentations

  • Blending Quotations

    Blending Quotations

    The Basics: Dropped Quotes Always integrate quotations into your text. NEVER just "drop" a quotation in your writing! In other words, don't let a piece of textual evidence stand alone as its own sentence (unless it's multiple sentences long). Use...
  • Git A distributed version control system 3/1/20 Version

    Git A distributed version control system 3/1/20 Version

    Download the current HEAD from the central repository. Make your changes. Commit your changes to your local repository. Check to make sure someone else on your team hasn't updated the central repository since you got it. Upload your changes to...
  • Bez nadpisu

    Bez nadpisu

    Medico-legal investigation of deaths Death is not simply the state of life absence, but is a process of life fading in a viable organism, which is characterized by irreversible cessation of vital functions and definitive stopping of metabolism
  • Warm-Up What is Manifest Destiny mean? **Think back

    Warm-Up What is Manifest Destiny mean? **Think back

    Login in glogster- setup your account. Make sure you to use your name and for your nickname perhaps your name with a number. Start rough draft; try to get it mostly completed, if not completed. Plan on who you are...
  • British North America: Colonial Governments and Imperial Policy

    British North America: Colonial Governments and Imperial Policy

    British North America: Colonial Governments and Imperial Policy British Colonial Trade Regulations During the reign of James II, economic restrictions were imposed on the colonists to foster economic dependence.
  • Bounded rationality, biases and superstitions

    Bounded rationality, biases and superstitions

    Heuristics, pragmatism and naturalism Konrad Talmont-Kaminski, KLI & UMCS
  • Newtonian Mechanics

    Newtonian Mechanics

    Types of Forces: Restoring Force / Spring Force. There are many objects in nature which take the form of a spring. Stretching a rubber band or a spring requires a force. Another example is an atom within the lattice of...
  • Overview of WMO Information System (WIS)

    Overview of WMO Information System (WIS)

    The WMO Information System (WIS) WIS-Overview 101. 17 November 2013. ... St Denis. Pretoria. Mogadiscio. Niamey. Dakar. Nairobi. ... The primary interface for interoperability between WIS and GEOSS occurs at the GEOSS Data Broker which acts as a Clearinghouse.