Software- und Organisations-Service Open Source JobScheduler JobScheduler in a Nutshell Product Overview Software- und Organisations-Service GmbH www.sos-berlin.com 2 Open Source JobScheduler Introduction to JobScheduler The JobScheduler is a workload automation product that is used to launch scheduling objects, such as jobs, job chains and orders, when time, file or calendar events occur. The JobScheduler provides solutions from simple to complex scheduling scenarios and is available with open source and commercial licenses. The JobScheduler has been under continual development since 2005 and has a mature status. The JobScheduler was recognized by Gartner Inc. with a placement in their Magic Quadrant for workload automation. 3 JobScheduler worldwide Open Source Users and Commercial Customers About us SOS GmbH was founded in 1983, we specialized in office automation, development of heterogeneous software programs and the provision of services and support. In 2005 SOS GmbH introduced Dual Licensing (Open Source & commercial licensing) for JobScheduler and in 2011 for YADE Managed File Transfer. Today SOS GmbH continues with the ongoing development of its open source products, the delivery of services and support to their customers
worldwide. Open source users in more than 100 countries 191 commercial customers in 27 countries 4 Contents Topics Architecture Components Platforms Security Use Cases Cross-Platform Scheduling File Watching File Transfer Features Interfaces Key Features High Availability Agent Cluster Master Cluster 5 Architecture JobScheduler Components More Information JobScheduler Master JobScheduler Universa l Agent JobScheduler Master processes objects, such as jobs, job chains, orders and schedules Job Scheduler Universal Agent execute jobs on any machines in the network User Interfaces User Interfaces: JOC Cockpit JOE Graphical Editor
JOC Cockpit: web-based interface for operating and monitoring JobScheduler in near real-time JOE: graphical editor for JobScheduler objects such as jobs, job chains, orders and schedules 6 JOC Cockpit / Web Service / Master / Agent Architecture: Components JOC Cockpit The JOC Cockpit is a user interface for job control with browsers Users access the Master using a Web Service that performs authentication and authorization optionally against an LDAP directory Interfaces The PowerShell Command Line Interface and External Applications use the same Web Service for access to a JobScheduler Master Authorization is available for individual requests to the JobScheduler Master Master / Agent The JobScheduler Master executes tasks and orchestrates Agents Agents are deployed on top of existing servers running the programs and scripts that should be scheduled User Interface for job control PowerShell CLI LDAP Directory Programming Interface Scripting Interface
Authentication and Authorization JobScheduler Master Agents JOC Cockpit User Interface External Applications JobScheduler Web Service JobScheduler Master JobScheduler Master Agents Agents Database Access Database Access Database Database Access 7 Platforms and Databases Architecture: Platforms More Information Supported Platforms and Databases Supported Platform s Supported Databas es JOC Cockpit & Web Service Windows and Linux
JobScheduler Master Windows and Linux JobScheduler Universal Agent Windows and Linux Solaris, AIX, HP-UX, Mac OS and any other platform that supports a Java Virtual Machine Databases Oracle, SQL Server MariaDB, MySQL and PostgreSQL 8 Supported Platforms Architecture: Platforms Cockpit / Web Service The JOC Cockpit and REST Web Service are available for Windows JobScheduler JOC Cockpit Windows JobScheduler Web Service Windows JobScheduler JOC Cockpit Linux and Linux JobScheduler Web Service Linux Database access Master / Agent JobScheduler Master is available for Windows and Linux JobScheduler Agents are available for any platform that supports JobScheduler
Master Windows Enables job execution: on JobScheduler Master instances on JobScheduler Agents for any platform JobScheduler Master Linux Database Works with supported databases including: Oracle SQL Server MariaDB/MySQL PostgreSQL a Java Virtual Machine Database The JobScheduler REST Web Service and Master use a database on any platform Jobs Jobs can be executed locally on the Master Jobs can be executed JobScheduler Agent Windows JobScheduler Agent HP-UX JobScheduler Agent Linux JobScheduler Agent Mac OS JobScheduler
Agent Solaris JobScheduler Agent Docker JobScheduler Agent AIX JobScheduler Agent Raspberry Pi Works with any platform that supports a Java Virtual Machine JobScheduler Agent ... any platform 9 JOC Cockpit / Web Service / Master / Agent Architecture: Security Users Network Zone with restricted user access Limited access that requires authentication Any connection to a Master is authenticated by the Web Service that can be configured to use LDAP Use of HTTPS for connections can be enforced Network Zone without user access Master and Agent instances are operated in this zone without direct user access The Master instances are accessed exclusively by the Web Service Agent instances are exclusively accessed by Master instances Network Zone with limited
user access HTTP / HTTPS JOC Cockpit User Interface HTTP / HTTPS PowerShell CLI LDAP Directory LDAP proprietary protocol External Applications Database proprietary protocol JobScheduler Web Service Network Zone without user access HTTP / HTTPS JobScheduler Master Agents HTTP / HTTPS HTTP / HTTPS HTTP / HTTPS JobScheduler Master JobScheduler Master Agents Agents Database proprietary protocol
Database proprietary protocol Database Database proprietary protocol 10 ld Wor l a e s R Case e s U Samples of Real World Use Cases Use Cases Cross-Platform Scheduling Processing with Agents across Platforms Agentless Scheduling across Platforms File Watching Watching incoming files on a Master Remote File Watching with Agents File Transfer Server-2-Server file transfer with YADE Push Files to Internet via Jump Host Pull Files from Internet via Jump Host 11 ld Wor l a e s R Case e s U Use Case The customer operates
servers with Windows, Linux, Solaris and AIX Jobs with dependencies have to be executed on the different servers Solution Remote execution with Universal Agents A JobScheduler Master for Windows optionally executes jobs locally and orchestrates four Agents Job dependency is configured with Master who contacts Agents to execute jobs Benefits JobScheduler covers all required platforms Job dependencies for multi-platform execution can be configured Single point of configuration & operation Zero configuration for Universal Agents and easy deployment to large clusters Processing with Agents across Platforms Use Case: Cross-Platform Scheduling Master contacts Agents to execute jobs JobScheduler Master Windows Master collects log output and execution results Sequence of jobs configured with the JobScheduler Master Job Chain Job Job 1 1 Job Chain
Job Job 2 2 JobScheduler Agent Linux Job Job 3 3 JobScheduler Agent Windows Job Job 4 4 JobScheduler Agent Solaris JobScheduler Agent AIX 12 Case Use ario Scen Use Case The customer operates servers with Windows, Linux, Solaris and AIX Jobs with dependencies have to be executed on the different servers Solution A JobScheduler Master for Windows executes jobs locally and orchestrates execution on three servers by SSH The job dependency is configured with the
Master who contacts Agents to execute jobs Benefits No software installation is required for job execution on existing SSH servers Job dependencies for multi-platform execution can be configured Single point of configuration and operation Agentless Scheduling across Platforms Use Case: Cross-Platform Scheduling JobScheduler Master Windows Master executes jobs by SSH Job Job 2 2 Master collects log output and execution results Job executed with Master Sequence of jobs configured with the JobScheduler Master Job Chain Job Job 1 1 Job Chain Job Job 3 3 SSH Server Linux Job Job 4
4 SSH Server Solaris SSH Server AIX 13 ld Wor l a e s R Case e s U Use Case Files have to be watched on different plat-forms, e.g. on Solaris Job starts shall be caused on Windows and Linux after arrival of a trigger file on Solaris Solution Agents watch for incoming files and send signals to the Master once a file arrives The Master is configured to know which workflows are to follow Benefits Remote file watching with Master & Agents is platform independent All configuration for the workflows are stored on the Master, the Agents execute jobs with zero configuration Remote file watching can be applied to any number of platforms and servers Watching incoming files on a Master
Use Case: File Watching Master receives Agent signal to start job chain JobScheduler Master Linux Master contacts Agents to execute jobs in a job chain Job Job 1 1 Agent signals arrival of incoming trigger files Master collects log output and execution results JobScheduler Agent Windows Job Job 2 2 JobScheduler Agent Solaris Watching for incoming files JobScheduler Agent Linux 14 Case Use ario Scen Use Case Files have to be watched on Solaris and sent to a
Linux server. After transfer the files are processed on Linux and a successor job on Windows is executed Solution Universal Agent watches for incoming files and sends signal to the Master once a file arrives The files are directly sent to a Linux server with a file transfer job on Solaris The Master is configured to know which workflows are to follow Benefits Remote file watching with Master and Agents is platform independent Configuration for the workflows are only required on the Master, the Universal Agents execute jobs with zero configuration Remote File Watching with Agents Use Case: File Watching Master receives Agent signal to continue job chain Agent signals arrival and transfer of incoming files Job Job 1 1 YADE Client YADE Client JITL JITL Job Job JobScheduler Agent Solaris Watching for
incoming files JobScheduler Master Linux Master contacts Agents to execute jobs in a job chain Job Job 2 2 Master collects log output and execution results JobScheduler Agent Linux Files to be send Job Job 3 3 JobScheduler Agent Windows 15 Managed File Transfer Use Case: File Transfer More Information JobScheduler Add-on YADE Managed FileTransfer Managed File Transfer JobScheduler provides different methods for managing the transfer of files JobScheduler Add-on YADE: JobScheduler component for Managed File Transfer YADE JITL job templates are pre-installed in the JobScheduler and can be called directly from the JOC Cockpit YADE Managed File Transfer: YADE Command Line Client is operated independently from
JobScheduler (standalone) Command Line Interface YADE Client CLI is executed by shell jobs YADE Managed File Transfer with JobScheduler YADE includes the components YADE JITL Jobs: job interface integrated with JobScheduler YADE Background Service: collects transfer protocols and history and adds them to the database YADE History Viewer: GUI to view history and transfer logs 16 ld Wor l a e s R Case e s U Server-to-Server File Transfer with YADE Use Case: File Transfer Use Case Files shall be transferred by using different transfer protocols for data sources and targets JobScheduler Master Solution A virtual file system approach is used by YADE. This allows to copy and move files between sources and targets with any given protocol Benefits Any combination of file transfer protocols is possible, e.g. FTP, FTPS, SFTP, HTTP, WebDAV No touchdown of files on JobScheduler
Master Transfers are performed in memory, resulting in very economical use of resources YADE YADE JITL JITL Job Job Push files with SFTP Pull files with HTTP Pull files with FTP Server A FTP Data Source Server C HTTP Data Source Push files with WebDAV Server D WebDAV Data Target Server B SFTP Data Target 17 ld Wor l a e s R
Case e s U Use Case Servers in the intranet have no internet access. Connection are only allowed from the intranet to the DMZ. From the DMZ transfers to the internet are allowed Solution YADE Client in the Intranet contacts YADE for DMZ to initiate a secure transfer of files from the intranet to the internet. YADE Client in intranet sends files to YADE in DMZ for temporary storing In a final step the YADE Client in DMZ pushes the files to the internet Benefits Files are stored temporarily in the DMZ and are removed after transfer No credential data are stored in the DMZ All configuration items are stored in the intranet Push Files to Internet via Jump Host Use Case: File Transfer DMZ Jump Host Intranet Source Host JobScheduler Master YADE YADE Client Client for for DMZ DMZ
1 1 YADE YADE Client Client JITL JITL Job Job Files to be send Internet Target Host Push Files 2 2 Push Files WebDAV, FTP, FTPS, SFTP Server Files are temporarily stored and removed after transfer Files have arrived 18 ld Wor l a e s R Case e s U Use Case Servers in the intranet have no internet access Connection are only
allowed from the intranet to the DMZ. From the DMZ transfers to the internet are allowed Solution The host in the DMZ acts as jump host for YADE YADE Client in intranet contacts YADE in DMZ to start a transfer and to pull files from the internet to the DMZ YADE Client in intranet pulls the files from the YADE in DMZ to the final destination in the intranet Benefits Files are stored temporarily in the DMZ and are removed after transfer No credential data are stored in the DMZ All configuration items are stored in the intranet Pull Files from Internet via Jump Host Use Case: File Transfer DMZ Jump Host Intranet Target Host JobScheduler Master Internet Source Host YADE YADE Client Client for for DMZ DMZ 1 1 Contact YADE DMZ
2 2 Initiate file transfer HTTP, WebDAV, FTP, FTPS, SFTP Server YADE YADE Client Client JITL JITL Job Job 3 3 4 4 Pull files from Internet Pull files from DMZ Files are temporarily stored and removed after transfer Files for transfer 19 Interfaces & Key Features Features Interfaces User Interfaces Programming Interfaces Key Features
Command Line Operation Start Times Logging and Scripting Jobs, Job Chains and Orders Error Handling and Event Handling Central Configuration Directory Monitoring and File Watching more 20 JOC Cockpit and JOE Object Editor Interfaces and Key Features: User Interfaces More Information JobScheduler Operations Center: JOC Cockpit JOE Object Editor JobScheduler Operations Center: JOC Cockpit Web-based GUI provides overview for all relevant information on objects, starting and monitoring jobs, job chains and orders Complete overview for the JobScheduler Master status including Cluster, healthy and unhealthy Agents Automated updates in near real-time JOE Object Editor Manage JobScheduler Objects 21 JOC Cockpit - Dashboard Interfaces and Key Features: User Interfaces Dashboard The Dashboard offers a comprehensive overview of most relevant information in the form of widgets Information in the Dashboard is updated automatically in near real-time The Dashboard shows the JobScheduler Master status including cluster information
The Agent overview shows healthy and unhealthy Agent Clusters The Dashboard is a starting point to navigate to objects of interest, e.g. failed orders, suspended orders, late orders etc. 22 JOE Object Editor - Job Management Interfaces and Key Features: User Interfaces JOE Job Editor JOE is used to create, configure and manage JobScheduler objects such as jobs, job chains, orders, schedules, process classes and locks JOE reads and writes XML files for objects to so-called hot folders where they are loaded by JobScheduler JOE can be operated directly in the scheduling environment or on any computer that supports a Java Virtual Machine 23 REST Web Service Interface Interfaces and Key Features: Programming Interfaces JobScheduler Programming Interfaces JobScheduler REST Web Service Permissions JobScheduler REST Web Service (XML) The JobScheduler comes with a powerful REST Web Service interface for access by external applications This interface is used for tasks such as adding orders, adding events
Jobs, Job Chains and Orders The JobScheduler's unique job and order concept includes the organization of jobs into job chains and the use of dependencies Jobs: basic unit for the processing of executable files Job Chains: assembly line on which job nodes are passed sequentially Orders: represent triggers that will cause a job chain to be started Error Handling A number of methods for error handling are available, e.g. stop a job, suspend an order, setback an order and more Event Handling Mechanism for implementing complex dependencies between jobs or between jobs and external events 28 Key Features (3/4) Interfaces and Key Features More Information Central Configuration Directory Monitoring an d File Watching Resource Contention M anager Central Configuration Central configuration allows the efficient distribution of configuration files from a central source to distributed JobScheduler instances by use of a Supervisor JobScheduler Directory Monitoring and File Watching Two methods to start jobs and job chains automatically based on the arrival of incoming files Resource Contention Manager Process classes and locks are used to manage the use of resources such as databases or printers 29 Key Features (4/4) Interfaces and Key Features More Information Notifications Monitoring Localization
Notifications JobScheduler comes with its own mail client which it can use to send notifying e-mails in the event of, for example, jobs ending in error Monitoring JobScheduler can be monitored by System Monitors. Such products include e.g. HP OpenView, Microsoft SCOM, Nagios, op5, Opsview, Zabbix etc. Localization Language files are provided for the installation of JobScheduler and for most of its operating interfaces. The default language is English Installation Languages: English and German Interface Languages: English, French, German, Japanese 30 Implementation for High Availability and Fault Tolerance High Availability: Overview More Information Cluster Operation Cluster Operation Master and Agents are available for Active and Passive Clusters Fault Tolerance Master / Agent Resilien ce Master / Agent Redund ancy Fault Tolerance Resilience and Redundancy provide high availability of JobScheduler for a number of outage scenarios Recovery Strategies Master / Agent Resilience measures for operational robustness, such as Master/Agent Reconciliation, Master Service Recovery, Database Service Recovery Master / Agent Redundancy includes a number of architecture decisions for Master Clusters and Agent Clusters Recovery Strategies provide an overview of means how to restore the scheduling service 31
Agent Cluster: Fixed Priority and Round-Robin Scheduling High Availability: Agent Cluster Master/Agent Platforms JobScheduler Master is available for Windows and Linux JobScheduler Agents are available for any platform that supports a Java Virtual Machine Agent Cluster Agents can be configured to work in a Cluster Fixed Priority Scheduling JobScheduler Master selects the first available Agent from a Cluster for job execution Should an Agent not be available then the next available Agent is used Round-Robin Scheduling JobScheduler Master switches the Agent used for each job execution Should an Agent not be available the next available Agent is used access database Database JobScheduler Master Windows JobScheduler Master Linux Works with the JobScheduler Master platforms: Windows Linux Enables job execution on JobScheduler Master instances on JobScheduler Agents for any platform
JobScheduler Agent ... any platform JobScheduler Agent ... any platform JobScheduler Agent ... any platform Works with any platform that supports a Java Virtual Machine JobScheduler Agent ... any platform JobScheduler Agent ... any platform Works with all supported databases: Oracle SQL Server DB2 MariaDB MySQL PostgreSQL JobScheduler Agent ... any platform Works with any platform that supports a Java Virtual Machine 32 JobScheduler Master Passive Cluster High Availability: Master Cluster Passive Cluster Primary and Backup
JobScheduler Master use the same database Primary JobScheduler Master is monitored by its failover instance Failover instance operates in stand-by mode All connections to servers use the SSH protocol SSH Connections access configuration access configuration access database Database Primary JobScheduler JS1 JITL Jobs Requires a JVM per task Memory resources SSH Client No pre- and postprocessing No substitution of parameters in script files Script files have to be provided on the target system Share or Supervisor JobScheduler passive stand-by Backup JobScheduler JS2 connect via ssh connect via ssh Agent
JobScheduler SVR1 Agent JobScheduler SVR2 Agent JobScheduler SVR3 33 JobScheduler Master Active Cluster High Availability: Master Cluster Master/Agent Active Cluster JobScheduler Cluster members use the same database JobSchedulers Cluster members share the workload of jobs All instances operate in active mode All Cluster instances use Agents to execute jobs on remote servers Job Execution Jobs are executed locally with the JobScheduler Agent. No central resources required for job execution Pre-/post-processing Use of JITL Jobs or script files with parameter substitution access configuration Share or Supervisor JobScheduler access configuration access configuration access database
Database Active Cluster JobScheduler JS1 connect via JobScheduler protocol Agent JobScheduler SVR1 Active Cluster JobScheduler JS2 connect via JobScheduler protocol Agent JobScheduler SVR2 Active Cluster JobScheduler JS3 connect via JobScheduler protocol Agent JobScheduler SVR3 34 JobScheduler Supervisor for Passive and Active Cluster High Availability: Supervisor JobScheduler Master/Agent Active Cluster JobScheduler Cluster members use the same database JobScheduler Cluster members share the workload of jobs All instances operate in active mode
All Cluster instances use Agents to execute jobs on remote servers Supervisor JobScheduler Distribute configuration to Cluster JobScheduler instances distribute configuration Supervisor JobScheduler JS4 Share or File System distribute configuration distribute configuration access database Database Active Cluster JobScheduler JS1 connect via JobScheduler protocol Agent JobScheduler SVR1 Active Cluster JobScheduler JS2 connect via JobScheduler protocol Agent JobScheduler SVR2 Active Cluster JobScheduler
JS3 connect via JobScheduler protocol Agent JobScheduler SVR3 35 Open Source JobScheduler JobScheduler in a Nutshell Software- und OrganisationsService GmbH Giesebrechtstr. 15 D-10629 Berlin Germany [email protected] www.sos-berlin.com Questions? Comments? Feedback?
The Basics: Dropped Quotes Always integrate quotations into your text. NEVER just "drop" a quotation in your writing! In other words, don't let a piece of textual evidence stand alone as its own sentence (unless it's multiple sentences long). Use...
Download the current HEAD from the central repository. Make your changes. Commit your changes to your local repository. Check to make sure someone else on your team hasn't updated the central repository since you got it. Upload your changes to...
Medico-legal investigation of deaths Death is not simply the state of life absence, but is a process of life fading in a viable organism, which is characterized by irreversible cessation of vital functions and definitive stopping of metabolism
Login in glogster- setup your account. Make sure you to use your name and for your nickname perhaps your name with a number. Start rough draft; try to get it mostly completed, if not completed. Plan on who you are...
British North America: Colonial Governments and Imperial Policy British Colonial Trade Regulations During the reign of James II, economic restrictions were imposed on the colonists to foster economic dependence.
Types of Forces: Restoring Force / Spring Force. There are many objects in nature which take the form of a spring. Stretching a rubber band or a spring requires a force. Another example is an atom within the lattice of...
The WMO Information System (WIS) WIS-Overview 101. 17 November 2013. ... St Denis. Pretoria. Mogadiscio. Niamey. Dakar. Nairobi. ... The primary interface for interoperability between WIS and GEOSS occurs at the GEOSS Data Broker which acts as a Clearinghouse.
Ready to download the document? Go ahead and hit continue!