Programming Guide for Lambari and Pacu versions



This tutorial is divided into four parts: High performance computing (HPC) services, task cost (TC) services, Internet of things (IoT) services, and GPU services. The four pillars form the JCL API. GPU computing will be part of future JCL services, so it is not considered in this document.

IMPORTANT: JCL installation guide must be considered before starting the JCL programming guide. JCL IoT version requires several installations details.


1. JCL HPC API

At Examples session in JCL site you will find some examples to download. The first example (jcl_examples_Month_Year.zip) introduces our JCL programming course. First, you need to import the Java project into your IDE (we use Eclipse IDE). The JCL Jar file can be found at jcl_JARS folder (a complete guide about how to include a Jar file to your project can be found in our installation guide).

Figure 1. A simple Java Project using JCL


Before we start jcl_examples Java project explanations, we must introduce some preliminaries. JCL has two versions: Lambari for multi-core computer architectures and Pacu for multi-computer ones. JCL is portable, so you write once and run your code using any JCL version. For that, you need to configure the JCL config file (detailed in JCL installation guide). Figure 2 illustrates JCL config file, where the property distOrParell can be true or false. True is when adopting Pacu and false when Lambari must be used. There are several other ways to start a version of JCL, but all of them are done programmatically. The unique way to guarantee JCL portability is configuring the JCL config file, as Figure 2 illustrates.

Figure 2. JCL Config file used to set JCL version


The first example used in this tutorial is the application named appl1.java. Figure 3 illustrates three ways to instantiate JCL. The first way is the unique that guarantees code portability between Lambari and Pacu versions. The second way obtains a specific Lambari instance and the third way a Pacu instance. All the returned JCL instances are of JCL_facade type, representingthe JCL API. The local variables jcl, jcl1 and jcl2 are ready to be used in the code.

Figure 3. Three ways to instantiate JCL

 At line 33 we have a class registration. Since JCL can execute any Java class, we must register it before calling its methods. The JCL method named register requires the Java class to be registered and a nickname for it. In our example, the nickname is identical to the class name (“UserServices”), but users can adopt any nickname. Complex modules, with several .class files, are also feasible in JCL, e.g., we can register Jar files with dependencies. During this tutorial, we will learn how to do that. JCL returns true if registered of false otherwise.

There is only one JCL instance per java project, so you can call JCL_FacadeImpl.getInstance() or its variants how many times you want and where you want. In our example, UserServices.class is part of jcl_examples Java project. This class contains methods for arithmetic-geometric progressions, Fibonacci, sorting algorithm and more.

The user implements his own UserServices class our download an existing code and then develop the main application, which imports a JCL version (precisely, JCL-User.jar component).

The UserServices.class is a simple Java class with many execute(…) methods and methods with different names. We tested JCL with both static and non-static method types. Specific method names and arguments are also possible, as Figure 4 illustrates. The first method, named execute, is a recursive Fibonacci version, the second is an Arithmetic Progression version and the third is a simple Java collection sorting with method name ordena.

Figure 4. An existing class or a new class to be executed by JCL


Now, let´s come back to appl1.java. We will learn how to invoke a method of your class with JCL. In appl1.java, we define a JCL_result variable named result, useful to get all asynchronous executions results. The result local variable is not registered in JCL. Each JCL_result encapsulates both the correct result of a method call or the exceptionsof error situations, since JCL does not shutdown or crash when user applications crash.

The code first defines an array of Java Objects as variable args1.

Object[] args1 ={new Integer("1"), new Integer("100"), new Integer(10)};

            Future<JCL_result> ticket= jcl.execute("UserServices", args1);



In this example, the array has three Integers(1, 100 and 10, respectively). After args1 definition, it’s possible to call JCL execute method. This method requires a class name and a set of arguments. In our example, we have the class name UserServices and the three arguments in args1.

Since there is no user defined method, JCL will invoke an execute method in UserServices.class, but which one? The args1 argument is responsible for defining which execute method will be executed in UserServices.class. In our example, we have Integer(1), Integer(100) and Integer(10) and in UserServices.class we have only one method with the name execute and three integer arguments. It is the Arithmetic progression method. In summary, in line jcl.execute("UserServices", args1); we are telling JCL to instantiate an object of type UserServices.class and to invoke the execute(Integer, Integer, Integer) method asynchronously, so there is a unique ticket for the submitted task. The ticket will be used to obtain the result forward in the user application appl1.java. This is the basis of asynchronous programming in JCL. JCL provides asynchronous remote method invocations (http://en.wikipedia.org/wiki/Asynchronous_method_invocation).The ticket is a Java future class returning a JCL_result or an exception. Future is an elegant way to get an asynchronous computation result, implementing cancel and different ways to get a JCL_result, including via timeout to wait a certain period for a task result. For more information about Java future, see https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Future.html.

Finally, we have the option of submitting non-execute methods in JCL. In appl1.java, it invokes a specific method named “ordena” with a set of arguments to sort a list of integers. When the user wants to invoke a specific method, its name must be explicit, as the code:

 

IMPORTANT: void methods will have JCL ticket. The ticket must be used to wait for a computation, as presented before. Methods with no arguments can be invoked using a local variable args like: (i) Object[] args=null or (ii) Object[] args = {};

Now it is time to collect the results from previous submitted tasks. Figure 5 illustrates appl1.java while waiting for results. The ticket named ticket1 and the method get() blocks appl1 until the computation finishes. We first wait for sorting to finish and then for PA to finish.

The waiting order of results is programmed by the user in JCL. JCL does not impose any order, since it is a completely asynchronous middleware that can be used to invoke methods sequentially or concurrently in distributed or shared memories computer architectures.

Figure 5. Getting results of method calls in JCL


Suppose we need to execute a method in all computers of a cluster. Going further, suppose we need to execute a method in all cluster cores. The appl2.java exemplify how JCL turns this activity a simple code. There is the alternative to execute methods with the same arguments “executeAll” (line 61 from Figure 6 to execute in all cluster computers) and “executeAllCores” (lines 60 and 62 from Figure 6 to execute in all cluster computer cores).

Figure 6. executeAll and executeAllCores JCL methods


There are scenarios where users need to execute the same method with different arguments. It is the classical single instruction multiple data (SIMD) scenario. In JCL, the user needs to create a matrix of Java objects where each line represents the arguments of the called method.Second, the number of lines of the matrix must be equal to the number of computers in the cluster (“executeAll” at line 40 from Figure 6) or equal to the number of computer cores in the cluster (“executeAllCores” at line 41 from Figure 6).

To create the matrix with correct size you can call “getDevices().size()” (line 36 from appl1.java) to get the number of HPC computers in the cluster or “getClusterCores()” (line 37 from the same code) to get the number of cores in the cluster.

// Get cluster sizes

                       int clusterSize = jcl.getDevices().size();

                       int clusterCoreSize = jcl.getClusterCores();


Both methods “executeAll” and “executeAllCores” return a list of tickets, so to block and wait to all submitted executions results you must call “getAllResultBlocking” (line 69 from Figure 7), but if you want to try to get all results and if some of them are not ready, JCL implements “getAllResultUnblocking” method, where the unfinished tasks return null and the finished ones return valid results or errors. The getAllResultUnblocking does not block the caller until all tasks are finished.

Figure 7. Getting results from executeAll JCL calls

 

We have learned 50% of JCL for HPC scenarios, so far. Let´s start the last 50%. The explanation of the last 50% of JCL HPC services will require appl3.java. JCL supports distributed shared memory (DSM) spaces to store any ordinary Java object in clusters, as well distributed hash maps. For more details about DSM see https://en.wikipedia.org/wiki/Distributed_shared_memory.

Figure 8 illustrates a matrix variable and a simple variable in JCL. The JCL instantiateGlobalVar must be called to create a Java object of user defined type in JCL. A JCL global variable write access is always safe, e.g., JCL guarantees concurrent safe variable write accesses in Lambari or Pacu versions. We will learn these JCL features forward in this tutorial.

Figure 8. JCL global variable definition

In Figure 8, the first JCL variable is a Java String named firstVar and with values equals “nada” and “hello world”. Note that, when we create a Java object with JCL, we must set its name and its default value. After a variable instantiation, the user can modify its content using the setValueUnlocking method (Line 24). The second variable instantiated is a matrix of Java integers. In appl3.java, we have integers created using a local variable named aux and then they are inserted in JCL using names like["Matrix_"+i + ":" + j]. Any variable value is accessible using getValue method with the variable nickname, as appl3.java exemplifies.

IMPORTANT: JCL Pacu version should have coarse grained variables due to Ethernet protocols overhead. In our example, a matrix with 100 cells is instantiated. Try to reduce the number of JCL global variables by ten, e.g., using each variable as an array of 10 Integers and see how fast coarse-grained variables are stored in JCL.

JCL is a middleware supporting concurrent accesses, thus thread-safe accesses to its variables are fundamental. The methods getValueLocking and setValueUnlocking are used when you want to retrieve a variable, update it, and return its new value to JCL without any interference from other threads. After a getValueLocking call no other thread can access the variable to write on it, so a secure concurrent environment is feasible. Deadlocks may occur if you do not pay attention to getValueLocking calls without the respective setValueUnlocking calls.

In Figures 8 and 9, we create variables synchronously, but in JCL there is the asynchronous option, similar to the examples previously explained to execute tasks remotely and asynchronously. A future Java object is returned when a global variable is instantiated, enabling local computations while JCL stores it.

JCL also supports user typed objects. Lambari and Pacu versions support both instantiateGlobalVar API methods to instantiate a user typed object. Figure 9 illustrates how to create a global variable using Jar files. We use a Book class from example1 eclipse project.

Figure 9. User defined types in JCL using a Java class

JCL users are encouraged to create Jar files, since user typed objects might have dependencies to be registered. JCL requires a nickname, a class to be used by JCL to instantiate the object, including its package (Ex. appl.simpleApplication), the Jars files containing the user type and dependencies, and finally the constructor arguments. To get a Book called “Aho Compilers” the user just adopts the nickname and a cast Java operation. A Book class has a print method and a set of Java types.

The last basic question before coding a complete and useful JCL application is: Why does JCL execute Jar files? Why do we have register(java.io.File[] jars,String classToBeExecuted) method in JCL API?

We will use the appl4ExecutingJars.java to exemplify Jar files being registered and executed in JCL. First, the JCL register(File[] jars, String classToBeExecuted) method must be used, as Figure 10 illustrates.

Figure 10. Executing complex Jar files in JCL

In appl4ExecutingJars.java, the user wants to execute a method to print a graphic. If your Jar file has dependencies, you must set such dependencies, like the variable complexApplJars in appl4ExecutingJars.java.

IMPORTANT: it is different from appl1.java example, which adopts class registration. When Jars are used the user must build an array of files, similar to the variable complexApplJars. The first file must be the user application and the other ones are the dependencies.

The Jars of the example: myChartAppl.jar instantiates a jfreechart application with Jar dependencies. This application has jcommon.jar and jfreechart.jar as dependencies. For more information about jfreechart see http://www.jfree.org/jfreechart/.

The main reason to use Jars and not only Class: JCL executes on multicore devices, but also on multicomputer environments, so it is more productive to export the modules of your solution as Jars and register them a single time.

The Jar files can become very large, so network can affect your application runtime. How can I solve this problem? An alternative solution is to run JCL Hosts with your dependencies. Let’s see how to configure JCL Host to execute appl4ExecutingJars.java, but without registering Jar files. The dependencies are inserted into JCL Host bat or sh files(-cp "./JCL_Host.jar;./jcommon-1.0.17.jar;./jfreechart-1.0.14.jar").

Figure 11. Changing JCL Host to include user dependencies


Now the JCL register method can be adopted in a simple way, as Figure 12 illustrates (line 80).

Figure 12. Executing a complex task in a simple way, before starting a customized JCL Host


Sometimes the developer wants more control in a cluster, e.g., he wants to store a global variable or execute a task in a specific JCL Host. In JCL, such a scenario is possible and appl6.java is an example that randomly selects a Host to execute a task. On each Host the task is to create a text file and present its content in the console (see Figure 13).

IMPORTANT: the developer can select Lambari version and call executeOnHost method, as well instantiateGobalVarOnHost method. JCL portability solution solves the problem, so both versions are possible.

Figure 13. executeOnHost example


In JCL, we have a map implementation. JCL map is a subtype of Java Map interface; therefore, all well-known Java Map interface methods are implemented in JCL. JCL distributed map, named JCL-HashMap, is not sorted, e.g., its keys are not sorted, but it is thread-safe over a cluster or a multi-core device.

Figure 14 illustrates two ways to instantiate a JCL-HashMap. The user can call getHashMap or a new JCLHashMap traditional instantiation can be performed. JCL-HashMap has a name because JCL is multi-user, e.g., multiple users with different JCL applications can share a common JCL-HashMap in a JCL cluster. The same can be done for sensing, global variables and registered modules. This advantage is fundamental in cloud and collaborative academic laboratories, as well as in a software house or for voluntary computing like World Community Grid project (https://secure.worldcommunitygrid.org/). Java types or user types can be implemented as JCL-HashMap key or value.

In Figure 14, two maps are instantiated. A Book object is stored in both maps over a cluster. The user can obtain a value from a key, as line 32 illustrates. Both maps are visible to other JCL applications in the same cluster. JCL extends the Java Map interface, inserting getLock and setUnlock methods for concurrent and thread-safe accesses, similar to the explained getValueLocking and setValueUnlocking methods used for ordinary global variables.

IMPORTANT: JCL Map distributed data structure implementation can be used with both JCL versions: Pacu and Lambari. It becomes local when necessary, preserving code portability.

Figure 14. JCL Map utilization


Now we are ready to build more complete and useful JCL HPC applications.

 


2. JCL Task Cost (TC) API


The Task Cost (TC) API is useful when you need to collect several times from a single task submitted to the cluster. JCL implements: i) network time, ii) execution time, iii) queue time and iv) total time. It is possible to collect all times from a task in a single JCL call (line 40 of Figure 15). JCL also collects the task memory consumption. Figure 15 (lines 40-49) illustrates the API usage in appl8.java application. These results are useful for capacity planning strategies!

Figure 15. Capacity Planning API usage

 


3. JCL IoT API


As mentioned before, JCL integrates HPC and IoT, so users can build integrated applications from a unique API. Two examples are introduced to explain some IoT services in JCL and they are appl9.java and appl10.java.

In appl9.java (Figure 16), a JCL instance is obtained at line 15 and two local variables are defined (numMaxRegistro and delay). These variables are used to define the maximum number of sensing data registers to be stored in JCL, e.g., each sensor must have its storage limit in terms of registers, so when the limit is reached JCL removes the old entries, thus avoiding storing sensing data indefinitely. The second variable is the delay to acquire sensing data, e.g., JCL will publish sensing data from a sensor according to this delay. Each sensor can have its own maximum number of sensing data registers and delay.

JCL has the method getIoTDevices to obtain a list of devices that do sensing in the cluster. Note that, JCL also has the getDevices method to obtain all HPC devices from the same cluster. At line 19, it is obtained the unique device of this example, so the first element of a list of devices is set to the arduino variable, demonstrating that the Host used is an Arduino device.

With a device, the user can configure any sensor of it, so the method setSensorMetadata is used for that. The user must specify the device, the nickname of the sensor (TYPE_LIGHT), the sensor type (JCL Constants class has several types of sensors. Ex.Constants.IoT.TYPE_LIGHT), the maximum number of registers to be stored, the publishing delay, and if the sensor is an input one (used for sensing) or an output one (used for actuating. Ex. servo-motors). In our example, a light sensor is configured in an Arduino device remotely and programmatically.

Figure 16. IoT API usage - example 1


After all configurations, the sensor named light is obtained (line 23), and for that the method getSensorByName must be adopted. This method returns a list of sensors, since the user can configure different sensors with the same name or with similar names. The adoption of a unique name for multiple sensors or devices is useful when sensing must be done, regardless some devices or sensors crash.

JCL is the first middleware to introduce context awareness implemented using the task-oriented programming model. The method registerContext (line 27) requires a device, a sensor, a JCL_Expression and the context name. A JCL_Expression is associated to each sensor value, e.g., a GPS produces two values at each sensing (latitude and longitude, respectively), so users can define expressions, like greater than, less than, equal, different and many more, to each GPS value. In our example, a light sensor is used, so there is a unique sensing value per time. At line 27 of Figure 16 the user is telling JCL to publish sensing data only when lighting values are less than 50.

JCL goes further and introduces actions to occur when contexts are reached. An action can be another sensing, an actuating or even a general-purpose task, like the one presented in line 29. The well-known userServices class is registered before adding it as an action to the myContext context. An object of userServices type will be instantiated and executed when a context is reached (line 31), e.g., when the lighting is less than 50. The method addContextAction is responsible for adding actions to a context. Multiple actions can be added to a single context. To configure an action, the user must specify the context name, a boolean value indicating if the sensing data must be used by the task, the nickname of the task, the method to be executed and the task arguments. In our example, the sensing data is not used by the method IoTActionTask and it requires no arguments.

When a context is reached, the IoTActionTask method is invoked once. To invoke the same method again, the sensing data must become above 50 and less than 50 a second time. This strategy avoids several executions while a context continues to be true.  JCL returns a future Java object, since actions are asynchronous, so users can perform local or any other JCL remote computation while waiting for a context. At line 32, the appl9.java object waits until the context myContext is reached. Users can define to wait indefinitely or for a period and for that they must use future class options of get method.

Figure 17a. IoT API usage (part 1) - example 2


Figure 17b. IoT API usage (part 2) - example 2


In Figure 17, there is a second IoT example, where a MOTO X Android device is obtained, its sensor named MOTO X GPS is configured and obtained using a local variable gps. From line 30 to 40 of Figure 17 the application is printing all IoT devices of a cluster and all sensors of the same cluster. At lines 43 and 46 the appl10.java is putting the android device to standby and turning it on. When a device is in standby mode, no sensing data is collected and no context or action occur.

JCL is the unique IoT middleware solution capable to perform active sensing, e.g., it enables sensing data acquisition when users want and not only according to both future contexts and sensing delays. At line 49, the device sensor named gps is accessed remotely and the GPS (lat/lng) valuesare obtained. The method showData provides sensing data visualization, e.g., JCL provides different alternatives to visualize different sensing data, so an audio starts a player, an image a panel and the numerical sensors start a console object. The sensing data is presented in the device where the appl10.java is running. Finally, users can remove previous configured sensors (line 52).

JCL also works with MQTT brokers, publishing data on them and consuming data from them. MQTT becomes a standard in IoT industry in the last years, so this is the justification for the integration. If the user configured a valid MQTT Broker in the config.properties file, automatically all sensing data of IoT Hosts are published to that MQTT broker. Therefore, it’s possible to subscribe to the broker to receive the sensing data without using JCL. The sensor data are published using as topic name the concatenation of the device name and the sensor name separated by a “/”.  Alternatively, it’s possible to subscribe to the topics using the JCL-MQTT component, which requires a Java or a JCL code, similar to previous explained examples or any existing Java code. JCL-MQTT component allows the user to configure a task to be executed on the cluster when a context occurs. The configuration is done using a txt file as figure 18 shows.  

Figure 18. How to configure a MQTT subscriber in JCL

A user can subscribe to as many topics as he wants. Figure 18 shows how to subscribe to two sensors previously configured on a Host. The first one is the potentiometer sensor and the second is the light sensor, both from “galileo1” Host. Threshold and operator indicates the condition to activate the context. Observe that is necessary to indicate the class that will be executed (jar or .class file), a nickname to it and the name of the method to be executed when the context is reached.

The method registerMqttContext allows the user to create a new topic to a specific sensor. Let’s say, for instance, there is a luminosity sensor configured on a Host. All its sensing data is being published via MQTT. This may cause high network traffic and some users may not be interested in receiving all data, but only the data that fits in a threshold. The registerMqttContext method will create a new topic where data is only published to it when the specified context by the user is reached. This way, the user can subscribe to this new topic instead of subscribing to the sensor topic where all data is being published. Figure 19 shows how to create a new MQTT topic using the potentiometer sensor from a Galileo Host. Potentiometer data will be published on this new topic only if its data is according to the specified value, in this case, higher than 500. After this, it’s also possible to use JCL-MQTT component similarly to the previous example using this new topic name to start tasks on the cluster.

Figure 19. How to register a MQTT context

 

4. JCL GPU API


In close future…