JCL CBC

 

What does the sample do?

The solver, named COIN-OR branch-and-cut (CBC) (https://projects.coin-or.org/Cbc), is executed several  times using JCL to calibrate input parameters for specific optimal results. The main goal is to find the best parameters that find the optimal solutions in shorter runtime or opening fewer branches. Note that, these CBC executions to calibrate parameters become a new combinatorial optimization problem. 

 

 

ALERT!!!

This sample uses a CBC and a

Benchmark_ITC3-Linux-x86-64 compiled version that run just in Linux 64bits OS. To run in other OS changes are needed.

 

This sample is organized into five steps:

  1.  Send bin files (cbc,Benchmark_ITC3-Linux-x86-64)
  2. Send CBC instance files.
  3. Calculate CPU time for each core. 
  4. Calculate AVG CPU time of all cores.
  5. Execute cbc and recover the result.

How do I run it?

It is necessary, first, to start a JCL cluster. Inside Host folder execute Host.sh:

To start JCL CBC you have three options:

          1. Execute in the terminal the command: java -jar DistCBC.jar arg1 arg2 arg3 arg4

##################################
"Exec format:"
"java -jar DistCBC.jar arg1 arg2 arg3 arg4"
"arg1: CBC exec file path"
"arg2: ITC3 exec file path"
"arg3: Input file path"
"arg4: C to create file or NC"
"arg5: Option File"
"##################################

         2. Execute in the terminal the command: ./cmdStartC.sh . In this script all steps of cbc are executed.

         3. Execute in the terminal the command: ./cmdStart.sh . In this script, the CBC JCL start in step three.

 

How do I use it?

Put all instance files inside Input folder and use the cbc configuration file named cbcopts1.txt to change all configuration options.

       

 

Alert !!!

The step 2 sends the instance file to the Host, but you should not use this method for a large number of instances. For these scenarios, add all instances in all Hosts previously and configure the .bat or .sh JCL Host script file before start the Hosts. The JCL programming guide illustrates the large JAR files dependencies problem. 

 

 

       If you want, you can add new arguments in cbcopts1.txt file:

 

   The outputs of the execution are shown below:

 

User
Host
 



          After the execution, the results are:

                     1 One "txt" file for each task.

                     2 A file named Tempo_dos_Inputs2.txt with cbc execution time.

                     3 A file named Tempo_dos_Inputs3.txt with all times of a task get using getTaskTimes JCL method.

 

 

The JCL CBC application starts with MainCBC class. 

Important lines:

3--4 => Get JCL instance.

21 => Register CBCLoad class 

27--32 => Load CBC file.

35--39 => Load ITC3 file.

42--48 => Send Bin files to all Hosts.  

51--57 => Read instance filename.

63--87 => Send instance files.

96--102 => Read instance filename.

107--121 => Calculate one hours CPU time

125--160 = > Create all tasks.

162--236 => Recouver result and create output files.

 

public MainCBC(String[] args) throws ExecutionException {		
		//Vars
		JCL_CPfacade jclCP = JCL_CPFacadeImpl.getInstance();
		JCL_facade jcl = jclCP.PacuHPC;			
		Map<String,byte[]> files = new HashMap<String,byte[]>();
		Map<String,byte[]> filesInput = new HashMap<String,byte[]>();
		List<String> fileInputName = new ArrayList<String>();
		Map<Future<JCL_result>,String> tCBC = new HashMap<Future<JCL_result>,String>();
//		Map<String,Long> inputTime = new JCLHashMap<String,Long>("inputTime");

		String CBCFile = args[0];
		String ITC3File = args[1];
		String inputFilePre = args[2];
		String optionsFile = args[4];
		
		try
        {			
			//Register
			System.out.println("Number of cores:"+jcl.getClusterCores());
			jcl.register(CBCLoad.class, "CBCLoad");
			Long ini;

			if(args[3].equals("C")){

			//Read CBC file to send
			RandomAccessFile fCBC = new RandomAccessFile(CBCFile, "r");
			byte[] bCBC = new byte[(int)fCBC.length()];
			fCBC.read(bCBC);
			fCBC.close();
			files.put("cbc", bCBC);
			
			//Read ITC3 file to send			
			RandomAccessFile fITC = new RandomAccessFile(ITC3File, "r");
			byte[] bITC = new byte[(int)fITC.length()];
			fITC.read(bITC);
			fITC.close();
			files.put("Benchmark_ITC3-Linux-x86-64", bITC);						

			//Send exec files
			System.out.println("Fase one, Send exec files.");
			ini = System.nanoTime(); 
			Object[] arg = {files};
			List<Future<JCL_result>> tSend = jcl.executeAll("CBCLoad","LoadExecFile",arg);
			jcl.getAllResultBlocking(tSend);
			files.clear();
			System.out.println("End fase one:"+(System.nanoTime()-ini));

			//List input			
			File[] filesDir = new File(inputFilePre).listFiles();
			for (File f:filesDir) {
			    if (f.isFile()) {
					fileInputName.add(f.getName());
			    }
			}
			
			//If this pathname does not denote a directory, then listFiles() returns null. 

			
			//Send input files
			System.out.println("Fase two, Send input files.");
			ini = System.nanoTime(); 
			
			//Read input file to send
			int cont = 1;
			for (File f:filesDir) {
			    if (f.isFile()) {
					RandomAccessFile finput = new RandomAccessFile(f, "r");
					byte[] binput = new byte[(int)finput.length()];
					finput.read(binput);
					finput.close();
					filesInput.put(f.getName(), binput);

			    if ((filesInput.size()==10) || (filesDir.length==cont) || (args[4].equals("P"))){
			    	Object[] arg3 = {filesInput};
			    	List<Future<JCL_result>> tSendI = jcl.executeAll("CBCLoad","LoadInputFile",arg3);
			    	jcl.getAllResultBlocking(tSendI);
			    	filesInput.clear();
			    	System.out.println("Send input "+f.getName()+" number:"+cont);
			    }
			}	
			    cont++;
			}
			

			
			System.out.println("End fase two:"+(System.nanoTime()-ini));

			} else{
				
				//List input
				fileInputName.clear();
				File[] filesDir = new File(inputFilePre).listFiles();
				for (File f:filesDir) {
				    if (f.isFile()) {
						fileInputName.add(f.getName());
				    }
				}				
			}
			
			
			//Calc time
			System.out.println("Fase three, calculate CPU time.");
			ini = System.nanoTime(); 
			Object[] arg1 = {new String("./Benchmark_ITC3-Linux-x86-64")};			
			List<Future<JCL_result>> tCalc = jcl.executeAllCores("CBCLoad","ExecITC3",arg1);
			jcl.getAllResultBlocking(tCalc);
			System.out.println("End fase three:"+(System.nanoTime()-ini));

			//Calc time mean
			System.out.println("Fase four, calculate mean CPU time.");
			ini = System.nanoTime(); 
			Object[] arg2 = {};
			List<Future<JCL_result>> tCalcM = jcl.executeAll("CBCLoad","CalcTimeMean",arg2);
			jcl.getAllResultBlocking(tCalcM);
			System.out.println("End fase four:"+(System.nanoTime()-ini));
			
			
			//Execute CBC
			System.out.println("Fase five, Execute CBC.");
			ini = System.nanoTime();
			List<String> lines = Files.readAllLines(Paths.get(optionsFile),StandardCharsets.UTF_8);
			int contExec = 0;
			int lineN = 0;
			for(String line:lines){
				String[] ops = line.split(" ");
									
				for(String name:fileInputName){
					ArrayList<String>  array = new ArrayList<String>();
					array.add(new String("./cbc"));
					array.add(new String(""));
					array.addAll(Arrays.asList(ops));
					array.add(new String("seco"));
					array.add(new String(""));
					array.add(new String("solve"));
					String[] cmd = new String[array.size()]; 
					array.toArray(cmd); 
					
					//./cbc   air04.mps   seco 3600   solve  
					cmd[1] = name;
					Object[] arg4 = {cmd,lineN};	
//					Object[] arg4 = {new String(name)};
					Future<JCL_result> t = jcl.execute("CBCLoad","ExecCBC",arg4);
					contExec++;
					tCBC.put(t,name);
					array = null;
					cmd = null;
				}
				
				ops = null;
				lineN++;
			}
			System.out.println("Call "+contExec+" executes Lines Parameter "+lineN+"...");

			int cont=0;
			long soma = 0;
			PrintWriter outT2 = new PrintWriter("Tempo_dos_Inputs2.txt");
			PrintWriter outT3 = new PrintWriter("Tempo_dos_Inputs3.txt");
			
			for(Entry<Future<JCL_result>, String> t:tCBC.entrySet()){
								
				 JCL_result	result = t.getKey().get();				
				 String[] ResultCBC = (String[])result.getCorrectResult();
								
				if (ResultCBC[1].length()==0){
					PrintWriter out = new PrintWriter("Saida_"+ResultCBC[2]+".txt");
					out.println(ResultCBC[0]);
					out.close();
				}else{
					System.out.println("Erro:");
					System.out.println(ResultCBC[1]);
				}
				try {
					soma+=Long.parseLong(ResultCBC[3]);	
					outT2.println(ResultCBC[2]+";"+ResultCBC[3]);
				} catch (Exception e) {
					// TODO: handle exception
					outT2.println("ERRO: "+t.getValue());
					e.printStackTrace();
				}
				
				try {
					
					List<Long> times = jclCP.getTaskTimes(t.getKey());
					outT3.print(ResultCBC[2]+";"+times.size()+";");
					for(Long ti:times){outT3.print(ti.toString()+";");}
					outT3.print(System.getProperty("line.separator"));
				} catch (Exception e) {
					// TODO: handle exception
					outT3.println("ERRO: "+t.getValue());
					e.printStackTrace();
				}

				System.out.println("Results file "+t.getValue()+" Return Name "+ResultCBC[2]+" cont:"+cont);
				outT2.flush();
				outT3.flush();
				result = null;
				ResultCBC = null;
				try {
				//	jcl.removeResult(t.getKey());
				} catch (Exception e) {
					// TODO: handle exception
					System.out.println("Erro Remove result");
					e.printStackTrace();
				}
				cont++;
			}
			
			outT2.println("Total nao Paralelo:"+soma);			
			outT2.println("Final Paralelo:"+(System.nanoTime()-ini));
			outT2.close();
			outT3.close();
			
			System.out.println("End fase five:"+(System.nanoTime()-ini));
			
			
			System.out.println("Criando arquivo de tempo");
			
//			PrintWriter outT = new PrintWriter("Tempo_dos_Inputs.txt");
//			for(Entry<String, Long> ent:inputTime.entrySet()){
//				outT.println(ent.getKey()+"		"+ent.getValue());
//			}
//			
//			outT.close();								
			jcl.destroy();							
        }
		
		
        catch (IOException | InterruptedException exc)
        {
            System.out.println(exc);
            System.exit(1);
        }
	}

	public static void main(String[] args) {
		// TODO Auto-generated method stub
		if (args.length!=5){
			System.out.println("##################################");
			System.out.println("Exec format:");
			System.out.println("java -jar DistCBC.jar arg1 arg2 arg3 arg4");
			System.out.println("arg1: CBC exec file path");
			System.out.println("arg2: ITC3 exec file path");
			System.out.println("arg3: Input file path");
			System.out.println("arg4: C to create file or NC");
			System.out.println("arg5: Option File");
			System.out.println("############## FIM ###############");
		}else{
		try {
			new MainCBC(args);
		} catch (ExecutionException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		}
	}
}

 

 

CBCLoad class creates a CBC(C++) process.

Important lines:

7--27 => receive bin files.

28--44 => receive instance files.

46--92 => execute ITC3 application.

94--147 => execute CBC application.

 

public class CBCLoad {
	
	private List<Integer> timeCpu = Collections.synchronizedList(new ArrayList<Integer>());
	private int timeCpuMean;

	
	public void LoadExecFile(Map<String,byte[]> files){
		
		try {
			for(Entry<String,byte[]> file:files.entrySet()){			
			
				//Create file on disk
				RandomAccessFile f;
				File fileDir = new File(file.getKey());
				f = new RandomAccessFile(fileDir, "rw");			
				f.write(file.getValue());
				f.close();
				fileDir.setExecutable(true, true);

			}
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}
	
	public void LoadInputFile(Map<String,byte[]> files){
		try {
			for(Entry<String,byte[]> file:files.entrySet()){			
			
				//Create file on disk
				RandomAccessFile f;
				File fileDir = new File(file.getKey());
				f = new RandomAccessFile(fileDir, "rw");			
				f.write(file.getValue());
				f.close();
			}
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}
	
	public void ExecITC3(String cmd){
		
		try {
		
			//Start Process
			ProcessBuilder pb = new ProcessBuilder(cmd);
			Process process = pb.start();
		
			//Read STDOut and STDErr
			BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
			BufferedReader erro = new BufferedReader(new InputStreamReader(process.getErrorStream()));

			//Create buffer
			StringBuilder builder = new StringBuilder();
			StringBuilder builderErro = new StringBuilder();

			//Read STDOut
			String line = null;
			while ( (line = reader.readLine()) != null) {
				builder.append(line);
			}

			//Read STDErr
			line = null;
			while ( (line = erro.readLine()) != null) {
				builderErro.append(line);
				builderErro.append(System.getProperty("line.separator"));
			}

		
			process.waitFor();
			timeCpu.add(CalcTimeRef(builder.toString()));
			
			//Print calcTime output 
			System.out.println(builder.toString());
			//Print calcTime Error 
			System.out.println(builderErro.toString());


		} catch (IOException | InterruptedException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}
	
	public String[] ExecCBC(String[] cmd, int id){
		
		try {
		
			//Start Process
			cmd[(cmd.length-2)] = String.valueOf(timeCpuMean);
//			cmd[(cmd.length-2)] = String.valueOf(3610);
			System.out.println("Exec CBC:"+cmd[1]);
//			Map<String,Long> inputTime = new JCLHashMap<String,Long>("inputTime");

			Long ini = System.nanoTime();
			ProcessBuilder pb = new ProcessBuilder(cmd);
			Process process = pb.start();
		
			//Read STDOut and STDErr
			BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
			BufferedReader erro = new BufferedReader(new InputStreamReader(process.getErrorStream()));

			//Create buffer
			StringBuilder builder = new StringBuilder();
			StringBuilder builderErro = new StringBuilder();

			//Read STDOut
			String line = null;
			while ( (line = reader.readLine()) != null) {
				builder.append(line);
				builder.append(System.getProperty("line.separator"));
			}

			//Read STDErr
			line = null;
			while ( (line = erro.readLine()) != null) {
				builderErro.append(line);
				builderErro.append(System.getProperty("line.separator"));
			}
		
			process.waitFor();
			
			long time = (System.nanoTime()-ini);
			String[] result = {builder.toString(),builderErro.toString(),(cmd[1]+"_"+id),(String.valueOf(time))};
			
//			inputTime.put(cmd[1]+id,time);
			System.out.println("Fim Exec "+cmd[1]+" CBC:"+(System.nanoTime()-ini));			
			return result;

		} catch (IOException | InterruptedException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
			return null;
		}
	}
	
	public int CalcTimeRef(String str){		
	    str = str.replaceAll("[^-?0-9]+", " ");
	    return (Integer.parseInt(str.trim())*6);
	}	
	public void CalcTimeMean(){		
		int total = 0; 
		for(Integer time:timeCpu){
			total+= time;
		}
		timeCpuMean = (total/timeCpu.size());
		System.out.println("Mean:"+timeCpuMean);
	}
}

 

This project can be imported in eclipse IDE and all Java source files are inside "src" folder. 

 

Questions or comments, where can I go?

Questions about the API or about the codes of this application? See our Programming Guide and Installation Guide.

If you have any questions, please contact the JCL team.