mar

09

Posted by : admin | On : 9 mars 2010


In this article, we will walk through some of the capabilities provided by Spring batch Framework

Why use Spring Batch? Personally, when I sit down to work with batch, I get frustrated with the amount of code that you need to write to do simple things I/O, manage batches error cases  …

Spring batch let’s us use all capability of spring and provide powerful tool with few lines XML configuration.

Source :

You can checkout the source in Eclipse IDE using SVN here is the address of the source :

http://code.google.com/p/artaud/source/browse/#svn/springbatch

In this article we’ll walk through these case :

-Case 1 : Hello World
-Case 2 : Import  CSV => Database
Case 3 : Export Database => flat file
-Case 4 : Export Database=> XML
-Case 5 : Couple Spring Batch and  Velocity template.
-Case 6 : export our database to a Jasper Report in PDF.

Technologies used :

I assume the reader to have good knowledge in basic Spring and Spring Annotations.

Spring

  1. Spring Annotation
  2. SpringBatch
  3. Spring OXM
  4. Spring Test

Related framework :

  1. Apache  Velocity
  2. Xstream /Stax  (for xml)
  3. Jasper

Database : MySQL

ENVIRONNEMENT

On windows :

JDK 5 or more
Apache Maven
MySQL or an other Database provider
Internet Connection

Eclipse

On Linux :

JDK 5 or more
Apache Maven
MySQL or an other Database provider
sudo apt-get  install apache2 mysql-server php5 php5-mysql phpmyadmin

Eclipse

APACHE MAVEN
In you ~/.bashrc add the following lines .

 

export JAVA_HOME=/usr/lib/jvm/java-6-sun/jre
 export PATH=${PATH}:$JAVA_HOME/bin
 export M2_HOME=/usr/local/apache-maven/apache-maven-2.2.1
 export M2=$M2_HOME/bin
 export MAVEN_OPTS="-Xms512m -Xmx1024m"
 export PATH=${PATH}:$M2
 export ROO_HOME=/home/<USER>/dev/spring-roo-1.0.2.RELEASE
 export ROO_OPTS="-Droo.bright=true"

 

RUN THE PROJET

SEE job.sh in in the folder documentation/ download the code source here.

Configuration of Maven2

First let’s see the pom.xml

 

 

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>org.batch</groupId>
 <artifactId>springbatch</artifactId>
 <name>SpringbatchDemo</name>
 <version>1.0</version>
 <packaging>jar</packaging>
 <description>Springbatch Sample Projet </description>
 <properties>
 <spring.version>2.5.6</spring.version>
 <!--<spring-batch.version>1.0.1.RELEASE</spring-batch.version>-->
 <spring-batch.version>2.1.0.RELEASE</spring-batch.version>
 <mysql-version>5.1.6</mysql-version>
 </properties>
 <dependencies>
 <!-- MySQL -->
 <dependency>
 <groupId>mysql</groupId>
 <artifactId>mysql-connector-java</artifactId>
 <version>${mysql-version}</version>
 </dependency>
 <!-- SPRING -->
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-jdbc</artifactId>
 <version>${spring.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework.batch</groupId>
 <artifactId>spring-batch-core</artifactId>
 <version>${spring-batch.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-aop</artifactId>
 <version>${spring.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-core</artifactId>
 <version>${spring.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-beans</artifactId>
 <version>${spring.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-context</artifactId>
 <version>${spring.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-aspects</artifactId>
 <version>${spring.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring</artifactId>
 <version>${spring.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-jms</artifactId>
 <version>${spring.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework.ws</groupId>
 <artifactId>spring-oxm</artifactId>
 <version>1.5.5</version>
 <optional>true</optional>
 </dependency>
 <dependency>
 <groupId>org.springframework.osgi</groupId>
 <artifactId>spring-osgi-core</artifactId>
 <version>1.1.2</version>
 <optional>true</optional>
 </dependency>
 <!-- Log  -->
 <dependency>
 <groupId>commons-logging</groupId>
 <artifactId>commons-logging</artifactId>
 <version>1.1.1</version>
 </dependency>
 <dependency>
 <groupId>log4j</groupId>
 <artifactId>log4j</artifactId>
 <version>1.2.14</version>
 <optional>true</optional>
 </dependency>
 <dependency>
 <groupId>log4j</groupId>
 <artifactId>log4j</artifactId>
 <version>1.2.13</version>
 </dependency>
 <!-- Junit -->
 <dependency>
 <groupId>junit</groupId>
 <artifactId>junit</artifactId>
 <version>4.4</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>org.springframework.test</artifactId>
 <version>2.5.6</version>
 <scope>test</scope>
 </dependency>
 <!-- Jasper report -->
 <dependency>
 <groupId>jasperreports</groupId>
 <artifactId>jasperreports</artifactId>
 <version>3.1.2</version>
 </dependency>
 <!-- XML STAX -->
 <dependency>
 <groupId>stax</groupId>
 <artifactId>stax</artifactId>
 <version>1.2.0</version>
 </dependency>
 <dependency>
 <groupId>org.springframework.batch</groupId>
 <artifactId>spring-batch-test</artifactId>
 <version>${spring-batch.version}</version>
 </dependency>
 <dependency>
 <groupId>commons-lang</groupId>
 <artifactId>commons-lang</artifactId>
 <version>2.4</version>
 </dependency>
 <!-- Velocity -->
 <dependency>
 <groupId>org.apache.velocity</groupId>
 <artifactId>velocity</artifactId>
 <version>1.6.3</version>
 </dependency>
 <dependency>
 <groupId>org.apache.velocity</groupId>
 <artifactId>velocity-tools</artifactId>
 <version>1.3</version>
 </dependency>
 </dependencies>
 <distributionManagement>
 <repository>
 <id>spring-release</id>
 <name>Spring Release Repository</name>
 <url>s3://maven.springframework.org/release</url>
 </repository>
 <snapshotRepository>
 <id>spring-snapshot</id>
 <name>Spring Snapshot Repository</name>
 <url>s3://maven.springframework.org/snapshot</url>
 </snapshotRepository>
 </distributionManagement>
 <repositories>
 <repository>
 <id>objectstyle</id>
 <name>ObjectStyle.org Repository</name>
 <url>http://objectstyle.org/maven2</url>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 </repository>
 <repository>
 <id>com.springsource.repository.bundles.release</id>
 <name>EBR Spring Release Repository</name>
 <url>http:// repository.springsource.com/maven/bundles/release</url>
 </repository>
 <repository>
 <id>com.springsource.repository.bundles.external</id>
 <name>EBR External Release Repository</name>
 <url>http:// repository.springsource.com/maven/bundles/external</url>
 </repository>
 <repository>
 <id>codehaus</id>
 <name>Maven Codehaus repository</name>
 <url>http://repository.codehaus.org/</url>
 </repository>
 </repositories>
 <pluginRepositories>
 <pluginRepository>
 <id>com.springsource.repository.bundles.milestone</id>
 <name>SpringSource Enterprise Bundle Repository - SpringSource Bundle Milestones</name>
 <url>http://repository.springsource.com/maven/bundles/milestone</url>
 <snapshots>
 <enabled>false</enabled>
 </snapshots>
 </pluginRepository>
 <pluginRepository>
 <id>com.springsource.repository.bundles.snapshot</id>
 <name>SpringSource Enterprise Bundle Repository - SpringSource Bundle Snapshots</name>
 <url>http://repository.springsource.com/maven/bundles/snapshot</url>
 </pluginRepository>
 </pluginRepositories>
 <ciManagement>
 <system>Hudson</system>
 </ciManagement>

 <build>
 <plugins>
 <plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-compiler-plugin</artifactId>
 <configuration>
 <source>1.5</source>
 <target>1.5</target>
 </configuration>
 </plugin>
 <plugin>
 <artifactId>maven-surefire-plugin</artifactId>
 <version>2.4.3</version>
 <configuration>
 </configuration>
 </plugin>
 <!--
 Decomment this blog if you want to run it with
 -Dexec.mainClass="org.springframework.batch.core.launch.support.CommandLineJobRunner"
 -Dexec.args="com/batch/simpletask/simpletaskletcontext.xml
 simpleJob"
 -->
 <plugin>
 <groupId>org.codehaus.mojo</groupId>
 <artifactId>exec-maven-plugin</artifactId>
 <executions>
 <execution>
 <id>glue-processing</id>
 <phase>install</phase>
 <goals>
 <goal>exec</goal>
 </goals>
 </execution>
 </executions>
 <configuration>
 <debug>true</debug>
 <executable>java</executable>
 <mainClass>app_example2.Example2</mainClass>
 <!--
 <mainClass>org.springframework.batch.core.launch.support.CommandLineJobRunner</mainClass>
 <arguments>
 <argument>com/batch/simpletask/simpletaskletcontext.xml</argument>
 <argument>simpleJob</argument> </arguments>
 -->
 </configuration>
 </plugin>
 </plugins>
 </build>
 </project>

 

 

We’ll use Spring annotation couple with standard Spring to realize our sample project

create a package src/main/java/com.batch

these 2  class AppJobExecutionListener and ItemFailureLoggerListener let us log the job execution.

 

 
 package com.batch;
 import org.apache.log4j.Logger;
 import org.springframework.batch.core.BatchStatus;
 import org.springframework.batch.core.JobExecution;
 import org.springframework.batch.core.JobExecutionListener;
 import org.springframework.stereotype.Component;
 @Component("appJobExecutionListener")
 public class AppJobExecutionListener implements JobExecutionListener {
 private final static Logger logger = Logger
 .getLogger(AppJobExecutionListener.class);
 public void afterJob(JobExecution jobExecution) {
 if (jobExecution.getStatus() == BatchStatus.COMPLETED) {
 logger.info("Job completed: " + jobExecution.getJobId());
 } else if (jobExecution.getStatus() == BatchStatus.FAILED) {
 logger.info("Job failed: " + jobExecution.getJobId());
 }
 }
 public void beforeJob(JobExecution jobExecution) {
 if (jobExecution.getStatus() == BatchStatus.COMPLETED) {
 logger.info("Job completed: " + jobExecution.getJobId());
 } else if (jobExecution.getStatus() == BatchStatus.FAILED) {
 logger.info("Job failed: " + jobExecution.getJobId());
 }
 }
 }
 package com.batch;
 import org.apache.log4j.Logger;
 import org.springframework.batch.core.listener.ItemListenerSupport;
 import org.springframework.stereotype.Component;
 @Component("itemFailureLoggerListener")
 public class ItemFailureLoggerListener extends ItemListenerSupport {
 private final static Logger logger = Logger
 .getLogger(ItemFailureLoggerListener.class);
 public void onReadError(Exception ex) {
 logger.error("Encountered error on read", ex);
 }
 public void onWriteError(Exception ex, Object item) {
 logger.error("Encountered error on write", ex);
 }
 }

 

 

First example in Spring Batch

HelloWorld in Spring Batch

Create a package name com.batch.simpletask under src/main/java

 

 

package com.batch.simpletask;
 import org.springframework.batch.core.StepContribution;
 import org.springframework.batch.core.scope.context.ChunkContext;
 import org.springframework.batch.core.step.tasklet.Tasklet;
 import org.springframework.batch.repeat.RepeatStatus;
 public class HelloTask implements Tasklet {
 private String taskStartMessage;
 public void setTaskStartMessage(String taskStartMessage) {
 this.taskStartMessage = taskStartMessage;
 }
 public RepeatStatus execute(StepContribution arg0, ChunkContext arg1)
 throws Exception {
 System.out.println(taskStartMessage);
 return RepeatStatus.FINISHED;
 }
 }

 

 

package com.batch.simpletask;
 import java.util.Calendar;
 import org.springframework.batch.core.StepContribution;
 import org.springframework.batch.core.scope.context.ChunkContext;
 import org.springframework.batch.core.step.tasklet.Tasklet;
 import org.springframework.batch.repeat.RepeatStatus;
 public class TimeTask implements Tasklet {
 public RepeatStatus execute(StepContribution arg0, ChunkContext arg1)
 throws Exception {
 System.out.println(Calendar.getInstance().getTime());
 return RepeatStatus.FINISHED;
 }
 }

 

 

Spring Configuration

 

 
 <?xml version="1.0" encoding="UTF-8"?>
 <beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:aop="http://www.springframework.org/schema/aop"
 xmlns:tx="http://www.springframework.org/schema/tx" xmlns:batch="http://www.springframework.org/schema/batch"
 xmlns:context="http://www.springframework.org/schema/context"
 xsi:schemaLocation="
 http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
 http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.0.xsd
 http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.0.xsd
 http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.0.xsd
 http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd">
 <!-- 1) USE ANNOTATIONS TO IDENTIFY AND WIRE SPRING BEANS. -->
 <!-- We indicate to Spring that we scan @Component annotation based on package com.batch -->
 <context:component-scan base-package="com.batch" />
 <bean id="transactionManager"
 class="org.springframework.batch.support.transaction.ResourcelessTransactionManager">
 </bean>
 <!-- 3) JOB REPOSITORY - WE USE IN-MEMORY REPOSITORY FOR OUR EXAMPLE -->
 <bean id="jobRepository"
 class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
 <property name="transactionManager" ref="transactionManager" />
 </bean>
 <!-- 4) LAUNCH JOBS FROM A REPOSITORY -->
 <bean id="jobLauncher"
 class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
 <property name="jobRepository" ref="jobRepository" />
 </bean>
 <!-- 5) Beans representing the 2 job steps. -->
 <!-- Step1 - print hello world -->
 <bean id="helloTask">
 <property name="taskStartMessage" value="Hello World - the time is now " />
 </bean>
 <!-- Step2 - print current time -->
 <bean id="timeTask" />
 <!-- 6) FINALLY OUR JOB DEFINITION. THIS IS A 2 STEP JOB -->
 <batch:job id="simpleJob">
 <batch:listeners>
 <batch:listener ref="appJobExecutionListener" />
 </batch:listeners>
 <batch:step id="step1" next="step2">
 <batch:tasklet ref="helloTask" />
 </batch:step>
 <batch:step id="step2">
 <batch:tasklet ref="timeTask" />
 </batch:step>
 </batch:job>
 </beans>

 

 

Test our Case 1 :

 

 package com.batch.simpletask;
 import org.apache.log4j.Logger;
 import org.apache.log4j.PropertyConfigurator;
 import org.junit.Before;
 import org.junit.Test;
 import org.junit.runner.RunWith;
 import org.springframework.batch.core.Job;
 import org.springframework.batch.core.JobParameters;
 import org.springframework.batch.core.launch.JobLauncher;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.test.AbstractDependencyInjectionSpringContextTests;
 import org.springframework.test.context.ContextConfiguration;
 import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
 import org.springframework.util.StopWatch;
 @ContextConfiguration(locations = "classpath*:com/batch/simpletask/simpletaskletcontext.xml")
 @RunWith(SpringJUnit4ClassRunner.class)
 public class SimpleTaskletTestCase extends
 AbstractDependencyInjectionSpringContextTests {
 private final static Logger logger = Logger
 .getLogger(SimpleTaskletTestCase.class);
 @Autowired
 private JobLauncher launcher;
 @Autowired
 private Job job;
 private JobParameters jobParameters = new JobParameters();
 @Before
 public void setup() {
 //PropertyConfigurator.configure("C:/Users/admin/workspace-test/springbatch2/src/com/batch/log4j.properties");
 }
 @Test
 public void testLaunchJob() throws Exception {
 StopWatch sw = new StopWatch();
 sw.start();
 launcher.run(job, jobParameters);
 sw.stop();
 logger.info(">>> TIME ELAPSED:" + sw.prettyPrint());
 }
 @Autowired
 public void setLauncher(JobLauncher bootstrap) {
 this.launcher = bootstrap;
 }
 @Autowired
 public void setJob(Job job) {
 this.job = job;
 }
 }

 

 

Install a LAMP server

  • on Linux :

 

sudo apt-get isntall apache2 mysql-server php5 php5-mysql phpmyadmin

 

Check that the process are running

 

 

sudo /etc/init.d/apache2 start
 sudo /etc/init.d/mysql restart

 

  • on Window

Download EasyPHP or Xampp and launch the application

Connect to MySQL

Connect to MySql whether by prompt or via Phpmyadmin http://localhost/phpmyadmin

Create a new user

  • via phpmyadmin

Enter login : test / password testand tick the right rules on the database as the following screenshot

Via prompt :

 

sudo mysql -u root -p

 

CREATE USER 'test'@'%' IDENTIFIED  BY  '***';
 GRANT  SELECT ,
 INSERT ,
 UPDATE ,
 DELETE ,
 CREATE ,
 DROP ,
 FILE ,
 INDEX ,
 ALTER ,
 CREATE  TEMPORARY  TABLES ,
 CREATE  VIEW ,
 EVENT,
 TRIGGER,
 SHOW  VIEW ,
 CREATE ROUTINE,
 ALTER ROUTINE,
 EXECUTE  ON  *  .  *  TO  'test'@'%' IDENTIFIED  BY  '***' WITH  MAX_QUERIES_PER_HOUR 0  MAX_CONNECTIONS_PER_HOUR 0  MAX_UPDATES_PER_HOUR 0  MAX_USER_CONNECTIONS 0 ;

 

 

Create a Database

  • create a new database name seamdb
 CREATE  DATABASE  `seamdb` ;

Create a new Table

[sql]

create table ledger (

ID INT NOT NULL AUTO_INCREMENT,

rcv_dt date,

mbr_nm VARCHAR(100) not null,

chk_nbr VARCHAR(10) not null,

chk_dt date,

pymt_typ VARCHAR(50) not null,

dpst_amt double,

pymt_amt double,

comments VARCHAR(100),

PRIMARY KEY (ID)

);

[/sql]

Example 2: CSV to Database

create a new package com.batch.todb in /src/main/java

Create a Model Bean call Ledger

 

package com.batch.todb;

import java.util.Date;

public class Ledger {

private int id;

private Date receiptDate;

private String memberName;

private String checkNumber;

private Date checkDate;

private String paymentType;

private double depositAmount;

private double paymentAmount;

private String comments;

public int getId() {

return id;

}

public void setId(int id) {

this.id = id;

}

public Date getReceiptDate() {

return receiptDate;

}

public void setReceiptDate(Date receiptDate) {

this.receiptDate = receiptDate;

}

public String getMemberName() {

return memberName;

}

public void setMemberName(String memberName) {

this.memberName = memberName;

}

public String getCheckNumber() {

return checkNumber;

}

public void setCheckNumber(String checkNumber) {

this.checkNumber = checkNumber;

}

public Date getCheckDate() {

return checkDate;

}

public void setCheckDate(Date checkDate) {

this.checkDate = checkDate;

}

public String getPaymentType() {

return paymentType;

}

public void setPaymentType(String paymentType) {

this.paymentType = paymentType;

}

public double getDepositAmount() {

return depositAmount;

}

public void setDepositAmount(double depositAmount) {

this.depositAmount = depositAmount;

}

public double getPaymentAmount() {

return paymentAmount;

}

public void setPaymentAmount(double paymentAmount) {

this.paymentAmount = paymentAmount;

}

public String getComments() {

return comments;

}

public void setComments(String comments) {

this.comments = comments;

}

}

package com.batch.todb;

public interface LedgerDAO {

public void save(final Ledger note);

}

package com.batch.todb;

import java.sql.PreparedStatement;

import java.sql.SQLException;

import javax.sql.DataSource;

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.jdbc.core.JdbcTemplate;

import org.springframework.jdbc.core.PreparedStatementSetter;

import org.springframework.stereotype.Component;

import org.springframework.transaction.annotation.Propagation;

import org.springframework.transaction.annotation.Transactional;

@Component

public class LedgerDAOImpl extends JdbcTemplate implements LedgerDAO {

@Autowired

public void setDataSource(DataSource dataSource) {

super.setDataSource(dataSource);

}

@Transactional(propagation = Propagation.REQUIRED)

public void save(final Ledger item) {

super

.update(

« insert into ledger (rcv_dt, mbr_nm, chk_nbr, chk_dt, pymt_typ, dpst_amt, pymt_amt, comments) values(?,?,?,?,?,?,?,?) »,

new PreparedStatementSetter() {

public void setValues(PreparedStatement stmt)

throws SQLException {

stmt.setDate(1, new java.sql.Date(item

.getReceiptDate().getTime()));

stmt.setString(2, item.getMemberName());

stmt.setString(3, item.getCheckNumber());

stmt.setDate(4, new java.sql.Date(item

.getCheckDate().getTime()));

stmt.setString(5, item.getPaymentType());

stmt.setDouble(6, item.getDepositAmount());

stmt.setDouble(7, item.getPaymentAmount());

stmt.setString(8, item.getComments());

}

});

}

}

package com.batch.todb;

import java.text.DecimalFormat;

import java.text.ParseException;

import org.springframework.batch.item.file.mapping.FieldSetMapper;

import org.springframework.batch.item.file.transform.FieldSet;

import org.springframework.stereotype.Component;

import org.springframework.stereotype.Service;

@Component(« ledgerMapper »)

public class LedgerMapper implements FieldSetMapper {

private final static String DATE_PATTERN = « mm/DD/yy »;

private final static String DOLLAR_PATTERN = « $###,###.### »;

public Object mapFieldSet(FieldSet fs) {

Ledger item = new Ledger();

int idx = 0;

item.setReceiptDate(fs.readDate(idx++, DATE_PATTERN));

item.setMemberName(fs.readString(idx++));

item.setCheckNumber(fs.readString(idx++));

item.setCheckDate(fs.readDate(idx++, DATE_PATTERN));

item.setPaymentType(fs.readString(idx++));

// deposit amount

try {

DecimalFormat fmttr = new DecimalFormat(DOLLAR_PATTERN);

Number number = fmttr.parse(fs.readString(idx++));

item.setDepositAmount(number.doubleValue());

} catch (ParseException e) {

item.setDepositAmount(0);

}

// payment amount

try {

DecimalFormat fmttr = new DecimalFormat(DOLLAR_PATTERN);

Number number = fmttr.parse(fs.readString(idx++));

item.setPaymentAmount(number.doubleValue());

} catch (ParseException e) {

item.setPaymentAmount(0);

}

//

return item;

}

}

package com.batch.todb;

import java.util.Iterator;

import java.util.List;

import org.springframework.batch.item.ItemWriter;

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.stereotype.Component;

@Component(« itemWriter »)

public class LedgerWriter implements ItemWriter {

@Autowired

private LedgerDAO itemDAO;

public void write(List items) throws Exception {

for (Iterator<Ledger> iterator = items.iterator(); iterator.hasNext();) {

Ledger item = iterator.next();

itemDAO.save(item);

}

}

}

Launch Junit Test

In src/test/java create this class of Test

We’ll launch our test using Spring Test framework by using these annotations :

[java]

<pre>@ContextConfiguration(locations = « classpath:com/batch/todb/contextToDB.xml »)

@RunWith(SpringJUnit4ClassRunner.class)

@TransactionConfiguration(transactionManager = « transactionManager », defaultRollback = false)</pre>

[/java]

[java]

package com.batch.todb;

import org.apache.log4j.Logger;

import org.apache.log4j.PropertyConfigurator;

import org.junit.Before;

import org.junit.Test;

import org.junit.runner.RunWith;

import org.springframework.batch.core.Job;

import org.springframework.batch.core.JobParameters;

import org.springframework.batch.core.launch.JobLauncher;

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.test.context.ContextConfiguration;

import org.springframework.test.context.junit4.AbstractTransactionalJUnit4SpringContextTests;

import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import org.springframework.test.context.transaction.TransactionConfiguration;

import org.springframework.util.StopWatch;

@ContextConfiguration(locations = « classpath:com/batch/todb/contextToDB.xml »)

@RunWith(SpringJUnit4ClassRunner.class)

@TransactionConfiguration(transactionManager = « transactionManager », defaultRollback = false)

public class ToDBBatchTestCase extends

AbstractTransactionalJUnit4SpringContextTests {

private final static Logger logger = Logger.getLogger(ToDBBatchTestCase.class);

@Autowired

private JobLauncher launcher;

@Autowired

private Job job;

private JobParameters jobParameters = new JobParameters();

@Before

public void setup() {

//PropertyConfigurator.configure(« C:/Users/admin/workspace-test/springbatch2/src/com/batch/log4j.properties »);

}

@Test

public void testLaunchJob() throws Exception {

StopWatch sw = new StopWatch();

sw.start();

launcher.run(job, jobParameters);

sw.stop();

logger.info(« >>> TIME ELAPSED: » + sw.shortSummary());

}

@Autowired

public void setLauncher(JobLauncher bootstrap) {

this.launcher = bootstrap;

}

@Autowired

public void setJob(Job job) {

this.job = job;

}

}

[/java]

Case 3 : Export Database => flat file

Case 4 : Export Database=> XML

Case 5 : Couple Spring Batch and  Velocity template.

the source are locate into com/batch/velocity

The apache Velocity framework is issue from Jakarta project and is a powerful tool for generating SQL code,Java class C++ , whatever ..

Aim :

In this study case I propose to see how with Spring batch we can couple the two framework together.

We could also use different implementation of  ItemProcessor<T,V> between the ItemReader<T> and ItemWriter<T>  provided by Spring batch , but in this study case it’s not the purpose.

The purpose it to respond fast to our need , generate a bunch of file according to a pattern file .

For instance , you want to generate « a large and complex bouchon class » or « the same SQL code a number of time «   .

The problem is that in a project time is precious so you don’t want to waste it by doing long and repetitif task . I think you grab I want to say ;)

in this sample I start from CSV file and want to replace two variable $name and $ project using this file pattern (the .vm file in Apache Velocity)

Raw input that I have

-Input :

Apache Velocity,Jakarta

Spring batch,Spring source

Spring Roo,Spring Source

-Output expected by the client :

Hello from $name in the $project project.

Spring configuration

What we need is to configure a of a velocityEngine and ItemReader to read our CSV .

Configuration of Velocity Engine

[xml]

<bean id= »velocityEngine »

>

<property name= »velocityProperties »>

<value>

resource.loader=class

class.resource.loader.class=org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader

</value>

</property>

</bean>

[/xml]

Configuration of the ItemReader

[xml]

<bean id= »itemReader »>

<property name= »resource » value= »classpath:com/batch/velocity/project.txt » />

<!– property name= »linesToSkip » value= »1″ /–>

<property name= »lineMapper »>

<bean>

<property name= »lineTokenizer »>

<bean

class= »org.springframework.batch.item.file.transform.DelimitedLineTokenizer »>

<property name= »names » value= »name,project » />

</bean>

</property>

<property name= »fieldSetMapper » ref= »projectMapper » />

</bean>

</property>

</bean>

[/xml]

Full spring configuration

[xml]

<?xml version= »1.0″ encoding= »UTF-8″?>

<beans xmlns= »http://www.springframework.org/schema/beans »

xmlns:xsi= »http://www.w3.org/2001/XMLSchema-instance » xmlns:p= »http://www.springframework.org/schema/p »

xmlns:aop= »http://www.springframework.org/schema/aop » xmlns:tx= »http://www.springframework.org/schema/tx »

xmlns:batch= »http://www.springframework.org/schema/batch »

xmlns:context= »http://www.springframework.org/schema/context »

xmlns:jms= »http://www.springframework.org/schema/jms » xmlns:amq= »http://activemq.apache.org/schema/core »

xmlns:util= »http://www.springframework.org/schema/util »

xsi:schemaLocation= »http://www.springframework.org/schema/beans   http://www.springframework.org/schema/beans/spring-beans.xsd

http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd

http://www.springframework.org/schema/context

http://www.springframework.org/schema/context/spring-context.xsd

http://www.springframework.org/schema/jms

http://www.springframework.org/schema/jms/spring-jms.xsd

http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd

http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd

« >

<!– 1) USE ANNOTATIONS TO CONFIGURE SPRING BEANS –>

<context:component-scan base-package= »com.batch.velocity » />

<!– Author Artaud Antoine  –>

<bean id= »velocityEngine »

class= »org.springframework.ui.velocity.VelocityEngineFactoryBean »>

<property name= »velocityProperties »>

<value>

resource.loader=class

class.resource.loader.class=org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader

</value>

</property>

</bean>

<bean id= »itemReader »>

<property name= »resource » value= »classpath:com/batch/velocity/project.txt » />

<!– property name= »linesToSkip » value= »1″ /–>

<property name= »lineMapper »>

<bean>

<property name= »lineTokenizer »>

<bean

class= »org.springframework.batch.item.file.transform.DelimitedLineTokenizer »>

<property name= »names » value= »name,project » />

</bean>

</property>

<property name= »fieldSetMapper » ref= »projectMapper » />

</bean>

</property>

</bean>

</beans>

[/xml]

Case 6 : export our database to a Jasper Report in PDF.

Case 7 : Couple Spring Batch and  Velocity template + treatement

-Aim : use the 2 framework together and add some basic treatement on data you receive from a CSV file for instance

The configuration XML is the same as previously.

- the only difference reside in the vm template (VTL) that we write in Apache Velocity template .

[xml]

<?xml version= »1.0″ encoding= »UTF-8″?>

<beans xmlns= »http://www.springframework.org/schema/beans »

xmlns:xsi= »http://www.w3.org/2001/XMLSchema-instance » xmlns:p= »http://www.springframework.org/schema/p »

xmlns:aop= »http://www.springframework.org/schema/aop » xmlns:tx= »http://www.springframework.org/schema/tx »

xmlns:batch= »http://www.springframework.org/schema/batch »

xmlns:context= »http://www.springframework.org/schema/context »

xmlns:jms= »http://www.springframework.org/schema/jms » xmlns:amq= »http://activemq.apache.org/schema/core »

xmlns:util= »http://www.springframework.org/schema/util »

xsi:schemaLocation= »http://www.springframework.org/schema/beans   http://www.springframework.org/schema/beans/spring-beans.xsd

http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd

http://www.springframework.org/schema/context

http://www.springframework.org/schema/context/spring-context.xsd

http://www.springframework.org/schema/jms

http://www.springframework.org/schema/jms/spring-jms.xsd

http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd

http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd

« >

<!– 1) USE ANNOTATIONS TO CONFIGURE SPRING BEANS –>

<context:component-scan base-package= »com.batch.velocity.genClass » />

<!– Author Artaud Antoine  –>

<bean id= »velocityEngine »

class= »org.springframework.ui.velocity.VelocityEngineFactoryBean »>

<property name= »velocityProperties »>

<value>

resource.loader=cla

ss

class.resource.loader.class=org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader

</value>

</property>

</bean>

<bean id= »itemReader »>

<property name= »resource » value= »classpath:com/batch/velocity/genClass/dataTest.csv » />

<!– property name= »linesToSkip » value= »1″ /–>

<property name= »lineMapper »>

<bean>

<property name= »lineTokenizer »>

<bean>

<!–<property name= »delimiter » value= »MonDel »/> –>

<property name= »names » value= »champ1,data1Type,documentation1,champ2,data2Type,documentation2″ />

</bean>

</property>

<property name= »fieldSetMapper » ref= »genMapperFactoryMapper » />

</bean>

</property>

</bean>

</beans>

[/xml]

Apache Velocity template

what interest us is the bold part of the bellow code .It indicate to apache Velocity that you call this custom metod treatement on the data you furnish on the « pipe ».

SEE the it /src/test/ressources/com/batch/velocity/genClass/example2.vm

[bash]

/**

*

*/

public class $name {

#foreach( $element in $mylist )

private  $element.Data1Type   $element.champ1;

/**

*

* Map : $element.champ1,$element.data1Type,$element.documentation1

* To : $element.champ2,$element.data2Type,$element.documentation2

* $element.documentation1

* @return the $element.champ1

*

*/

public $element.data1Type <strong>get${utility.getInputU($element.champ1)}</strong>() {

return  $element.champ1;

}

/**

* Map : $element.champ1,$element.data1Type,$element.documentation1

* To : $element.champ2,$element.data2Type,$element.documentation2

* $element.documentation1

* @param $element.champ1 the $element.champ1 to set

* #set( $setter = « set » )

*/

public void  <strong>set${utility.getInputU($element.champ1)}</strong>($element.data1Type $element.champ1) {

this.$element.champ1 = $element.champ1;

}

#end

}

[/bash]

Bellow the Java code that is being called by Apache Velocity

[java]

public class GeneratorUtility {

/**

* set$utility.firstToUpperCase($att.Name)

* @param input the input to set

*/

public String  <strong>getInput</strong>U(String pInput) {

pInput.split(« _ »);

return StringUtils.capitalizeFirstLetter(pInput);

}

}

[/java]

Junit Test

[java]

@ContextConfiguration(locations = « classpath:com/batch/velocity/genClass/contextVelocity.xml »)

@RunWith(SpringJUnit4ClassRunner.class)

public class VelocityCodeTemplateGenClassGenerator {

@Autowired

private VelocityEngine velocityEngine;

@Autowired

private FlatFileItemReader vFlatFileItemReader;

@Test

public void testFlatReaderItemToVelocityProcessor(){

System.out.println(« [INFO] ———————————————————————————- »);

System.out.println(« [INFO] Running test : testFlatReaderItemToVelocityProcessor / « + getClass().getName());

System.out.println(« [INFO] ———————————————————————————- »);

vFlatFileItemReader.open(new ExecutionContext());

boolean hasNext = true ;

GenMapperFactory  vGenMapperFactory = null;

GeneratorUtility vUtility = new GeneratorUtility();

List<GenMapperFactory> vGenMapperFactoryList = new ArrayList<GenMapperFactory>();

//vMyClasseGen.setGenMapperFactory(new ArrayList<GenMapperFactory>());

while (hasNext) {

try {

vGenMapperFactory = (GenMapperFactory) vFlatFileItemReader.read();

vGenMapperFactoryList.add(vGenMapperFactory);

} catch (UnexpectedInputException e) {

e.printStackTrace();

} catch (ParseException e) {

e.printStackTrace();

} catch (Exception e) {

e.printStackTrace();

}

if (vGenMapperFactory == null) {

hasNext = false;

}

else {

Map model = new HashMap();

model.put(« utility », vUtility);

model.put(« mylist », vGenMapperFactoryList);

model.put(« name », « ClassDeTest »);

String text = VelocityEngineUtils.mergeTemplateIntoString(velocityEngine, « com/batch/velocity/genClass/example2.vm », model);

System.out.println( » « +text);

System.out.println(vGenMapperFactory.toString());

}

}

}

}

[/java]

déc

02

Posted by : admin | On : 2 décembre 2011

Le Grand collisionneur de hadrons (LHC) est un gigantesque instrument scientifique situé près de Genève, à cheval sur la frontière franco-suisse, à environ 100 mètres sous terre. C’est un accélérateur de particules, avec lequel les physiciens étudient les plus petites particules connues : les composants fondamentaux de la matière. Le LHC va révolutionner notre compréhension du monde, de l’infiniment petit, à l’intérieur des atomes, à l’infiniment grand de l’Univers.

Deux faisceaux de particules subatomiques de la famille des « hadrons » (des protons ou des ions de plomb) circulent en sens inverse à l’intérieur de l’accélérateur circulaire, emmagasinant de l’énergie à chaque tour. En faisant entrer en collision frontale les deux faisceaux à une vitesse proche de celle de la lumière et à de très hautes énergies, le LHC recrée les conditions qui existaient juste après le Big Bang. Des équipes de physiciens du monde entier analysent les particules issues de ces collisions en utilisant des détecteurs spéciaux.

Il existe de nombreuses théories quant aux résultats de ces collisions. Les physiciens s’attendent en tous cas à une nouvelle ère de physique, apportant de nouvelles connaissances sur le fonctionnement de l’Univers. Pendant des décennies, les physiciens se sont appuyés sur le modèle standard de la physique des particules pour essayer de comprendre les lois fondamentales de la Nature. Mais ce modèle est insuffisant. Les données expérimentales obtenues grâce aux énergies très élevées du LHC permettront de repousser les frontières du savoir, mettant au défi ceux qui cherchent à confirmer les théories actuelles et ceux qui rêvent à de nouveaux paradigmes.

source CERN

déc

19

Posted by : admin | On : 19 décembre 2014

Sécurité informatique

 

Introduction

La sécurité informatique est devenu de nos jours un élément indispensable dans la sécurisation de l’information tant pour un particulier que pour une entreprise. Afin d’évaluer le degré de sécurité à appliquer il faudra s’attacher dans un premier temps de définir le degré d’importance de l’information visant à être protéger et les moyens technique que nous souhaitons mettre en oeuvre afin de protéger ces informations .

Pour sécuriser les systèmes d’information, la démarche consiste à :

  • évaluer les risques et leur criticité : quels risques et quelles menaces, sur quelle donnée et quelle activité, avec quelles conséquences ?
On parle de « cartographie des risques ». De la qualité de cette cartographie dépend la qualité de la sécurité qui va être mise en oeuvre.
  • rechercher et sélectionner les parades : que va-t-on sécuriser, quand et comment ?
Etape difficile des choix de sécurité : dans un contexte de ressources limitées (en temps, en compétences et en argent), seules certaines solutions pourront être mises en oeuvre.
  • mettre en œuvre les protections, et vérifier leur efficacité.
C’est l’aboutissement de la phase d’analyse et là que commence vraiment la protection du système d’information. Une faiblesse fréquente de cette phase est d’omettre de vérifier que les protections sont bien efficaces (tests de fonctionnement en mode dégradé, tests de reprise de données, tests d’attaque malveillante, etc.)

Nous pouvons décliner à plusieurs niveau la sécurité informatique :

Dans cet article nous allons essayer de voir plus particulièrement la sécurité applicative que nous pouvons appliquer via des algorithme de chiffrement et de hashage. Nous allons essayer d’explorer la librairie Java Security API (JCA Cryptography Architecture) ainsi que la librairie Cipher


Pour commencer un bonne introduction sur le domaine de la sécurité informatique un article que j’ai trouvé intéressant sur techno science : http://www.techno-science.net/?onglet=glossaire&definition=6154 http://igm.univ-mlv.fr/~dr/XPOSE2006/depail/fonctionnement.html

Cryptographie asymétrique

Les concepts de signature numérique sont principalement basés sur la cryptographie asymétrique. Cette technique permet de chiffrer avec un mot de passe et de déchiffrer avec un autre, les deux étant indépendants. Par exemlpe, imaginons que Bob souhaite envoyer des messages secret à Alice. Ils vont pour cela utiliser la cryptographie symétrique. Alice génère tout d’abord un couple de clés. Une clé privée (en rouge) et une clé publique (en vert). Ces clés ont des propriétés particulière vis à vis des algorithmes utilisés. En effet, un message chiffré avec une clé ne peut être déchiffré qu’avec l’autre clé. Il s’agit de fonctions à sens unique.

Alice génère un couple de clés

Alice transmet ensuite la clé publique (en vert) à Bob. Grâce à cette clé, Bob peut chiffrer un texte et l’envoyer à Alice.

Bob chiffre avec la clé publique d'Alice

En utilisant la clé publique d’Alice, Bob est certain de deux choses :

  • Personne ne peut lire le message, puisqu’il est crypté
  •  

  • Seule Alice peut déchiffrer le message, car elle est la seule a possèder la clé privée.
  •  

Nous venons de répondre au besoin de confidentialité des données. Mais la cryptographie asymétrique peut être utilisée d’une autre façon. En effet, on peut également utiliser la clé privée pour chiffrer, la clé publique servant alors à déchiffrer. Le message ainsi chiffré est lisible par toute personne disposant de la clé publique. Ceci n’est pas très utile si l’on cherche la confidentialité. En revanche, une seule personne est susceptible d’avoir chiffré ce message : Alice. Ainsi, si l’on peut déchiffrer un message avec la clé publique d’Alice, c’est forcément la personne à avoir chiffré ce message.

Fonctions de hachage

Je vais maintenant décrire les mécanismes permettant de s’assurer que des données n’ont pas été modifiées : les fonctions de hachage. Une fonction de hachage est un procédé à sens unique permettant d’obtenir une suite d’octets (une empreinte) caractérisant un ensemble de données. Pour tout ensemble de données de départ, l’empreinte obtenue est toujours la même. Dans le cadre de la signature numérique, nous nous intéresseront tout particulièrement aux fonctions de hachage cryptographiques. Celles-ci assurent qu’il est impossible de créer un ensemble de données de départ donnant la même empreinte qu’un autre ensemble. Nous pouvons donc utiliser ces fonctions pour nous assurer de l’intégrité d’un document. Les deux algorithme les plus utilisées sont MD5 et SHA.  A noter que MD5 n’est plus considéré comme sûr par les spécialistes. En effet, une équipe chinoise aurait réussi à trouver une collision complète, c’est à dire deux jeux de données donnant la même empreinte, sans utiliser de méthode de force brute. Aujourd’hui, il serait notamment possible de créer deux pages html au contenu différent, ayant pourtant les mêmes empreintes MD5 (en utilisant notamment les balises <meta>, invisibles dans le navigateur). La falsification de documents pourrait donc être possible.

Signer un document

La signature d’un document utilise à la fois la cryptographie asymétrique et les fonctions de hachage. C’est en effet par l’association de ces deux techniques que nous pouvons obtenir les 5 caractéristiques d’une signature (authentique, infalsifiable, non réutilisable, inaltérable, irrévocable). Imaginons que Alice souhaite envoyer un document signé à Bob.

  • Tout d’abord, elle génére l’empreinte du document au moyen d’une fonction de hachage.
  •  

  • Puis, elle crypte cette empreinte avec sa clé privée.
  •  

Alice signe le document pour l'envoyer à Bob

 

  • Elle obtient ainsi la signature de son document. Elle envoie donc ces deux éléments à Bob
  •  

Alice envoie les deux éléments à Bob
  • Pour vérifier la validité du document, Bob doit tout d’abord déchiffrer la signature en utilisant la clé publique d’Alice. Si cela ne fonctionne pas, c’est que le document n’a pas été envoyé par Alice.
  •  

  • Ensuite, Bob génère l’empreinte du document qu’il a reçu, en utilisant la même fonction de hachage qu’Alice (On supposera qu’ils suivent un protocole établi au préalable).
  •  

  • Puis, il compare l’empreinte générée et celle issue de la signature.
  •  

Bob vérifie la signature d'Alice
  • Si les deux empreintes sont identiques, la signature est validée. Nous sommes donc sûr que :
    • C’est Alice qui a envoyé le document,
    •  

    • Le document n’a pas été modifié depuis qu’Alice l’a signé.
    •  

     

  •  

  • Dans le cas contraire, cela peut signifier que :
    • Le document a été modifié depuis sa signature par Alice,
    •  

    • Ce n’est pas ce document qu’Alice a signé
    •  

     

  •  

 

 

 

/**
 * <p>
 * step 1 create a singleton of the service
 * step 2 Java security API (JCA (Java Cryptography Architecture) to obtain an instance of a message digest object using the algorithm supplied
 * step 3 convert plain data to  byte-representation using UTF-8 encoding format. generate an array of bytes that represent the digested (encrypted) password value
 * step 4 Create a String representation of the byte array representing the digested password value.
 * step 5 newly generated hash stock in String
 *
 * </p>
 * @author artaud antoine
 *
 */

public final class PasswordServiceEncryption
{
	private static PasswordServiceEncryption instance;

	private PasswordServiceEncryption()
	{
	}
       //step 1 create a singleton of the service
	public static synchronized PasswordServiceEncryption getInstance()
	{
		if(instance == null)
		{
			instance = new PasswordServiceEncryption();
		}
		return instance;
	}

	public synchronized String encrypt(String plaintext) throws NoSuchAlgorithmException, UnsupportedEncodingException {
		MessageDigest md = null;
		md = MessageDigest.getInstance("SHA-512"); //step 2 Java security API (JCA (Java Cryptography Architecture) to obtain an instance of a message digest object using the algorithm supplied
		System.out.println(plaintext);
		md.reset();
		md.update(plaintext.getBytes("UTF-8")); //step 3 convert plain data to  byte-representation using UTF-8 encoding format. generate an array of bytes that represent the digested (encrypted) password value

		byte raw[] = md.digest(); //step 4 Create a String representation of the byte array representing the digested password value.
		String hash = (new BASE64Encoder()).encode(raw); //step 5 newly generated hash stock in String 

		System.out.println(hash);
		return hash; //step 6
	}

nov

11

Posted by : admin | On : 11 novembre 2014

 

tested in version opensuse 13.2

download the rpm on google music

then execute the command line

sudo zypper in google-musicmanager-beta_current_x86_64.rpm source https://www.alionet.org/content.php?415-Skype-dropbox-google-earth-google-musicmanager-sur-openSUSE-12-3-RC1

Extract mp3 from flv

Tonight I tried to rip the audio track from a flv file. First I wanted to convert it into ogg but as the audio track is a mp3 stream itself I decided just to rip the mp3 data into a file (same quality as in the video file. A higher samplerate would only increase filesize).

UPDATE: Today the audiostream is often aac (not mp3 anymore). That means, you have to use mp4 instead of mp3 with ffmpeg … -f mp4 … .mp4).

One of those commands will do the job:
mplayer -dumpaudio in-video.flv -dumpfile out-audio.mp3
ffmpeg -i input.flv -f mp3 -vn -acodec copy output.mp3

To convert all files (*.flv to *.mp3) in the directory use one of the following commands:
for i in *.flv; do mplayer -dumpaudio -dumpfile « ${i/.flv} ».mp3 « $i »; done
for i in *.flv; do ffmpeg -i « $i » -f mp3 -vn -acodec copy « ${i/.flv} ».mp3; done

#ffmpeg: -f mp3 => force mp3 format, -vn => novideo, -acodec copy => stream-copy
# ${i/.flv} is cutting the filename $i from « .flv »

UPDATE: As the files are not always in the same format and my portable music player couldn’t play mp4 I decided to convert them to ogg.
ffmpeg -i input.flv/mp4 -vn -acodec vorbis output.ogg

This script will transform all flv files in your directory into mp3 files.

1. Create a new file e.g. with kate and name it flv2mp3.sh

#!/bin/bash
for i in *.flv
do
echo « $i »
# Alternative 1
# mplayer -dumpaudio -dumpfile « ${i/.flv} ».mp3 « $i »
#Alternative 2
ffmpeg -i « $i » -f mp3 -vn -acodec copy « ${i/.flv} ».mp3
done

2. Change the right of this file:
chmod +x flv2mp3.sh

3. Goto the directory and run the script

juil

01

Posted by : admin | On : 1 juillet 2014

http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx

http://msdn.microsoft.com/en-us/library/ms228966.aspx

juil

01

Posted by : admin | On : 1 juillet 2014

docs

http://jaliyaudagedara.blogspot.fr/2013/03/asynchronous-operations-in-wcf.html

http://www.codeproject.com/Articles/646239/NET-Asynchronous-Patterns

resources

https://onedrive.live.com/?cid=5f84eceeec6f3f8e&id=5F84ECEEEC6F3F8E!237&authkey=!ADqvp2nWpn6QtSw

 

 

 

sample code illustration

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace AsyncWcfClient
{
    class Program
    {
        static wsService1.Service1Client client1 = new wsService1.Service1Client();
        static wsService2.Service2Client client2 = new wsService2.Service2Client();
        static wsService3.Service3Client client3 = new wsService3.Service3Client();

        static void Main(string[] args)
        {
            client1.GetDataCompleted += client_GetDataCompleted;
            client1.GetDataAsync("event-based asynchronous pattern");
            Console.WriteLine("Waiting for async operation...");

            //client1.BeginGetData("IAsyncResult asynchronous pattern (Client-Side)", new AsyncCallback(GetDataCallBack), null);
            //Console.WriteLine("Waiting for async operation...");

            //client2.BeginWorkerMethod("IAsyncResult asynchronous pattern (Server-Side)", new AsyncCallback(GetDataCallBack), null);
            //Console.WriteLine("Waiting for async operation...");

            //InvokeAsyncMethod("task-based asynchronous pattern");
            //Console.WriteLine("Waiting for async operation...");

            Console.ReadLine();
        }

        static void client_GetDataCompleted(object sender, wsService1.GetDataCompletedEventArgs e)
        {
            Console.WriteLine(e.Result.ToString());
        }

        ////IAsyncResult asynchronous pattern (Client-Side) callback
        //static void GetDataCallBack(IAsyncResult result)
        //{
        //    Console.WriteLine(client1.EndGetData(result).ToString());
        //}

        ////IAsyncResult asynchronous pattern (Server-Side) callback
        //static void GetDataCallBack(IAsyncResult result)
        //{
        //    Console.WriteLine(client2.EndWorkerMethod(result).ToString());
        //}

        //static async void InvokeAsyncMethod(string message)
        //{
        //    Console.WriteLine(await client3.MyWorkerMethodAsync(message));
        //}
    }
}

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.ServiceModel.Web;
using System.Text;

namespace AsyncWcfService
{
 // NOTE: You can use the "Rename" command on the "Refactor" menu to change the interface name "IService1" in both code and config file together.
 [ServiceContract]
 public interface IService1
 {
 //event-based asynchronous pattern
 [OperationContract]
 string GetData(string message);
 }
}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.ServiceModel.Web;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace AsyncWcfService
{
 // NOTE: You can use the "Rename" command on the "Refactor" menu to change the class name "Service1" in code, svc and config file together.
 // NOTE: In order to launch WCF Test Client for testing this service, please select Service1.svc or Service1.svc.cs at the Solution Explorer and start debugging.
 public class Service1 : IService1
 {
 public string GetData(string message)
 {
 Thread.Sleep(5000);
 return message;
 }
 }
}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.Text;
using System.Threading.Tasks;

namespace AsyncWcfService
{
 // NOTE: You can use the "Rename" command on the "Refactor" menu to change the interface name "IService2" in both code and config file together.
 [ServiceContract]
 public interface IService2
 {
 //IAsyncResult asynchronous pattern
 [OperationContractAttribute(AsyncPattern = true)]
 IAsyncResult BeginWorkerMethod(string message, AsyncCallback callback, object asyncState);

 string EndWorkerMethod(IAsyncResult result);
 }
}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace AsyncWcfService
{
 // NOTE: You can use the "Rename" command on the "Refactor" menu to change the class name "Service2" in code, svc and config file together.
 // NOTE: In order to launch WCF Test Client for testing this service, please select Service2.svc or Service2.svc.cs at the Solution Explorer and start debugging.
 public class Service2 : IService2
 {
 public IAsyncResult BeginWorkerMethod(string message, AsyncCallback callback, object asyncState)
 {
 var task = Task<string>.Factory.StartNew((res) => MyMethod(asyncState,message), asyncState);
 return task.ContinueWith(res => callback(task));
 }

 public string EndWorkerMethod(IAsyncResult result)
 {
 return ((Task<string>)result).Result;
 }

 private string MyMethod(object asyncState,string message)
 {
 Thread.Sleep(5000);
 return message;
 }
 }
}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.Text;
using System.Threading.Tasks;

namespace AsyncWcfService
{
 // NOTE: You can use the "Rename" command on the "Refactor" menu to change the interface name "IService3" in both code and config file together.
 [ServiceContract]
 public interface IService3
 {
 //task-based asynchronous pattern
 [OperationContract]
 Task<string> MyWorkerMethodAsync(string message);
 }
}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace AsyncWcfService
{
 // NOTE: You can use the "Rename" command on the "Refactor" menu to change the class name "Service3" in code, svc and config file together.
 // NOTE: In order to launch WCF Test Client for testing this service, please select Service3.svc or Service3.svc.cs at the Solution Explorer and start debugging.
 public class Service3 : IService3
 {
 public async Task<string> MyWorkerMethodAsync(string message)
 {
 return await Task.Factory.StartNew(() => MyMethod(message));
 }

 private string MyMethod(string message)
 {
 Thread.Sleep(5000);
 return message;
 }
 }
}

juin

30

Wcf

Posted by : admin | On : 30 juin 2014

Async WCF Today and Tomorrow – Blog – Stephen Cleary

juin

24

Posted by : admin | On : 24 juin 2014

http://www.kanazawa-net.ne.jp/~pmansato/parallel/parallel_taskfactory.htm

http://codereview.stackexchange.com/questions/44547/writing-highly-asynchronous-code

https://www.dropbox.com/s/f053j1pgw2qo9fz/AsynchronousProgramming.rar

http://chsakell.com/2014/01/11/asynchronous-programming-using-tasks/

http://slynetblog.blogspot.fr/2013/04/downloading-multiple-files-on-windows.html

Connection SQL Serveur

https://www.connectionstrings.com/sql-server-2008/

 

 

WCF Asynchronous

http://robbincremers.me/2011/12/31/wcf-asynchronous-client-proxy-and-asynchronous-service-operations/

http://stackoverflow.com/questions/8040002/how-to-make-a-wcf-rest-method-entirely-asynchronous-with-the-task-parallel-libra

 

using System;
using System.Collections.Generic;
using System.Data;
using System.Data.SqlClient;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace AsynchronousProgramming
{
    class Program
    {
        //const string connectionString = "Data source = localhost; Initial catalog = Chinook; Integrated security = SSPI;";
        const string connectionString = "Data Source=ARTAUDHOME\\SQLEXPRESS;Initial Catalog=Chinook;Integrated Security=True;Connect Timeout=15;Encrypt=False;TrustServerCertificate=False";
        static void Main(string[] args)
        {
            Thread.CurrentThread.Name = "MainThread";

            #region This will block the thread...
            DataSet dsArtists = GetArtistsAsync().Result;
            foreach (DataRow row in dsArtists.Tables["Artists"].Rows)
            {
               foreach (DataColumn col in dsArtists.Tables[0].Columns)
                {
                    Console.Write(row[col] + "\t");
                }
               Console.WriteLine();
            }
            #endregion

            #region This won't block the main thread
            //GetArtistsAsync().ContinueWith(task =>
            //{
            //    DataSet dsArtists = task.Result;
            //    foreach (DataRow row in dsArtists.Tables["Artists"].Rows)
            //    {
            //        foreach (DataColumn col in dsArtists.Tables[0].Columns)
            //        {
            //            Console.Write(row[col] + "\t");
            //        }
            //        Console.WriteLine();
            //    }
            //});
            #endregion

            #region This won't block the thread and catches exceptions
            //GetArtistsAsync().ContinueWith(task =>
            //{
            //    DataSet dsArtists = task.Result;
            //    foreach (DataRow row in dsArtists.Tables["Artists"].Rows)
            //    {
            //        foreach (DataColumn col in dsArtists.Tables[0].Columns)
            //        {
            //            Console.Write(row[col] + "\t");
            //        }
            //        Console.WriteLine();
            //    }
            //}, TaskContinuationOptions.NotOnFaulted);

            //GetArtistsAsync().ContinueWith(task =>
            //{
            //    Console.WriteLine(task.Exception.InnerException.Message);
            //}, TaskContinuationOptions.OnlyOnFaulted);

            #endregion

            #region Task Composition - Multithreading
            //var watch = Stopwatch.StartNew();

            //Task<DataSet> artistsTask = GetArtistsAsync();
            //Task<DataSet> albumsTask = GetAlbumsAsync();

            //Task.Factory.ContinueWhenAll(new[] { artistsTask, albumsTask }, (tasks) =>
            //{
            //    foreach (var task in tasks)
            //    {
            //        if (task.Status == TaskStatus.RanToCompletion)
            //        {
            //            DataSet ds = task.Result;
            //            if (ds.Tables[0].TableName == "Artists")
            //            {
            //                foreach (DataRow row in ds.Tables["Artists"].Rows)
            //                {
            //                    foreach (DataColumn col in ds.Tables[0].Columns)
            //                    {
            //                        Console.Write(row[col] + "\t");
            //                    }
            //                    Console.WriteLine();
            //                }
            //            }
            //            else if (ds.Tables[0].TableName == "Albums")
            //            {
            //                foreach (DataRow row in ds.Tables["Albums"].Rows)
            //                {
            //                    foreach (DataColumn col in ds.Tables[0].Columns)
            //                    {
            //                        Console.Write(row[col] + "\t");
            //                    }
            //                    Console.WriteLine();
            //                }
            //            }
            //        }
            //        else
            //        {
            //            Console.WriteLine("An error has occurred..");
            //            Console.WriteLine(task.Exception.InnerException.Message);
            //        }
            //        Console.WriteLine();
            //        Console.WriteLine("------------------------------------------------");
            //        Console.WriteLine();
            //    }

            //    watch.Stop();
            //    Console.WriteLine("Time elapsed: " + watch.ElapsedMilliseconds + " milliseconds");
            //});
            #endregion

            #region Asynchronous - No multithreading
            //GetCustomersAsync().ContinueWith((task) =>
            //{
            //    DataSet dsCustomers = task.Result;
            //    foreach (DataRow row in dsCustomers.Tables["Customers"].Rows)
            //    {
            //        foreach (DataColumn col in dsCustomers.Tables[0].Columns)
            //        {
            //            Console.Write(row[col] + "\t");
            //        }az
            //        Console.WriteLine();
            //    }
            //});

            #endregion

            Console.WriteLine();
            Console.WriteLine("Thread: " + Thread.CurrentThread.Name);
            Console.WriteLine("Press any key...");
            Console.ReadKey();
        }

        static Task<DataSet> GetArtistsAsync()
        {
            DataSet ds = new DataSet();
            return Task<DataSet>.Factory.StartNew(() =>
            {
                using (SqlConnection con = new SqlConnection(connectionString))
                {
                    string sqlSelect = @"WAITFOR DELAY '000:00:05'
                                        SELECT TOP 10 * FROM Artist";
                    SqlDataAdapter da = new SqlDataAdapter(sqlSelect, con);
                    da.Fill(ds);
                    ds.Tables[0].TableName = "Artists";
                }
                Console.WriteLine("Thread: " + Thread.CurrentThread.Name);
                return ds;
            });
        }

        static Task<DataSet> GetAlbumsAsync()
        {
            DataSet ds = new DataSet();
            return Task<DataSet>.Factory.StartNew(() =>
            {
                using (SqlConnection con = new SqlConnection(connectionString))
                {
                    string sqlSelect = @"WAITFOR DELAY '000:00:05'
                                        SELECT TOP 10 * FROM Album";
                    SqlDataAdapter da = new SqlDataAdapter(sqlSelect, con);
                    da.Fill(ds);
                    ds.Tables[0].TableName = "Albums";
                }
                Console.WriteLine("Thread: " + Thread.CurrentThread.Name);
                return ds;
            });
        }

        static Task<DataSet> GetCustomersAsync()
        {
            var tcs = new TaskCompletionSource<DataSet>();

            DataSet ds = new DataSet();

            using (SqlConnection con = new SqlConnection(connectionString))
            {
                string sqlSelect = @"WAITFOR DELAY '000:00:05'
                                        SELECT TOP 10 * FROM Customer";
                SqlDataAdapter da = new SqlDataAdapter(sqlSelect, con);
                da.Fill(ds);
                ds.Tables[0].TableName = "Customers";
            }
            Console.WriteLine("Thread in GetCustomersAsync: " + Thread.CurrentThread.Name);
            tcs.SetResult(ds);
            return tcs.Task;
        }
    }
}

juin

16

Posted by : admin | On : 16 juin 2014

sample code

 

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Threading;
using System.Collections.Concurrent;

namespace ConsoleApplication2
{
    class Program
    {
        static void Main(string[] args)
        {
            Action<Action> mesure = (body) =>
                {
                    var startTime = DateTime.Now;
                    body();
                    Console.WriteLine("{0} {1}", DateTime.Now - startTime, Thread.CurrentThread.ManagedThreadId);
                };

            Action calJob = () => { for (int i = 0; i < 350000000; i++); };
            Action ioJob = () => { Thread.Sleep(1000); };

            // TPL on top of thread
            /* mesure(() =>
                 {
                  var tasks = Enumerable.Range(1,10)
                              .Select(_ => Task.Factory.StartNew(() => mesure(ioJob)))
                              .ToArray();
                  Task.WaitAll(tasks); // Start 2 task in background
                 });*/

            //Parallel.For(1, 10, _ => { mesure(ioJob); });

            //PLINQ on top TPL
            //use of multiple core

            //Enumerable.Range(1,10).ToList().ForEach(_=>mesure(ioJob)); //no sort order
            //Enumerable.Range(1, 10).AsParallel().WithDegreeOfParallelism(10).ForAll(_ => mesure(ioJob));
            //ThreadPool.SetMinThreads(5, 5);
            // ParallelEnumerable.Range(1, 10).WithDegreeOfParallelism(10).ForAll(_ => mesure(calJob)); 

            //producer Xml reader => queue => consumer
            //var queue = new Queue<int>();

            var queue = new BlockingCollection<int>(100);// concurrent Queue
            var producers = Enumerable.Range(1, 10).Select(_ => Task.Factory.StartNew(() =>
            {
                Enumerable.Range(1, 100)
                    .ToList()
                    .ForEach(i =>
                {
                    queue.Add(i);
                    Thread.Sleep(100);
                });
            })).ToArray();

            var consumers = Enumerable.Range(1, 2).Select(_ => Task.Factory.StartNew(() =>
            {
                foreach(var item in queue.GetConsumingEnumerable()) //get next element and remove from the queue 
                {
                   Console.WriteLine(item);
                };
            })).ToArray();

            Task.WaitAll(producers); // run all producers 
            queue.CompleteAdding(); // when all producers have finished so that it will notify to the consumers when the queue is dequeue 

            Task.WaitAll(consumers);
            Console.ReadKey();
        }
    }
}

juin

09

Posted by : admin | On : 9 juin 2014

 

http://www.symbolsource.org/Public/Metadata/NuGet/Project/SignalR.Client/0.3.3/Release/.NETFramework,Version%3Dv4.0/SignalR.Client/SignalR.Client/SignalR/TaskAsyncHelper.cs?ImageName=SignalR.Client

http://dotnetinside.com/en/type/SignalR.Hosting.AspNet/TaskAsyncHelper/0.5.1.10822

Linq grouping

http://weblogs.asp.net/dixin/understanding-linq-to-objects-3-query-methods

Nice Helper code


using System;

using System.Diagnostics;

using System.Linq;

using System.Threading.Tasks;

namespace SignalR {

 internal static class TaskAsyncHelper {

 private static Task MakeEmpty() {

 return FromResult<object>(null);

 }

 public static Task Empty {

 get {

 // we have to return a new one every time, other wise the task will be disposed

 return MakeEmpty();

 }

 }

 public static Task Catch(this Task task) {

 return task.ContinueWith(t => {

 if (t != null && t.IsFaulted) {

 var ex = t.Exception;

 Trace.TraceError("SignalR exception thrown by Task: {0}", ex);

 }

 return t;

 }).Unwrap();

 }

 public static Task<T> Catch<T>(this Task<T> task) {

 return task.ContinueWith(t => {

 if (t != null && t.IsFaulted) {

 var ex = t.Exception;

 Trace.TraceError("SignalR exception thrown by Task: {0}", ex);

 }

 return t;

 })

 .Unwrap();

 }

 public static Task Success(this Task task, Action<Task> successor) {

 return task.ContinueWith(_ => {

 if (task.IsCanceled || task.IsFaulted) {

 return task;

 }

 return Task.Factory.StartNew(() => successor(task));

 }).Unwrap();

 }

 public static Task Success<TResult>(this Task<TResult> task, Action<Task<TResult>> successor) {

 return task.ContinueWith(_ => {

 if (task.IsCanceled || task.IsFaulted) {

 return task;

 }

 return Task.Factory.StartNew(() => successor(task));

 }).Unwrap();

 }

 public static Task<TResult> Success<TResult>(this Task task, Func<Task, TResult> successor) {

 return task.ContinueWith(_ => {

 if (task.IsFaulted) {

 return FromError<TResult>(task.Exception);

 }

 if (task.IsCanceled) {

 return Cancelled<TResult>();

 }

 return Task.Factory.StartNew(() => successor(task));

 }).Unwrap();

 }

 public static Task<TResult> Success<T, TResult>(this Task<T> task, Func<Task<T>, TResult> successor) {

 return task.ContinueWith(_ => {

 if (task.IsFaulted) {

 return FromError<TResult>(task.Exception);

 }

 if (task.IsCanceled) {

 return Cancelled<TResult>();

 }

 return Task.Factory.StartNew(() => successor(task));

 }).Unwrap();

 }

 public static Task AllSucceeded(this Task[] tasks, Action continuation) {

 return AllSucceeded(tasks, _ => continuation());

 }

 public static Task AllSucceeded(this Task[] tasks, Action<Task[]> continuation) {

 return Task.Factory.ContinueWhenAll(tasks, _ => {

 var cancelledTask = tasks.FirstOrDefault(task => task.IsCanceled);

 if (cancelledTask != null)

 throw new TaskCanceledException();

 var allExceptions =

 tasks.Where(task => task.IsFaulted).SelectMany(task => task.Exception.InnerExceptions).ToList();

 if (allExceptions.Count > 0) {

 throw new AggregateException(allExceptions);

 }

 return Task.Factory.StartNew(() => continuation(tasks));

 }).Unwrap();

 }

 public static Task<T> AllSucceeded<T>(this Task[] tasks, Func<T> continuation) {

 return Task.Factory.ContinueWhenAll(tasks, _ => {

 var cancelledTask = tasks.FirstOrDefault(task => task.IsCanceled);

 if (cancelledTask != null)

 throw new TaskCanceledException();

 var allExceptions =

 tasks.Where(task => task.IsFaulted).SelectMany(task => task.Exception.InnerExceptions).ToList();

 if (allExceptions.Count > 0) {

 throw new AggregateException(allExceptions);

 }

 return Task.Factory.StartNew(continuation);

 }).Unwrap();

 }

 public static Task<T> FromResult<T>(T value) {

 var tcs = new TaskCompletionSource<T>();

 tcs.SetResult(value);

 return tcs.Task;

 }

 private static Task<T> FromError<T>(Exception e) {

 var tcs = new TaskCompletionSource<T>();

 tcs.SetException(e);

 return tcs.Task;

 }

 private static Task<T> Cancelled<T>() {

 var tcs = new TaskCompletionSource<T>();

 tcs.SetCanceled();

 return tcs.Task;

 }

 }

}

 

Resources

http://msdn.microsoft.com/en-us/library/ff963553.aspx

http://www.microsoft.com/en-us/download/confirmation.aspx?id=19222

Article

http://www.albahari.com/nutshell/cs4ch22.aspx

 

http://dotnetcodr.com/2014/03/18/continuation-tasks-in-net-tpl-many-tasks-continued-by-a-single-task/

 

Stackoverflow

http://stackoverflow.com/questions/15938254/task-results-into-a-single-list

mai

30

Posted by : admin | On : 30 mai 2014

 

Know your .NEt version

>cd %systemroot%\Microsoft.NET\Framework

>dir /A:D

Useful plugin for VS 2010

http://visualstudiogallery.msdn.microsoft.com/d0d33361-18e2-46c0-8ff2-4adea1e34fef


Nuget Package installer

http://docs.nuget.org/docs/start-here/using-the-package-manager-console

Func

http://stackoverflow.com/questions/13128271/task-run-and-func

mai

01

Posted by : admin | On : 1 mai 2014

 

Tibco GEMS

For download go to https://tibbr.tibcommunity.com/tibbr/#client-stream

Gems is a graphical user interface utility for TIBCO Enterprise Message Service  (EMS). It can be used by JMS developers as a general purpose test debugging tool and by  administrative support staff as a management and monitorig tool.

 

Gems provides the following main features:

  • Server Monitoring. Server state and main statistics are automtically updated,  warning and error limits may be configured. Server generated events are also  captured.
  • Server Management. Including, general server configuration, JMS destinations,  JNDI factories, users/groups, permissions, bridges, routes etc.
  • JMS support. Messages may be sent/received, queues may be browsed and message  contents inspected. Selectors and filters may be specified.
  • JMS Message Monitoring. Messages may be monitored (snooped) as they pass trough  the server. Request and reply messages can be correlated to provide service  response times.
  • JMS Message Management. Eg; purging messages, copy messages from a queue to  another queue on a different server.
  • Charting. Server statistics may be charted in real time, data may be saved to  CSV files for export to other tools such as Excel.
  • Logging. Server statistics may be logged automatically when warning or error  limits are breached.
  • Security. SSL connectivity, view only mode.
  • Customisable display and look and feel.
  • Support for Managing and Monitoring TIBCO SubStation
  • Support for Monitoring TIBCO EMS Appliance.
  • Support for Migrating to TIBCO EMS Appliance.

 

Requires:

  • TIBCO EMS 5.0 or later
  • JRE 1.6 or later