File Polling using the Spring Integration DSL

Get the Code:  https://github.com/iainporter/spring-file-poller

Introduction

One of the most common use cases for messaging applications is to retrieve files from a directory and process the results.
Spring Integration has great support for this and the DSL makes it very easy to set up.

This post will demonstrate how to set up a transactional file poller using the DSL and offer strategies for unit testing it.

The sample application will

1.  Poll a directory for files that match a regex pattern. 
2.  On reading in a file it will transform the content to a String and move it an Inbound message channel
3.  Another integration flow will listen in on the Inbound channel and process the incoming message by simply reversing the string and writing the result out to another directory.

The whole message flow is transactional and on successful commit the processed file will be moved to a processed directory. If there was a failure then on rollback the file will be moved to a failed directory. The file polling integration acts as a conduit for message processing downstream. The actual processing for this sample project is unimportant, the main point being to demonstrate the management of the file endpoint.

File Polling Components

The directories are configured in the class:  com.porterhead.integration.file.FilePollingConfiguration

@Configuration
public class FilePollingConfiguration {

    @Bean(name="inboundReadDirectory")
    public File inboundReadDirectory(@Value("${inbound.read.path}") String path) {
        return makeDirectory(path);
    }

    @Bean(name="inboundProcessedDirectory")
    public File inboundProcessedDirectory(@Value("${inbound.processed.path}") String path) {
        return makeDirectory(path);
    }

    @Bean(name="inboundFailedDirectory")
    public File inboundFailedDirectory(@Value("${inbound.failed.path}") String path) {
        return makeDirectory(path);
    }

    @Bean(name="inboundOutDirectory")
    public File inboundOutDirectory(@Value("${inbound.out.path}") String path) {
        return makeDirectory(path);
    }

    private File makeDirectory(String path) {
        File file = new File(path);
        file.mkdirs();
        return file;
    }

}

The paths are configured in application.yml and can be easily overriden with external properties.

The spring integration components are in the class: com.porterhead.integration.file.FilePollingIntegrationFlow

Message Source

The file reading endpoint needs a configured MessageSource, in this case the one provided as part of the spring-integration-file library, FileReadingMessageSource:

    @Bean
    public FileReadingMessageSource fileReadingMessageSource(DirectoryScanner directoryScanner) {
        FileReadingMessageSource source = new FileReadingMessageSource();
        source.setDirectory(this.inboundReadDirectory);
        source.setScanner(directoryScanner);
        source.setAutoCreateDirectory(true);
        return source;
    }

NEW Since Spring 5.0
Spring 5 introduced a RecursiveDirectoryScanner class to allow for ease of polling in sub directories.
The bean for this is:

    @Bean
    public DirectoryScanner directoryScanner(@Value("${inbound.filename.regex}") String regex) {
        DirectoryScanner scanner = new RecursiveDirectoryScanner();
        CompositeFileListFilter filter = new CompositeFileListFilter<>(
                Arrays.asList(new AcceptOnceFileListFilter<>(),
                        new RegexPatternFileListFilter(regex))
        );
        scanner.setFilter(filter);
        return scanner;
    }

The component is configured to filter files based on a regex pattern defined in the property inbound.filename.regex
Using the AcceptOnceFileListFilter ensures that duplicate files don't get processed.

Transaction Management

Transactions are handled using the spring provided PseudoTransactionManager. The java docs for this explain perfectly how and why to use this class:

 * An implementation of {@link PlatformTransactionManager} that provides transaction-like semantics to
 * {@link MessageSource}s that are not inherently transactional. It does <b>not</b> make such
 * sources transactional; rather, together with a {@link TransactionSynchronizationFactory}, it provides
 * the ability to synchronize operations after a flow completes, via beforeCommit, afterCommit and
 * afterRollback operations.

The afterCommit expression moves the file to a processed directory. The afterRollback expression moved the file to a failed directory:

    @Bean
    PseudoTransactionManager transactionManager() {
        return new PseudoTransactionManager();
    }

    @Bean
    TransactionSynchronizationFactory transactionSynchronizationFactory() {
        ExpressionParser parser = new SpelExpressionParser();
        ExpressionEvaluatingTransactionSynchronizationProcessor syncProcessor =
                new ExpressionEvaluatingTransactionSynchronizationProcessor();
        syncProcessor.setBeanFactory(applicationContext.getAutowireCapableBeanFactory());
        syncProcessor.setAfterCommitExpression(parser.parseExpression("payload.renameTo(new java.io.File(@inboundProcessedDirectory.path " +
                " + T(java.io.File).separator + payload.name))"));
        syncProcessor.setAfterRollbackExpression(parser.parseExpression("payload.renameTo(new java.io.File(@inboundFailedDirectory.path " +
                " + T(java.io.File).separator + payload.name))"));
        return new DefaultTransactionSynchronizationFactory(syncProcessor);
    }

Spring Integration Flow

The flow configures a poller and task executor, adds in the transaction components and converts the file content to a string before sending it the Inbound Queue.

    @Bean
    public IntegrationFlow inboundFileIntegration(@Value("${inbound.file.poller.fixed.delay}") long period,
                                                  @Value("${inbound.file.poller.max.messages.per.poll}") int maxMessagesPerPoll,
                                                  TaskExecutor taskExecutor,
                                                  MessageSource fileReadingMessageSource) {
        return IntegrationFlows.from(fileReadingMessageSource,
                c -> c.poller(Pollers.fixedDelay(period)
                        .taskExecutor(taskExecutor)
                        .maxMessagesPerPoll(maxMessagesPerPoll)
                        .transactionSynchronizationFactory(transactionSynchronizationFactory())
                        .transactional(transactionManager())))
                .transform(Files.toStringTransformer())
                .channel(ApplicationConfiguration.INBOUND_CHANNEL)
                .get();
    }

Testing

There are four scenarios that we want to test:

1. Polling the directory results in a successful poll when the file name matches the regex pattern
2. Files that don't match the regex pattern are ignored
3. Files that have been seen once are not processed again
4. Transactional behaviour is configured so that files go to the correct directory after processing

The test class is at com.porterhead.integration.file.FilePollingTest

@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = FilePollingTest.TestConfig.class)
@DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_EACH_TEST_METHOD)
public class FilePollingTest  {

    @ClassRule
    public final static TemporaryFolder tempFolder = new TemporaryFolder();

    @Autowired
    @Qualifier("inboundReadDirectory")
    public File inboundReadDirectory;

    @Autowired
    @Qualifier("inboundProcessedDirectory")
    public File inboundProcessedDirectory;

    @Autowired
    @Qualifier("inboundFailedDirectory")
    public File inboundFailedDirectory;

    @Autowired
    @Qualifier("inboundOutDirectory")
    public File inboundOutDirectory;

    @After
    public void tearDown() throws Exception {
        TestUtils.deleteRecursive(inboundReadDirectory);
        TestUtils.deleteRecursive(inboundProcessedDirectory);
        TestUtils.deleteRecursive(inboundFailedDirectory);
        TestUtils.deleteRecursive(inboundOutDirectory);
    }

    @Autowired
    @Qualifier(ApplicationConfiguration.INBOUND_CHANNEL)
    public DirectChannel filePollingChannel;

    @EnableAutoConfiguration
    @ComponentScan(basePackages = "com.porterhead.integration.file,
                                   com.porterhead.integration.configuration")
    public static class TestConfig {

        @Bean
        public File inboundReadDirectory() throws IOException {
            return tempFolder.newFolder("in");
        }

        @Bean
        public File inboundProcessedDirectory() throws IOException {
            return tempFolder.newFolder("processed");
        }

        @Bean
        public File inboundFailedDirectory() throws IOException {
            return tempFolder.newFolder("failed");
        }

        @Bean
        public File inboundOutDirectory() throws IOException {
            return tempFolder.newFolder("out");
        }


        @Bean
        public IntegrationFlow loggingFlow(@Qualifier(ApplicationConfiguration.INBOUND_CHANNEL)
                                          MessageChannel inChannel) {
            return IntegrationFlows.from(inChannel)
                    .handle(this.loggingHandler())
                    .get();
        }

        @Bean
        public MessageHandler loggingHandler() {
            LoggingHandler logger = new LoggingHandler("INFO");
            logger.setShouldLogFullMessage(true);
            return logger;
        }
    }

    @Test
    public void pollFindsValidFile() throws Exception {
        final CountDownLatch latch = new CountDownLatch(1);
        filePollingChannel.addInterceptor(new ChannelInterceptorAdapter() {
            @Override
            public void postSend(Message message, MessageChannel channel, boolean sent) {
                latch.countDown();
                super.postSend(message, channel, sent);
            }
        });
        FileCopyUtils.copy(TestUtils.locateClasspathResource(
              TestUtils.FILE_FIXTURE_PATH), new File(inboundReadDirectory,
              TestUtils.FILE_FIXTURE_NAME ));
        assertThat(latch.await(5, TimeUnit.SECONDS), is(true));
        TestUtils.assertThatDirectoryIsEmpty(inboundReadDirectory);
        TestUtils.assertThatDirectoryIsEmpty(inboundFailedDirectory);
        TestUtils.assertThatDirectoryHasFiles(inboundProcessedDirectory, 1);
    }

    @Test
    public void pollIgnoresInvalidFile() throws Exception {
        FileCopyUtils.copy(TestUtils.locateClasspathResource(
               TestUtils.FILE_FIXTURE_PATH), new File(inboundReadDirectory,
               TestUtils.FILE_FIXTURE_NAME + ".tmp" ));
        TestUtils.assertThatDirectoryIsEmpty(inboundProcessedDirectory);
        TestUtils.assertThatDirectoryIsEmpty(inboundFailedDirectory);
        TestUtils.assertThatDirectoryHasFiles(inboundReadDirectory, 1);
    }

    @Test
    public void pollIgnoresFileAlreadySeen() throws Exception {
        final CountDownLatch stopLatch = new CountDownLatch(1);
        filePollingChannel.addInterceptor(new ChannelInterceptorAdapter() {
            @Override
            public void postSend(Message message, MessageChannel channel, boolean sent) {
                stopLatch.countDown();
                super.postSend(message, channel, sent);
            }
        });
        FileCopyUtils.copy(TestUtils.locateClasspathResource(
               TestUtils.FILE_FIXTURE_PATH), new File(inboundReadDirectory,
               TestUtils.FILE_FIXTURE_NAME ));
        assertThat(stopLatch.await(5, TimeUnit.SECONDS), is(true));
        TestUtils.assertThatDirectoryIsEmpty(inboundReadDirectory);
        TestUtils.assertThatDirectoryIsEmpty(inboundFailedDirectory);
        TestUtils.assertThatDirectoryHasFiles(inboundProcessedDirectory, 1);
        //put file with same name in directory
        FileCopyUtils.copy(TestUtils.locateClasspathResource(
               TestUtils.FILE_FIXTURE_PATH), new File(inboundReadDirectory,
               TestUtils.FILE_FIXTURE_NAME ));
        TestUtils.assertThatDirectoryIsEmpty(inboundFailedDirectory);
        TestUtils.assertThatDirectoryHasFiles(inboundReadDirectory, 1);
        TestUtils.assertThatDirectoryHasFiles(inboundProcessedDirectory, 1);
    }

    @Test
    public void rollbackMovesFileToFailed() throws Exception {
        final CountDownLatch stopLatch = new CountDownLatch(1);
        filePollingChannel.addInterceptor(new ChannelInterceptorAdapter() {
            @Override
            public void postSend(Message message, MessageChannel channel, boolean sent) {
                stopLatch.countDown();
                throw new RuntimeException("Forcing an Exception to trigger rollback");
            }
        })
        FileCopyUtils.copy(TestUtils.locateClasspathResource(
                 TestUtils.FILE_FIXTURE_PATH), new File(inboundReadDirectory, 
                 TestUtils.FILE_FIXTURE_NAME ));
        assertThat(stopLatch.await(5, TimeUnit.SECONDS), is(true));
        TestUtils.assertThatDirectoryIsEmpty(inboundReadDirectory);
        TestUtils.assertThatDirectoryIsEmpty(inboundProcessedDirectory);
        TestUtils.assertThatDirectoryHasFiles(inboundFailedDirectory, 1);
    }

}

The test is run with the usual spring runner:

@RunWith(SpringJUnit4ClassRunner.class)

Rather than load the entire application we only want to load the components that we are interested in testing. We can configure this in a static inner class:

@SpringApplicationConfiguration(classes = FilePollingTest.TestConfig.class)

This allows us to specify the packages that we want spring boot to scan on startup:

    @EnableAutoConfiguration
    @ComponentScan(basePackages = "com.porterhead.integration.file, 

                                   com.porterhead.integration.configuration")

Note that the writer package is excluded. The consequence of this is that the Inbound channel does not have any subscribers so we have to define one in the TestConfig:

        @Bean
        public IntegrationFlow loggingFlow(@Qualifier(ApplicationConfiguration.INBOUND_CHANNEL) MessageChannel inChannel) {
            return IntegrationFlows.from(inChannel)
                    .handle(this.loggingHandler())
                    .get();
        }

The test methods do what their names suggest and make use of channel interceptors to assert that messages arrived, or, in the case of the transaction test, force a rollback.

Runtime Testing

Using maven execute the following command to build the application:

$ mvn clean install

Once built execute the spring boot maven plugin:

$ mvn spring-boot:run

Once running copy the file src/test/resources/data/foo.txt to the created inbound/read directory in the project root. This should result in the file being processed and removed from the read directory.

The processed directory should now have a copy of the original file and the out directory will have a version of the file with the contents reversed.

Implementing JSR-250 hierarchical roles in SpringSecurity

Get the code: https://github.com/iainporter/oauth2-provider

In a previous post I walked through setting up an OAuth2 provider service using Spring Security. I used JSR-250 role-based annotations to protect access to resources. An example of this is the /v1.0/me API to return user details:

    @RolesAllowed({"ROLE_USER"})
    @GET
    public ApiUser getUser(final @Context SecurityContext securityContext) {
        User requestingUser = loadUserFromSecurityContext(securityContext);
        if(requestingUser == null) {
            throw new UserNotFoundException();
        }
        return new ApiUser(requestingUser);
    }

What happens when we create a new role for admin users called ROLE_ADMIN?
This user should have the same access rights as ROLE_USER as well as other restrictive rights not available to regular users. The temptation would be to simply add another role to the list of allowed roles:

@RolesAllowed({"ROLE_USER", "ROLE_ADMIN"})

The better solution would be to keep the access as ROLE_USER but allow anyone with a role that extends ROLE_USER to access the resource. To do that we would have to implement hierarchical roles and the access decision mechanism in Spring Security would need to know how to handle this hierarchy.

Creating an access controlled resource

I'll create a simple service that can be accessed by anyone with the role of ROLE_GUEST. Users with a higher level of access will have a role of ROLE_USER and can access anything that ROLE_GUEST users can. The url of the resource will be /v1.0/samples.

@Path("/v1.0/samples")
@Component
@Produces({MediaType.APPLICATION_JSON})
@Consumes({MediaType.APPLICATION_JSON})
public class SampleResource extends BaseResource {

    @RolesAllowed({"ROLE_GUEST"})
    @GET
    public Response getSample(@Context SecurityContext sc) {
        User user = loadUserFromSecurityContext(sc);
        return Response.ok().entity("{\"message\":\"" + user.getEmailAddress() + 

                  " is authorized to access\"}").build();
    }
}

The resource is in a package that Jersey does not yet know about so it has to be registered with the com.porterhead.RestResourceApplication class

packages("com.porterhead.resource", "com.porterhead.user.resource", "com.porterhead.sample");

Next Spring needs to know about the class in order to apply method level security so we need to add the package to component scanning in application-context.xml:

<context:component-scan base-package="com.porterhead.sample"/>

Writing a failing functional test

Before implementing the changes I'll add a failing functional test that will register a user and then attempt to access the service.
This should result in a 401 status as users are assigned a ROLE_USER role on registration and the system does not know anything about ROLE_GUEST.

    public void testHierarchicalRole() {

        //sign up a user with role of ROLE_USER
        def username = createRandomUserName()
        httpSignUpUser(getCreateUserRequest(username, TEST_PASSWORD))

        //login and get the oauth token
        def loginResponse = httpGetAuthToken(username, TEST_PASSWORD)

        //get the resource that requires a role of ROLE_GUEST
        def sampleResponse = getRestClient().get(path: "/v1.0/samples", 

              contentType: ContentType.JSON, headers: ['Authorization': "Bearer " 

              + loginResponse.responseData["access_token"])

        assertEquals(200, sampleResponse.status)

        return Response.ok().entity("{\"message\":\"" + username.toLowerCase() 

              + " is authorized to access\"}").build();
    }

Execute the tests:

./gradlew integrationTest

This will result in ONE failure which you can view at

  oauth2-provider/build/reports/tests/index.html.

You can also try it out with curl
Start the application by executing ./gradlew tomcatRun and ensure you have mongodb running

First create a user:

   curl -v -X POST -H "Content-Type: application/json" \
   -H "Authorization: Basic MzUzYjMwMmM0NDU3NGY1NjUwNDU2ODdlNTM0ZTdkNmE6Mjg2OTI0Njk3ZTYxNWE2NzJhNjQ2YTQ5MzU0NTY0NmM=" \
   -d '{"user":{"emailAddress":"foo@example.com"}, "password":"password"}' \
   'http://localhost:8080/oauth2-provider/v1.0/users'

Extract the access token from the response and send the request for samples:

curl -v -X GET \    -H "Content-Type: application/json" \
   -H "Authorization: Bearer <the access token from user registration>" \
   'http://localhost:8080/oauth2-provider/v1.0/samples'

You should see a response similar to this:

   HTTP/1.1 401 Unauthorized
   Server Apache-Coyote/1.1 is not blacklisted
   Server: Apache-Coyote/1.1
   Content-Type: application/json
   Content-Length: 168
   Date: Tue, 14 Oct 2014 19:10:26 GMT

   {"errorCode":"401","consumerMessage":"You do not have the appropriate privileges to access this resource","applicationMessage":"Access is denied","validationErrors":[]}

The Role hierarchy class

We only need a simple hierarchy with ROLE_ADMIN extending ROLE_USER which in turn extends ROLE_GUEST.
Fortunately we don't have to do too much work as Spring already has an implementation that is fit for this purpose. We just need to add a bean definition with our role hierarchies to security-configuration.xml.

    <bean id="roleHierarchy"
          class="org.springframework.security.access.hierarchicalroles.

                   RoleHierarchyImpl">
        <property name="hierarchy">
            <value>
                ROLE_ADMIN > ROLE_USER
                ROLE_USER > ROLE_GUEST
            </value>
        </property>
    </bean>

Customising JSR-250 Voter class

The spring implementation of JSR-250 is almost there but does not know how to handle hierarchies. Spring has another class (RoleHierarchyVoter) that knows how to handle hierarchical roles so we can use some of that to extract the roles.

So if we subclass Jsr250Voter and modify the vote method so that it extracts the hierarchical roles that should do the trick.

public class HierarchicalJsr250Voter extends Jsr250Voter {

    private RoleHierarchy roleHierarchy = null;

    public HierarchicalJsr250Voter(RoleHierarchy roleHierarchy) {
        Assert.notNull(roleHierarchy, "RoleHierarchy must not be null");
        this.roleHierarchy = roleHierarchy;
    }

    @Override
    public int vote(Authentication authentication, Object object, 

                     Collection<ConfigAttribute> definition) {
        boolean jsr250AttributeFound = false;

        for (ConfigAttribute attribute : definition) {
            if (Jsr250SecurityConfig.PERMIT_ALL_ATTRIBUTE.equals(attribute)) {
                return ACCESS_GRANTED;
            }

            if (Jsr250SecurityConfig.DENY_ALL_ATTRIBUTE.equals(attribute)) {
                return ACCESS_DENIED;
            }

            if (supports(attribute)) {
                jsr250AttributeFound = true;
                // Attempt to find a matching granted authority
                for (GrantedAuthority authority : extractAuthorities(authentication)) {
                    if (attribute.getAttribute().equals(authority.getAuthority())) {
                        return ACCESS_GRANTED;
                    }
                }
            }
        }

        return jsr250AttributeFound ? ACCESS_DENIED : ACCESS_ABSTAIN;
    }

    Collection<? extends GrantedAuthority> extractAuthorities(

           Authentication authentication) {
        return roleHierarchy.getReachableGrantedAuthorities(

               authentication.getAuthorities());
    }

Wire the new class into the application context

Adding the bean definition to security-configuration.xml:

    <bean id="roleVoter" class="com.porterhead.security.HierarchicalJsr250Voter">
        <constructor-arg ref="roleHierarchy" />
    </bean>

And the final piece of the puzzle is to change the bean that is being referenced by the AccessDecisionManager:

    <bean id="accessDecisionManager" 

          class="org.springframework.security.access.vote.UnanimousBased"
          xmlns="http://www.springframework.org/schema/beans">
        <property name="decisionVoters">
            <list>
                <ref bean="roleVoter"/>
            </list>
        </property>
    </bean>

Testing the changes

First make sure the integration test passes by executing ./gradlew integrationTest

Testing with curl should produce a response similar to this:

   HTTP/1.1 200 OK
   Server Apache-Coyote/1.1 is not blacklisted
   Server: Apache-Coyote/1.1
   Content-Type: application/json
   Content-Length: 42
   Date: Tue, 14 Oct 2014 19:54:28 GMT

  {"message":"You are authorized to access"}

Other posts in this series

• Part 1: An introduction to writing REST Services in Java
• Email Verification
• Lost Password
• Moving to Production
• JSR 303 Validation
• Securing REST Services with Spring Security and OAuth2

Securing REST Services with Spring Security and OAuth2

Get the code: https://github.com/iainporter/oauth2-provider

This post will walk through setting up an OAuth2 provider service for protecting access to REST resources. The code is available in github. You can fork the code and start writing services that will be protected by OAuth access. This is a separate module but builds on services covered in a previous series that includes:

* User Registration and Login
* Email Verification
* Lost Password
* JSR 303 Validation

It would be useful but not required to be familiar with some of the technologies covered such as:

* OAuth2 Protocol
* Spring Security
* Spring Integration
* Spring Data
* Jersey/JAX-RS
* Gradle / Groovy
* MongoDB

Resource Owner Password Flow

Subsequent posts will deal with the other types of authorization flow, such as using third party providers (Facebook, Google, etc). The intent of this post is a walk through of the Resource Owner Password flow. This is a typical use case if you are the system of record for user credentials and trust client applications. It is simply the exchange of the user's username and password for an access token. This token can then be used on subsequent requests to authorize access to resources. It is also important to support token expiration and by extension token refresh.

Let's test out the code and then I'll walk through the application and explain how it is built

Building the project

Check out the source code

> git clone  git@github.com:iainporter/oauth2-provider.git

> cd oauth2-provider

> ./gradlew clean build integrationTest

Running the Web Application

The application uses MongoDB as the persistence store. Before running the application ensure that mongod is running on port 27017. If you don't have mongo installed get it here

Once mongoDB is installed and running fire up the web application using the following command

> ./gradlew tomcatRun

To try it out you can open a browser window and navigate to http://localhost:8080/oauth2-provider/index.html

Testing with Curl

To create a user:

> curl -v -X POST \
   -H "Content-Type: application/json" \
   -H "Authorization: Basic MzUzYjMwMmM0NDU3NGY1NjUwNDU2ODdlNTM0ZTdkNmE6Mjg2OTI0Njk3ZTYxNWE2NzJhNjQ2YTQ5MzU0NTY0NmM=" \
   -d '{"user":{"emailAddress":"user@example.com"}, "password":"password"}' \
   'http://localhost:8080/oauth2-provider/v1.0/users'

The result:

{"apiUser":
   {"emailAddress":"user@example.com",
   "firstName":null,
   "lastName":null,
   "age":null,
   "id":"8a34d009-3558-4c8c-a8da-1ad2b2a393c7",
   "name":"user@example.com"},
   "oauth2AccessToken":
   {"access_token":"7e0e4708-7837-4a7e-9f87-81c6429b02ac",
   "token_type":"bearer",
   "refresh_token":"d0f248ab-e30f-4a85-860c-bd1e388a39b5",
   "expires_in":5183999,
   "scope":"read write"
   }
}

Requesting an access token:

> curl -v -X POST \
   -H "Content-Type: application/json" \
   -H "Authorization: Basic MzUzYjMwMmM0NDU3NGY1NjUwNDU2ODdlNTM0ZTdkNmE6Mjg2OTI0Njk3ZTYxNWE2NzJhNjQ2YTQ5MzU0NTY0NmM=" \
   'http://localhost:8080/oauth2-provider/oauth/token?grant_type=password&username=user@example.com&password=password'

The result:

{
  "access_token":"a838780e-35ef-4bd5-92c0-07a45aa74948",
  "token_type":"bearer",
  "refresh_token":"ab06022f-247c-450a-a11e-2ffab116e3dc",
  "expires_in":5183999
}

Refreshing a token:

> curl -v -X POST \
   -H "Content-Type: application/json" \
   -H "Authorization: Basic MzUzYjMwMmM0NDU3NGY1NjUwNDU2ODdlNTM0ZTdkNmE6Mjg2OTI0Njk3ZTYxNWE2NzJhNjQ2YTQ5MzU0NTY0NmM=" \
   'http://localhost:8080/oauth2-provider/oauth/token?grant_type=refresh_token&refresh_token=ab06022f-247c-450a-a11e-2ffab116e3dc'

The result:

{
   "access_token":"4835cd11-8bb7-4b76-b857-55c6e7f36fc4",
   "token_type":"bearer",
   "refresh_token":"ab06022f-247c-450a-a11e-2ffab116e3dc",
   "expires_in":5183999
}

That's all that is needed to support registration of Users and managing tokens on their behalf. The source code also contains everything needed to support lost password and email verification. See previous posts for how to set it up.

In a typical deployment the authorization server and the resource server(s) would be separate, but for the purposes of this tutorial they are managed within the same application.
Let's delve into the details:

Web Context

There are two servlets.

A Jersey servlet as the default to handle all resource calls:

<servlet-mapping>
        <servlet-name>jersey-servlet</servlet-name>
        <url-pattern>/*</url-pattern>
</servlet-mapping>

A Spring servlet to handle all oauth calls:

<servlet-mapping>
        <servlet-name>spring</servlet-name>
        <url-pattern>/oauth/*</url-pattern>
</servlet-mapping>

Wire in spring security by defining a filter:

    <filter>
        <filter-name>springSecurityFilterChain</filter-name>
        <filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
        <init-param>
            <param-name>contextAttribute</param-name>
            <param-value>org.springframework.web.servlet.FrameworkServlet.CONTEXT.spring</param-value>
        </init-param>
    </filter>

and then map it to the root context so all calls are filtered:

    <filter-mapping>
        <filter-name>springSecurityFilterChain</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>

Configuring OAuth flows to support

<oauth:authorization-server client-details-service-ref="client-details-service" token-services-ref="tokenServices">
        <oauth:refresh-token/>
        <oauth:password/>
    </oauth:authorization-server>

In this scenario only password flow and refresh token are being supported. The default token endpoint is /oauth/token.

Protecting the token endpoint

It is a good idea to protect access to token requests to only those client applications that you know about. Using Spring security we can set up basic authentication on calls to the token endpoint:

    <http pattern="/oauth/token" create-session="stateless" authentication-manager-ref="clientAuthenticationManager"
          xmlns="http://www.springframework.org/schema/security">
        <anonymous enabled="false"/>
        <http-basic entry-point-ref="clientAuthenticationEntryPoint"/>
        <access-denied-handler ref="oauthAccessDeniedHandler"/>
    </http>

Next we configure the authentication manager and client details service

    <bean id="clientCredentialsTokenEndpointFilter"
          class="org.springframework.security.oauth2.provider.client.ClientCredentialsTokenEndpointFilter">
        <property name="authenticationManager" ref="clientAuthenticationManager"/>
    </bean>

    <authentication-manager id="clientAuthenticationManager" xmlns="http://www.springframework.org/schema/security">
        <authentication-provider user-service-ref="client-details-user-service"/>
    </authentication-manager>


    <bean id="client-details-user-service" class="org.springframework.security.oauth2.provider.client.ClientDetailsUserDetailsService">
        <constructor-arg ref="client-details-service" />
    </bean>

Depending on how many clients we expect to access the service will determine what type of client details service we implement. A simple file-based service might be sufficient. If you wanted to go a lot further with client sign up and managing API keys then you would have to involve a persistence tier for the client details service. Configuring with the default Spring file-based service is trivial:

<oauth:client-details-service id="client-details-service">

        
        <oauth:client
                client-id="353b302c44574f565045687e534e7d6a"
                secret="286924697e615a672a646a493545646c"
                authorized-grant-types="password,refresh_token"
                authorities="ROLE_TEST"
                access-token-validity="${oauth.token.access.expiresInSeconds}"
                refresh-token-validity="${oauth.token.refresh.expiresInSeconds}"
                />

        
        <oauth:client
                client-id="7b5a38705d7b3562655925406a652e32"
                secret="655f523128212d6e70634446224c2a48"
                authorized-grant-types="password,refresh_token"
                authorities="ROLE_WEB"
                access-token-validity="${oauth.token.access.expiresInSeconds}"
                refresh-token-validity="${oauth.token.refresh.expiresInSeconds}"
                />

        
        <oauth:client
                client-id="5e572e694e4d61763b567059273a4d3d"
                secret="316457735c4055642744596b302e2151"
                authorized-grant-types="password,refresh_token"
                authorities="ROLE_IOS"
                access-token-validity="${oauth.token.access.expiresInSeconds}"
                refresh-token-validity="${oauth.token.refresh.expiresInSeconds}"
                />

        
        <oauth:client
                client-id="302a7d556175264c7e5b326827497349"
                secret="4770414c283a20347c7b553650425773"
                authorized-grant-types="password,refresh_token"
                authorities="ROLE_ANDROID"
                access-token-validity="${oauth.token.access.expiresInSeconds}"
                refresh-token-validity="${oauth.token.refresh.expiresInSeconds}"
                />

</oauth:client-details-service>

Accessing the oauth endpoint now requires a basic authentication header with the client id and secret concatenated with a ":" separator and base64 encoded.
e.g. -H "Authorization: Basic MzUzYjMwMmM0NDU3NGY1NjUwNDU2ODdlNTM0ZTdkNmE6Mjg2OTI0Njk3ZTYxNWE2NzJhNjQ2YTQ5MzU0NTY0NmM="

Configuring User Authentication Services

The Resource Owner Password flow requires an authentication manager for managing users.

    <bean id="passwordEncoder" class="org.springframework.security.crypto.password.StandardPasswordEncoder"/>

    <sec:authentication-manager alias="userAuthenticationManager">
        <sec:authentication-provider user-service-ref="userService">
            <sec:password-encoder ref="passwordEncoder"/>
        </sec:authentication-provider>
    </sec:authentication-manager>

The password encoder is used to encrypt the password on authentication. The user service also uses the same encoder to encrypt passwords before persisting them.
The user service must implement UserDetailsService and retrieve the user by username.

    @Override
    public UserDetails loadUserByUsername(String username) throws UsernameNotFoundException {
        notNull(username, "Mandatory argument 'username' missing.");
        User user = userRepository.findByEmailAddress(username.toLowerCase());
        if (user == null) {
            throw new AuthenticationException();
        }
        return user;
    }

Configuring Token Services

The final piece of configuration is to manage storage and retrieval of access tokens.

    <bean id="tokenServices" class="org.springframework.security.oauth2.provider.token.DefaultTokenServices">
        <property name="tokenStore" ref="tokenStore"/>
        <property name="supportRefreshToken" value="true"/>
        <property name="clientDetailsService" ref="client-details-service"/>
    </bean>

Spring has a handy in-memory implementation that is useful when testing:

     <bean id="tokenStore" class="org.springframework.security.oauth2.provider.token.InMemoryTokenStore"/>

For this tutorial I have chosen MongoDB as the persistence technology, but it would be easy to wire in an alternative such as MySql or Redis It is a matter of implementing org.springframework.security.oauth2.provider.token.TokenStore

<bean id="tokenStore" class="com.porterhead.oauth2.mongodb.OAuth2RepositoryTokenStore">
      <constructor-arg ref="OAuth2AccessTokenRepository"/>
      <constructor-arg ref="OAuth2RefreshTokenRepository"/>
</bean>

Protecting access to resources

First we need to configure a spring resource server filter. This will check that there is a valid access token in the request header.

    <oauth:resource-server id="resourceServerFilter" token-services-ref="tokenServices"/>

Spring security has some useful classes for fine-grain control of roles and permissions. I prefer to use JSR-250 annotations. Luckily it is easy to wire this in. First we need to enable JSR-250 annotations with the following configuration:

    <sec:global-method-security jsr250-annotations="enabled" access-decision-manager-ref="accessDecisionManager"/>

Configure an access manager that uses the Jsr250Voter

    <bean id="accessDecisionManager" class="org.springframework.security.access.vote.UnanimousBased">
        <property name="decisionVoters">
            <list>
                <bean class="org.springframework.security.access.annotation.Jsr250Voter"/>
            </list>
        </property>
    </bean>

Now we can protect REST resource methods with JSR-250 annotations such as @RolesAllowed

Trying it out

Typically the glue between the OAuth server and the application is a user identifier. When a client gets an access token for a user the next step is to typically load data related to that user. This will usually involve building a url with the userId as part of the path
i.e. /v1.0/users/{id}/someresource
A useful service to provide is a way to get user information based on the access token. The application can identify the user that owns the token and return information on that user to the client.

@Path("/v1.0/me")
@Component
@Produces({MediaType.APPLICATION_JSON})
@Consumes({MediaType.APPLICATION_JSON})
public class MeResource extends BaseResource {

    @RolesAllowed({"ROLE_USER"})
    @GET
    public ApiUser getUser(final @Context SecurityContext securityContext) {
        User requestingUser = loadUserFromSecurityContext(securityContext);
        if(requestingUser == null) {
            throw new UserNotFoundException();
        }
        return new ApiUser(requestingUser);
    }

    protected User loadUserFromSecurityContext(SecurityContext securityContext) {
        OAuth2Authentication requestingUser = (OAuth2Authentication) securityContext.getUserPrincipal();
        Object principal = requestingUser.getUserAuthentication().getPrincipal();
        User user = null;
        if(principal instanceof User) {
            user = (User)principal;
        } else {
            user = userRepository.findByEmailAddress((String)principal);
        }
        return user;
    }
}

To test this out start up the application:

> ./gradlew tomcatRun

Execute the following curl statement, substituting the access token that was returned from the login curl above

> curl -v -X GET \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer [your token here]" \
  'http://localhost:8080/oauth2-provider/v1.0/me'

To try out some of the other services included in the code see previous posts:

• Part 1: An introduction to writing REST Services in Java
• Email Verification
• Lost Password
• Moving to Production
• JSR 303 Validation

Writing REST Services in Java: Part 8 JSR 303 Validation

Previous Post : Part Seven: Moving To Production

Get the Code: https://github.com/iainporter/rest-java

The introduction of JSR 303 for validation really simplifies the process of validation. It consists of a meta data model using annotations and an API for running validations against annotated classes.

 It cleanly separates the concerns and prevents pojo classes from being cluttered with custom validation code.

In this rest sample project the DTO objects that comprise the API layer are annotated with validations. The service tier is then responsible for handling any validation failures and wrapping them in ValidationExceptions.

Using this approach makes it possible to publish your API with all of the DTO classes and service interfaces and rely on the consumer of the API to make their own decisions on Validation handling.

JSR 303 Implementation

The implementation used in the project is Hibernate Validator.

The dependencies in gradle:

'org.hibernate:hibernate-validator:4.3.1.Final'
'javax.validation:validation-api:1.1.0.Final'

for maven:


<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-validator</artifactId>
<version>5.0.1.Final</version>
</dependency>

<dependency>
<groupId>javax.validation</groupId>
<artifactId>validation-api</artifactId>
<version>1.1.0.Final</version>
</dependency>

Applying Constraints

@XmlRootElement
public class CreateUserRequest {

    @NotNull
    @Valid
    private ExternalUser user;

    @NotNull
    @Valid
    private PasswordRequest password;


    public CreateUserRequest() {
    }

    public CreateUserRequest(final ExternalUser user, final PasswordRequest password) {
        this.user = user;
        this.password = password;
    }

    public ExternalUser getUser() {
        return user;
    }

    public void setUser(ExternalUser user) {
        this.user = user;
    }

    public PasswordRequest getPassword() {
        return password;
    }

    public void setPassword(PasswordRequest password) {
        this.password = password;
    }

}

The two properties user and password can not be null. The @Valid constraint performs validation recursively on the objects.

The relevant part of ExternalUser.java

@XmlRootElement
public class ExternalUser implements Principal {

    private String id;
    
    @Length(max=50)
    private String firstName;
    
    @Length(max=50)
    private String lastName;
    
    @NotNull
    @Email
    private String emailAddress;

    ............

and PasswordRequest.java

@XmlRootElement
public class PasswordRequest {

    @Length(min=8, max=30)
    @NotNull
    private String password;

    public PasswordRequest() {}

    public PasswordRequest(final String password) {
        this.password = password;
    }

    public String getPassword() {
        return password;
    }

    public void setPassword(String password) {
        this.password = password;
    }

}

@Length is part of the Bean Validation spec whilst @Email is a Hibernate custom validation

Validating Constraints

First we must create an instance of Validator and then pass it into any service that requires it.

 @Autowired
    public UserServiceImpl(UsersConnectionRepository usersConnectionRepository, Validator validator) {
        this(validator);
        this.jpaUsersConnectionRepository = usersConnectionRepository;
        ((JpaUsersConnectionRepository)this.jpaUsersConnectionRepository).setUserService(this);
    }

We can call validate on any request object in the service class and throw a Validation Exception if there are any failures

    @Transactional
    public AuthenticatedUserToken createUser(CreateUserRequest request, Role role) {
        validate(request);
        User searchedForUser = userRepository.findByEmailAddress(request.getUser().getEmailAddress());
        if (searchedForUser != null) {
            throw new DuplicateUserException();
        }

        User newUser = createNewUser(request, role);
        AuthenticatedUserToken token = new AuthenticatedUserToken(newUser.getUuid().toString(), createAuthorizationToken(newUser).getToken());
        userRepository.save(newUser);
        return token;
    }

The validate method

   protected void validate(Object request) {
        Set<? extends ConstraintViolation<?>> constraintViolations = validator.validate(request);
        if (constraintViolations.size() > 0) {
            throw new ValidationException(constraintViolations);
        }
    }

The Validation Exception class wraps any errors in a response object to return to the client to provide a meaningful contextual error response.

public class ValidationException extends WebApplicationException {

    private final int status = 400;
    private String errorMessage;
    private String developerMessage;
    private List<ValidationError> errors = new ArrayList<ValidationError>();

    public ValidationException() {
        errorMessage = "Validation Error";
        developerMessage = "The data passed in the request was invalid. Please check and resubmit";
    }

    public ValidationException(String message) {
        super();
        errorMessage = message;
    }

    public ValidationException(Set<? extends ConstraintViolation<?>> violations) {
        this();
        for(ConstraintViolation<?> constraintViolation : violations) {
            ValidationError error = new ValidationError();
            error.setMessage(constraintViolation.getMessage());
            error.setPropertyName(constraintViolation.getPropertyPath().toString());
            error.setPropertyValue(constraintViolation.getInvalidValue() != null ? constraintViolation.getInvalidValue().toString() : null);
            errors.add(error);
        }
    }

    @Override
    public Response getResponse() {
        return Response.status(status).type(MediaType.APPLICATION_JSON_TYPE).entity(getErrorResponse()).build();
    }

    public ErrorResponse getErrorResponse() {
        ErrorResponse response = new ErrorResponse();
        response.setApplicationMessage(developerMessage);
        response.setConsumerMessage(errorMessage);
        response.setValidationErrors(errors);
        return response;
    }

}

Testing Validation

Testing is easy. Just create an instance of Validator and pass it the object to test.

public class PasswordRequestTest {

    protected Validator validator;

    @Before
    public void setUp() {
        validator = Validation.buildDefaultValidatorFactory().getValidator();
    }

    @Test
    public void validPassword() {
        PasswordRequest request = new PasswordRequest("password");
        Set<ConstraintViolation<PasswordRequest>> constraints = validator.validate(request);
        assertThat(constraints.size(), is(0));
    }

    public void passwordTooShort() {
        PasswordRequest request = new PasswordRequest(RandomStringUtils.randomAlphanumeric(7));
        Set<ConstraintViolation<PasswordRequest>> constraints = validator.validate(request);
        assertThat(constraints.size(), is(1));
    }

    public void passwordTooLong() {
        PasswordRequest request = new PasswordRequest(RandomStringUtils.randomAlphanumeric(36));
        Set<ConstraintViolation<PasswordRequest>> constraints = validator.validate(request);
        assertThat(constraints.size(), is(1));
    }
}

Testing the API

To test is against the running application start up the server by executing gradle tomcatRun

Execute an curl statement with an invalid request such as the following

curl -v -H "Content-Type: application/json" -X POST -d '{"user":{"firstName","lastName":"Bar","emailAddress":"@##@.com"}, "password":"123"}' http://localhost:8080/java-rest/user

You should see s result similar to this

HTTP/1.1 400 Bad Request Server: Apache-Coyote/1.1 Content-Type: application/json Transfer-Encoding: chunked Date: Mon, 13 May 2013 21:22:23 GMT Connection: close * Closing connection #0 {"errorCode":null,"consumerMessage":"Validation Error","applicationMessage":"The data passed in the request was invalid. Please check and resubmit","validationErrors":[{"propertyName":"user.emailAddress","propertyValue":"@##@.com","message":"not a well-formed email address"},{"propertyName":"password.password","propertyValue":"123","message":"length must be between 8 and 30"}]}

There are lots more cool things you can do with validation. See the spec for more details.

Writing REST Services in Java: Part 7 Moving To Production

Previous Post : Part Six: Security & Authorization
Get the Code: https://github.com/iainporter/rest-java

This post will discuss the steps needed to configure the project to deploy it in a staging or production environment.
It is only of interest if you wish to use the code and deploy it beyond the in-memory defaults.

Database Configuration

The project makes use of Spring's active profiles.

Profiles can be set with a System property or, when running gradle by setting the property in gradle.properties
Choices are dev | local | staging | production

When running the dev profile the database is an in-memory H2 database.

The first step is to test that the application will run against a real database. The local profile can be used for this purpose.

The configuration file is at 

src/main/resources/META-INF/spring/data-context.xml

You can change the properties to suit your local environment.

Test the application by setting the active profile to local and executing:

gradle clean build integrationTest

If the build is successful there should now be test data in the database.

Adding indexes and Message Store tables

On start up if the tables do not exist they will be created. The next step is to add the relevant indexes by executing the script in:

src/main/resources/schema/indexes.sql

Finally we need to add the tables used by Spring Integration to persist messages on the queues. There are scripts for various database types that ship with the Spring source code in the spring-integration-jdbc.jar
A default script for MySql is available in

src/main/resources/schema/message_store.sql

There are placeholders in data-context.xml for adding the staging and production profile database configurations. Repeat the above steps for each of these environments.

Email Configuration

Both the dev and local profiles use a MockJavaMailSender which does not deliver any messages but stores them in a HashMap. When you are ready to send real mail messages then you will have to configure the mail properties in

src/main/resources/META-INF/spring/email-template-context.xml

As discussed in Part Three you can choose from a variety of different mail services. The simplest is to use a gmail or yahoo account. There are also many useful bulk mail services that you would want to use in production such as CritSend, MailJet, MailChimp, etc.

One final configuration point to mention is that the mail service is backed by several message queues implemented in Spring Integration. These queues have a backing message store to which to temporarily persist messages until they are delivered. When running with the dev or local profiles the message store is a simple in-memory store which will not survive a crash or restart. For staging and production the messages are persisted to the datastore configured for that profile, hence the requirement for the sql script mentioned above to create those tables.

Additional Configuration

There are a few other properties that you can set in

src/main/resources/properties/app.properties

Here you can change how long session tokens should remain active, which authorization method to use, set email preferences, etc. See the comments in that file for full details

The next post will discuss how to write new services and the final post will discuss deploying to the cloud.

Writing REST Services in Java: Part 6 Security & Authorization

Note: Whilst some of the code in this post is still useful the preferred method for authentication is OAuth2 (see my blog post here - http://porterhead.blogspot.co.uk/2014/05/securing-rest-services-with-spring.html)

Previous Post : Part Five: Lost Password
Get the Code: https://github.com/iainporter/rest-java

Part Two dealt with signing up and authenticating users using the conventional email and password approach. Part Four also covered authentication but using the OAuth protocol. This post will cover authorization of services.

To that end what we want to accomplish is:

• Identify who is making the request
• Have a high degree of confidence that they are who they say they are
• Limit access to only those resources that they are allowed to access
• Prevent malicious tampering of requests or session hijacking

SSL

Ideally all API traffic should be over a secure connection using SSL. If that is not feasible then at the very least all requests that would be open to MITM attacks should be secured, such as account creation, login, etc

Security Filter

All API requests first go through a security filter. An authorization service is registered with the filter on startup. Depending on the level of security you are comfortable with and how much effort you want to subject clients to will determine the choice of service. I'll cover that later in the post. First we need to register the filter with the servlet container.

In src/main/webapp/WEB-INF/web.xml a ResourceFilterFactory is added to the config

        <init-param>
            <param-name>com.sun.jersey.spi.container.ResourceFilters</param-name>
            <param-value>com.porterhead.rest.filter.ResourceFilterFactory</param-value>
        </init-param>

The ResourceFilterFactory extends the Jersey class RolesAllowedResourceFilterFactory. In it we register the security filter and ensure it is the first filter in line.

@Component
@Provider
public class ResourceFilterFactory extends RolesAllowedResourceFilterFactory {

    @Autowired
    private SecurityContextFilter securityContextFilter;

    @Override
    public List<ResourceFilter> create(AbstractMethod am) {
        List<ResourceFilter> filters = super.create(am);
        if (filters == null) {
            filters = new ArrayList<ResourceFilter>();
        }
        List<ResourceFilter> securityFilters = new ArrayList<ResourceFilter>(filters);
        //put the Security Filter first in line
        securityFilters.add(0, securityContextFilter);
        return securityFilters;
    }
}

The SecurityContextFilter class gathers some information from the request and then delegates to the AuthorizationService implementation to handle authorizing the request and loading the user. If the request is valid a user is returned unless the resource is not access controlled in which case no authorization headers will be required. The user is wrapped in a SecurityContext and added to the ContainerRequest.

    public ContainerRequest filter(ContainerRequest request) {
        String authToken = request.getHeaderValue(HEADER_AUTHORIZATION);
        String requestDateString = request.getHeaderValue(HEADER_DATE);
        String nonce = request.getHeaderValue(HEADER_NONCE);
        AuthorizationRequestContext context = new AuthorizationRequestContext(request.getPath(), request.getMethod(),
                            requestDateString, nonce, authToken);
        ExternalUser externalUser = authorizationService.authorize(context);
        request.setSecurityContext(new SecurityContextImpl(externalUser));
        return request;
    }

Role Based Access

Using JAX-RS annotations we can protect resources based on roles. It is the task of the SecurityContext implementation that was added to the Container Request in the above step that will answer the question of whether the user has the role.

    public boolean isUserInRole(String role) {
        if(role.equalsIgnoreCase(Role.anonymous.name())) {
             return true;
        }
        if(user == null) {
            throw new InvalidAuthorizationHeaderException();
        }
        return user.getRole().equalsIgnoreCase(role);
    }

If anyone can access the resource, as in the case of user/login then it is annotated with @PermitAll

    @PermitAll
    @Path("login")
    @POST

If we want to limit access then we use the annotation @RolesAllowed with a list of allowed roles

    @RolesAllowed({"authenticated"})
    @Path("{userId}")
    @GET
    public Response getUser(@Context SecurityContext sc, @PathParam("userId") String userId) {
        ExternalUser userMakingRequest = (ExternalUser)sc.getUserPrincipal();
        ExternalUser user =  userService.getUser(userMakingRequest, userId);
        return Response.ok().entity(user).build();
    }

Note that we can also use annotations to retrieve the user that was wrapped in the SecurityContext as part of the Authorization procedure. We can then check that this user can not only access this resource method but also at the service level whether they can access the user object that is being requested. In this situation we want to check that they can only request getUser on their own instance. We could add "administrative" role access and allow that user to access any user instance.

Session Token Authorization

The simplest method of authorization is to get a session token on login or sign up and pass that back on every request. This is obviously not secure enough to be transmitted over anything other than an SSL connection. It can be useful for testing services quickly and easily with curl without the overhead of signing the request each time.

        public ExternalUser authorize(AuthorizationRequestContext securityContex) {
        String token = securityContext.getAuthorizationToken();
        ExternalUser externalUser = null;
        if(token == null) {
            return externalUser;
        }
        User user =  userRepository.findBySession(token);
        if(user == null) {
            throw new AuthorizationException("Session token not valid");
        }
        AuthorizationToken authorizationToken = user.getAuthorizationToken();
            if (authorizationToken.getToken().equals(token)) {
                externalUser = new ExternalUser(user);
            }
        return externalUser;
    }

We extract the token from the request header and query the User Repository for a User with that Session token.

Testing Session Token Authorization

1. In src/main/resources/properties/app.properties set the following property

security.authorization.requireSignedRequests=false

2. Start the application by executing

gradle tomcatRun

3. Create a user with the curl statement

curl -v -H "Content-Type: application/json" -X POST -d '{"user": {"firstName":"Foo","lastName":"Bar","emailAddress":"foobar@example.com"}, "password":"password"}' http://localhost:8080/java-rest/user

You should receive back an AuthenticatedUserToken similar to this

< HTTP/1.1 201 Created < Server: Apache-Coyote/1.1 < Location: http://localhost:8080/java-rest/user/d26079db-62e9-4819-964a-be954d2c47ed < Content-Type: application/json < Transfer-Encoding: chunked < Date: Tue, 29 Jan 2013 19:48:05 GMT {"userId":"d26079db-62e9-4819-964a-be954d2c47ed","token":"86e8be75-3eac-45aa-acc3-f043333c8608"}

4. For Session token authorization we only need the token part. Now construct a curl statement based on the location url in the response to GET a user similar to the following

curl -v -H "Content-Type: application/json" -X GET -H "Authorization: 86e8be75-3eac-45aa-acc3-f043333c8608" 'http://localhost:8080/java-rest/user/d26079db-62e9-4819-964a-be954d2c47ed'

You should receive a response similar to this one

< HTTP/1.1 200 OK < Server: Apache-Coyote/1.1 < Content-Type: application/json < Transfer-Encoding: chunked < Date: Tue, 29 Jan 2013 19:58:13 GMT {"id":"d26079db-62e9-4819-964a-be954d2c47ed","firstName":"Foo","lastName":"Bar","emailAddress":"foobar@example.com","socialProfiles":[],"name":"foobar@example.com","verified":false}

Request Signing Authorization

A surer method of securing requests that does not require an SSL connection other than for passing back and forth credentials is to sign requests. This implementation is based somewhat on the OAuth spec, although does not go as far as to require the request body to be signed. That can easily be plugged in though for an added layer of security.

On login or sign up an AuthenticatedUserToken is returned. An AuthenticatedUserToken is comprised of:

• userId - passed with every role-based request as a means to identify the user.
• token - used as a shared secret between the client and the application to verify the hash of a token passed with the request.

To generate a hashed token a String is composed of the following

• The session token
• A : separator
• The relative url of the resource (i.e. user/ff7ffcf0-cfe0-4c1e-9971-3b934612b154) followed by ,
• The HTTP method (GET, POST, PUT, DELETE) followed by ,
• A time stamp of the request in Iso8061 format followed by ,
• A unique nonce token generated by the client

This string is then hashed using SHA-256 and then Base64 encoded. The Authorization header is composed of the user token followed by a : separator and then the hashed signature
In order for the server to reconstitute the string the timestamp and nonce must also be passed as headers.

The main methods to authorize signed requests

    public ExternalUser authorize(AuthorizationRequestContext context) {

        ExternalUser externalUser = null;
        if (context.getAuthorizationToken() != null && context.getRequestDateString() != null && context.getNonceToken() != null) {
            String userId = null;
            String hashedToken = null;
            String[] token = context.getAuthorizationToken().split(":");
            if (token.length == 2) {
                userId = token[0];
                hashedToken = token[1];
                //make sure date and nonce is valid
                validateRequestDate(context.getRequestDateString());
                validateNonce(context.getNonceToken());

                User user = userRepository.findByUuid(userId);
                if (user != null) {
                    externalUser = new ExternalUser(user);
                    if (!isAuthorized(externalUser, context, hashedToken)) {
                        throw new AuthorizationException("Request rejected due to an authorization failure");
                    }
                }
            }
        }
        return externalUser;
    }


    private boolean isAuthorized(User user, AuthorizationRequestContext authorizationRequest, String hashedToken) {
        Assert.notNull(user);
        Assert.notNull(authorizationRequest.getAuthorizationToken());
        String unEncodedString = composeUnEncodedRequest(authorizationRequest);
        AuthorizationToken authorizationToken = user.getAuthorizationToken();
        String userTokenHash = encodeAuthToken(authorizationToken.getToken(), unEncodedString);
            if (hashedToken.equals(userTokenHash)) {
                return true;
            }
        LOG.error("Hash check failed for hashed token: {} for the following request: {} for user: {}",
                new Object[]{authorizationRequest.getAuthorizationToken(), unEncodedString, user.getId()});
        return false;
    }

If the request headers are present then we attempt to authorize the request.
The timestamp is checked to ensure it conforms within the configurable boundaries of the server clock.
The nonce value is checked to ensure it is unique
The user is loaded and we iterate over their session tokens and attempt to verify the request token using the shared secret.

Testing Signed Request Authorization

1. In src/main/resources/properties/app.properties set the following property

security.authorization.requireSignedRequests=true

2. Start the application by executing

gradle tomcatRun

3. Create a user with the curl statement

curl -v -H "Content-Type: application/json" -X POST -d '{"user": {"firstName":"Foo","lastName":"Bar","emailAddress":"foobar@example.com"}, "password":"password"}' http://localhost:8080/java-rest/user

4. Construct a signed request to call user/id GET using the returned AuthenticatedUserToken
{"userId":"ff7b93ad-27d0-49f6-90bd-9937951e5fcc","token":"6eebc0ea-b637-4033-925b-3b3cba9880e4"}
First the string to hash:

• 6eebc0ea-b637-4033-925b-3b3cba9880e4 (the session token)
• user/ff7b93ad-27d0-49f6-90bd-9937951e5fcc (the resource url)
• GET (The Http Method)
• 2013-01-30T10:50:00+00:00 (Timestamp)
• 22e327d732 (nonce value)

The full string

6eebc0ea-b637-4033-925b-3b3cba9880e4:user/ff7b93ad-27d0-49f6-90bd-9937951e5fcc,GET,2013-01-30T10:50:00+00:00,22e327d732

Hash the string using SHA-256 and then Base64 encode the result. There is a utility class at com.incept5.rest.util.HashUtil that you can use.
This should render a result similar to

ncYoA5n5s2nFSm7qyvf5hDgL4pmmPOUP3zo/UYfaQKg=

5. Construct the curl statement with relevant headers which should look similar to the following

curl -v -H "Content-Type: application/json" -H "Authorization: ff7b93ad-27d0-49f6-90bd-9937951e5fcc:ncYoA5n5s2nFSm7qyvf5hDgL4pmmPOUP3zo/UYfaQKg=" -H "x-java-rest-date:2013-01-30T10:50:00+00:00" -H "nonce:22e327d732" -X GET -d localhost http://localhost:8080/java-rest/user/ff7b93ad-27d0-49f6-90bd-9937951e5fcc

After executing the curl statement you should get back something like:

< HTTP/1.1 200 OK < Server: Apache-Coyote/1.1 < Content-Type: application/json < Transfer-Encoding: chunked < Date: Wed, 30 Jan 2013 10:48:56 GMT {"id":"ff7b93ad-27d0-49f6-90bd-9937951e5fcc","firstName":"Foo","lastName":"Bar","emailAddress":"foobar@example.com","socialProfiles":[],"name":"foobar@example.com","verified":false}

Tampering with any of the header values or resubmitting the request should result in a failure such as

< HTTP/1.1 403 Forbidden < Server: Apache-Coyote/1.1 < Content-Type: application/json < Transfer-Encoding: chunked < Date: Wed, 30 Jan 2013 10:52:26 GMT {"errorCode":"40301","consumerMessage":"Not authorized","applicationMessage":"Nonce value is not unique"}

The application has some javascript functions for creating signed requests on the client side (see src/main/webapp/js/javarest.js)
That now covers everything that you need to create and manage simple user accounts in a REST API. The next post will focus on configuring the application for production

Writing REST services in Java: Part 5 Lost Password

Previous Post : Part Four: Facebook Authentication
Get the Code: https://github.com/iainporter/rest-java

Password reset is similar to email registration covered in Part Three. The essential parts involve generating a short-lived unique token, emailing it to the user and handling the return of the token.

• User clicks on lost password link
• User enters their email address and submits
• The server generates a short-lived token and sends email to user address with the Base64 encoded token in an embedded link
• User clicks on link (or pastes it into browser window)
• User enters new password which is submitted along with the token to the server
• Server validates the token and password and matches it up to the User
• Password is hashed and saved to User account

The Verification Token

The main properties of a VerificationToken are:

• token - a UUID that is used to identify the token. It is Base64 encoded before being sent
• expiryDate - time to live for the Token. Configured in app.properties
• tokenType - enum (lostPassword, emailVerification, emailRegistration)
• verified - has this token been verified

Verification Token Service

The method for generating and sending the token

    /**
     * generate token if user found otherwise do nothing
     *
     * @param emailAddress
     * @return  a token or null if user not found
     */
    @Transactional
    public VerificationToken sendLostPasswordToken(String emailAddress) {
        Assert.notNull(emailAddress);
        VerificationToken token = null;
        User user = userRepository.findByEmailAddress(emailAddress);
        if (user != null) {
            token = user.getActiveLostPasswordToken();
            if (token == null) {
                token = new VerificationToken(user, VerificationToken.VerificationTokenType.lostPassword,
                        config.getLostPasswordTokenExpiryTimeInMinutes());
                user.addVerificationToken(token);
                userRepository.save(user);
            }
            emailServicesGateway.sendVerificationToken(new EmailServiceTokenModel(user, token, getConfig().getHostNameUrl()));
        }

        return token;
    }

First, find the user by Email Address (line 11). If there is no account matching the address then we don't want to throw an exception but just ignore processing. We could wire in some logic to send an email telling the user that they attempted to change their password but their account does not exist. The main reason for the obfuscation is to prevent malicious trolling of the application to determine if a particular email account is registered.
Once a new token is generated it is passed off for asynchronous processing to the email services gateway.

Email Services Gateway

The service gateway uses Spring Integration to route email tasks. The task is first queued to guarantee delivery and marks the thread boundary of the calling process.

<int:gateway id="emailServicesGateway" service-interface="com.porterhead.rest.gateway.EmailServicesGateway"
                 default-reply-timeout="3000">
        <int:method name="sendVerificationToken" request-channel="emailVerificationRouterChannel"
                    request-timeout="3000"/>
    </int:gateway>

A router polls the queue and routes the email task to the appropriate service.

    <int:channel id="emailVerificationRouterChannel">
        <int:queue capacity="1000" message-store="emailVerificationMessageStore"/>
    </int:channel>

    <int:router id="emailVerificationRouter" input-channel="emailVerificationRouterChannel"
                expression="payload.getTokenType()">
        <int:poller fixed-rate="2000">
            <int:transactional/>
        </int:poller>
        <int:mapping value="emailVerification" channel="emailVerificationTokenSendChannel"/>
        <int:mapping value="emailRegistration" channel="emailRegistrationTokenSendChannel"/>
        <int:mapping value="lostPassword" channel="emailLostPasswordTokenSendChannel"/>
    </int:router>

    <int:channel id="emailLostPasswordTokenSendChannel"/>
    <int:service-activator id="emailLostPasswordSenderService" input-channel="emailLostPasswordTokenSendChannel"
                           output-channel="nullChannel" ref="mailSenderService"
                           method="sendLostPasswordEmail">
    </int:service-activator>

Mail Sender Service

The service loads a velocity template and merges it with the email token model

    public EmailServiceTokenModel sendLostPasswordEmail(final EmailServiceTokenModel emailServiceTokenModel) {
        Map<String, String> resources = new HashMap%lt;String, String>();
         return sendVerificationEmail(emailServiceTokenModel, config.getLostPasswordSubjectText(),
                 "META-INF/velocity/LostPasswordEmail.vm", resources);
    }

When the template has been merged the email is sent using JavaMailSender

    private EmailServiceTokenModel sendVerificationEmail(final EmailServiceTokenModel emailVerificationModel, final String emailSubject,
                                                         final String velocityModel, final Map<String, String> resources) {
        MimeMessagePreparator preparator = new MimeMessagePreparator() {
            public void prepare(MimeMessage mimeMessage) throws Exception {
                MimeMessageHelper messageHelper = new MimeMessageHelper(mimeMessage, MimeMessageHelper.MULTIPART_MODE_RELATED, "UTF-8");
                messageHelper.setTo(emailVerificationModel.getEmailAddress());
                messageHelper.setFrom(config.getEmailFromAddress());
                messageHelper.setReplyTo(config.getEmailReplyToAddress());
                messageHelper.setSubject(emailSubject);
                Map model = new HashMap();
                model.put("model", emailVerificationModel);
                String text = VelocityEngineUtils.mergeTemplateIntoString(velocityEngine, velocityModel, model);
                messageHelper.setText(new String(text.getBytes(), "UTF-8"), true);
                      for(String resourceIdentifier: resources.keySet()) {
                   addInlineResource(messageHelper, resources.get(resourceIdentifier), resourceIdentifier);
                }
            }
        };
        LOG.debug("Sending {} token to : {}",emailVerificationModel.getTokenType().toString(), emailVerificationModel.getEmailAddress());
        this.mailSender.send(preparator);
        return emailVerificationModel;
    }

Password Resource Controller

The controller is a simple pass through to the Verification Token Service. Note that we always return 200 to the client regardless of whether an email address was found or not.

    @PermitAll
    @Path("tokens")
    @POST
    public Response sendEmailToken(LostPasswordRequest request) {
        verificationTokenService.sendLostPasswordToken(request.getEmailAddress());
        return Response.ok().build();
    }

Handling the Reset Request

The email sent to the user should contain a link to a static page so the user can input their new password. This, along with the token, is sent to the server.

    @PermitAll
    @Path("tokens/{token}")
    @POST
    public Response resetPassword(@PathParam("token") String base64EncodedToken, PasswordRequest request) {
        verificationTokenService.resetPassword(base64EncodedToken, request.getPassword());
        return Response.ok().build();
    }

Again this a pass through to the Verification Token service.

    @Transactional
    public VerificationToken resetPassword(String base64EncodedToken, String password) {
        Assert.notNull(base64EncodedToken);
        validate(passwordRequest);
        VerificationToken token = loadToken(base64EncodedToken);
        if (token.isVerified()) {
            throw new AlreadyVerifiedException();
        }
        token.setVerified(true);
        User user = token.getUser();
        user.setHashedPassword(user.hashPassword(password));
        //set user to verified if not already and authenticated role
        user.setVerified(true);
        if (user.hasRole(Role.anonymous)) {
            user.setRole(Role.authenticated);
        }
        userRepository.save(user);
        return token;
    }

The token is matched and if it has already been verified an exception is thrown.
The user's password is hashed and reset.

Testing the API

See Part Three for configuring email settings and spring profiles
Start the application by executing:

gradle tomcatRun

Create a new user with a curl request

curl -v -H "Content-Type: application/json" -X POST -d '{"user":{"firstName":"Foo","lastName":"Bar","emailAddress":"<your email address>"}, "password":"password"}' http://localhost:8080/java-rest/user

Send a password reset request using curl

curl -v -H "Content-Type: application/json" -X POST -d '{"emailAddress":"<your email address>"}' http://localhost:8080/java-rest/password/tokens

Clicking on the link in the email will take you to a static page served from web-app.
Enter a new password and submit or alternatively cut and paste the token and use a curl statement

curl -v -H "Content-Type: application/json" -X POST -d '{"password":"password123"}' http://localhost:8080/java-rest/password/tokens/<your token>

To test that it worked you can use the login page at http://localhost:8080/java-rest/index.html or use curl

curl -v -H "Content-Type: application/json" -X POST -d '{"username":"<your email address>","password":"password123"}' http://localhost:8080/java-rest/user/login

You can also go through the complete cycle using the simple web pages provided.

So far in the series I have covered

• User sign up and login with email
• User sign up and login with OAuth
• Email Verification
• Lost Password

The next posts will focus on accessing role-based resources and session handling.