Monday, January 14, 2013

Writing REST Services in Java: Part 3 Email Verification

Previous Post : Part Two: User sign up and login  Next PostPart Four: Facebook Authentication  
Get the Code: https://github.com/iainporter/rest-java

In this post I'll discuss the process of verifying an email address after a new sign up. The logic flow for this is:

1. Generate a token for the user and persist it
2. Send an email with an embedded link that includes the token to the user
3. User clicks on the link which takes them to a static page
4. The static page calls the API passing the token
5. If the token is valid the user is set to verified
6. Response returned to the user

The Verification Token Service

This service is responsible for generating tokens, persisting them, communicating with the email services gateway and verifying returned tokens. The main properties of a VerificationToken are:
  • token - a UUID that is used to identify the token. It is Base64 encoded before being sent
  • expiryDate - time to live for the Token. Configured in app.properties
  • tokenType - enum (lostPassword, emailVerification, emailRegistration)
  • verified - has this token been verified

Looking at the method to send an Email Registration token:


    @Transactional
    public VerificationToken sendEmailRegistrationToken(String userId) {
        User user = ensureUserIsLoaded(userId);
        VerificationToken token = new VerificationToken(user,
                VerificationToken.VerificationTokenType.emailRegistration,
                config.getEmailRegistrationTokenExpiryTimeInMinutes());
        user.addVerificationToken(token);
        userRepository.save(user);
        emailServicesGateway.sendVerificationToken(new EmailServiceTokenModel(user,
                token, getConfig().getHostNameUrl()));
        return token;
    }

A new token is generated and persisted.
This token is then sent to the EmailServicesGateway where it is queued and the method returns, leaving the gateway to handle processing of the token asynchronously.

The Email Services Gateway

Email services are implemented with Spring Integration. The context file is at:

 src/main/resources/META-INF/spring/email-services-context.xml

The entry point is the Gateway component which routes to a queue-backed channel. This queue has a message store which in dev and local profiles uses a SimpleMessageStore. For staging and production a datasource is used. I'll cover setting up for production in a later post.

    <int:gateway id="emailServicesGateway" 
           service-interface="com.porterhead.rest.gateway.EmailServicesGateway"
           default-reply-timeout="3000">
    <int:method name="sendVerificationToken" 
            request-channel="emailVerificationRouterChannel" request-timeout="3000"/>
    </int:gateway>

    <int:channel id="emailVerificationRouterChannel">
        <int:queue capacity="1000" message-store="emailVerificationMessageStore"/>
    </int:channel>

A router polls the queue and routes to a channel based on the token type:


<int:router id="emailVerificationRouter" input-channel="emailVerificationRouterChannel"
   expression="payload.getTokenType()">
     <int:poller fixed-rate="2000">
        <int:transactional/>
     </int:poller>
     <int:mapping value="emailVerification" 
          channel="emailVerificationTokenSendChannel"/>
     <int:mapping value="emailRegistration"
          channel="emailRegistrationTokenSendChannel"/>
     <int:mapping value="lostPassword" channel="emailLostPasswordTokenSendChannel"/>
</int:router>

From there the appropriate MailSender method is invoked:


<int:service-activator id="emailRegistrationMailSenderService" 
      input-channel="emailRegistrationTokenSendChannel"
      output-channel="nullChannel" ref="mailSenderService"
      method="sendRegistrationEmail">
</int:service-activator>

The Mail Sender Service

The MailSenderService is responsible for constructing mime messages and sending them to their destination. Velocity is used as the template engine. The templates are in:

 src/main/resources/META-INF/velocity

The handler method:

    private EmailServiceTokenModel sendVerificationEmail(
                final EmailServiceTokenModel emailVerificationModel,
                final String emailSubject, 
                final String velocityModel, 
                final Map<String, String> resources) {
        MimeMessagePreparator preparator = new MimeMessagePreparator() {
            public void prepare(MimeMessage mimeMessage) throws Exception {
              MimeMessageHelper messageHelper = new MimeMessageHelper(mimeMessage,
              MimeMessageHelper.MULTIPART_MODE_RELATED, "UTF-8");
              messageHelper.setTo(emailVerificationModel.getEmailAddress());
              messageHelper.setFrom(config.getEmailFromAddress());
              messageHelper.setReplyTo(config.getEmailReplyToAddress());
              messageHelper.setSubject(emailSubject);
              Map model = new HashMap();
              model.put("model", emailVerificationModel);
              String text = 
                 VelocityEngineUtils.mergeTemplateIntoString(velocityEngine, 
                 velocityModel, model);
                messageHelper.setText(new String(text.getBytes(), "UTF-8"), true);
                    for(String resourceIdentifier: resources.keySet()) {
                       addInlineResource(messageHelper, 
                       resources.get(resourceIdentifier),
                      resourceIdentifier);
                    }
              }
          };
        LOG.debug("Sending Verification Email to : {}",
                   emailVerificationModel.getEmailAddress());
        this.mailSender.send(preparator);
        return emailVerificationModel;
    }
The velocity templates are just bare-bones and will need to be customised to your needs.

Handling the Verification Request


The controller method passes through to the Verification Token Service

@PermitAll
    @Path("tokens/{token}")
    @POST
    public Response verifyToken(@PathParam("token") String token) {
        verificationTokenService.verify(token);
        return Response.ok().build();
    }


The service method to verify the token

    @Transactional
    public VerificationToken verify(String base64EncodedToken) {
        VerificationToken token = loadToken(base64EncodedToken);
        if (token.isVerified() || token.getUser().isVerified()) {
            throw new AlreadyVerifiedException();
        }
        token.setVerified(true);
        token.getUser().setVerified(true);
        userRepository.save(token.getUser());
        return token;
    }

    private VerificationToken loadToken(String base64EncodedToken) {
        Assert.notNull(base64EncodedToken);
        String rawToken = new String(Base64.decodeBase64(base64EncodedToken));
        VerificationToken token = tokenRepository.findByToken(rawToken);
        if (token == null) {
            throw new TokenNotFoundException();
        }
        if (token.hasExpired()) {
            throw new TokenHasExpiredException();
        }
        return token;
    }


The token is decoded and matched from the repository to an existing user. If that user has not already been verified then they are set to verified.

Testing the API

Before we can test the API the email service has to be configured. The context file containing the MailSender bean is at:

src/main/resources/META-INF/spring/email-template-context.xml

Choose a Mail Sender and insert the values into the mail properties. You can use a gmail or yahoo account for testing or there are several useful bulk mail sending options such as CritSend or MailJet.

<beans profile="production, staging">
    <bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl">
      <!-- Email provider here-->
      <property name="host" value="<insert your Host name here>"/>
      <property name="port" value="<insert the port here>"/>
      <property name="username" value="<insert your username here>"/>
      <property name="password" value="<insert your password>"/>

        <property name="javaMailProperties">
          <props>
           <prop key="mail.debug">false</prop>
           <prop key="mail.smtp.auth">true</prop>
           <prop key="mail.smtp.starttls.enable">true</prop>
          </props>
        </property>
   </bean>

As you can see only the staging and production profiles are set up to send real mail. When running the dev and local profiles the MockJavaMailSender is invoked which simply stores the Mime messages in memory. To test out sending mail we can temporarily add the dev profile:

<beans profile="dev, production, staging">

and remove it from the other beans profile:

<beans profile="local">

Now start the application by executing:

gradle tomcatRun

Create a new user with the following curl statement (substituting in your real email address):

curl -v -H "Content-Type: application/json" -X POST -d '{"user":{"firstName":"Foo","lastName":"Bar","emailAddress":"<your email address>"}, "password":"password"}' http://localhost:8080/java-rest/user

You should then receive an email with the subject:
"Welcome to the Java-REST sample application"
This is configurable in app.properties although that will be covered in a later post.

You can either click on the link in the email and this will invoke a static page in the application that will forward on the token to another API call to verify the token. Alternatively you can copy the token in the link and construct a curl statement to do the same thing.

A sample curl to verify the token (substitute the token in your email):

curl -v -H "Content-Type: application/json"  -X POST -d  localhost http://localhost:8080/java-rest/verify/tokens/NjM1NjBkNTAtNjEwZi00N2I3LTk0MmQtYWMwMjVkY2MwNTc1

You should see this response:

< HTTP/1.1 200 OK
< Server: Apache-Coyote/1.1
< Content-Type: application/json
< Content-Length: 0
< Date: Mon, 14 Jan 2013 08:30:18 GMT

If you execute the curl again you will receive a 409 response with a message saying the token has already been verified:

< HTTP/1.1 409 Conflict
< Server: Apache-Coyote/1.1
< Content-Type: application/json
< Transfer-Encoding: chunked
< Date: Mon, 14 Jan 2013 08:30:58 GMT
{"errorCode":"40905","consumerMessage":"Already verified","applicationMessage":"The token has already been verified"}

What if a user changes their email address? In that case we should resend a verification email and have them verify their email address again. In the user/<id> PUT resource handler that is exactly what happens:

    @RolesAllowed({"authenticated"})
    @Path("{userId}")
    @PUT
    public Response updateUser(@Context SecurityContext sc, @PathParam("userId") 
                 String userId, UpdateUserRequest request) {
        ExternalUser userMakingRequest = (ExternalUser)sc.getUserPrincipal();
        if(!userMakingRequest.getId().equals(userId)) {
            throw new AuthorizationException("User not authorized to 
               modify this profile");
        }
        boolean sendVerificationToken = 
                 StringUtils.hasLength(request.getEmailAddress()) &&
                !request.getEmailAddress().equals(userMakingRequest.getEmailAddress());
        ExternalUser savedUser = userService.saveUser(userId, request);
        if(sendVerificationToken) {
            verificationTokenService.sendEmailVerificationToken(savedUser.getId());
        }
        return Response.ok().build();
    }
This is an access controlled resource which will be covered in a Part six: security and authorization.

The next post will cover Facebook authentication.






24 comments:

  1. Hi, thanks for all the posts. I'm waiting for the remaining parts.

    ReplyDelete
  2. Should have Part Four up this week

    ReplyDelete
  3. Can you show us an example with google api?

    ReplyDelete
  4. Hello Mr Porter,

    Nice blog! I am editor at Java Code Geeks (www.javacodegeeks.com). We have the JCG program (see www.javacodegeeks.com/join-us/jcg/), that I think you’d be perfect for.

    If you’re interested, send me an email to eleftheria[dot]kiourtzoglou[at]javacodegeeks[dot]com and we can discuss further.

    Best regards,
    Eleftheria Kiourtzoglou

    ReplyDelete
  5. Hi Iain, after signup, is it expected that a user can still login even though he has not yet clicked on the email confirmation link ? (i.e. token is not verified)

    ReplyDelete
    Replies
    1. It depends on the rules of the site. The point of verification is that the site can trust that the user is who they say they are and the email account is one that they have access to.

      For example a site may restrict access to some areas until the user has verified their account.
      Or perhaps they are given a certain period of time to verify their account after which it will be frozen.

      If they have verified their account then you can be confident that other features that depend on a valid email address (Lost Password, etc) will work correctly.

      Delete
    2. Ok understood. Is there an api in your code that would fail if the user is not verified ? (as an example)

      Delete
    3. How about something like this:

      @RolesAllowed("authenticated")
      @GET
      public Response getSpecialOffer(@Context SecurityContext sc) {
      ExternalUser user = (ExternalUser)sc.getUserPrincipal();
      if(!user.isVerified()) {
      throw new AuthorizationException("User not authorized: Account not verified");
      } else {
      return Response.ok().entity(getSpecialOffers(user)).build();
      }

      Delete
  6. Nice tutorial. One question: When a user registers, if mailSender.send(preparator) fails for some reason, the main flow will still continue but user won't be able to register or change password. How can we handle a scenario like this? Thanks.

    ReplyDelete
    Replies
    1. That is a tricky one and it is one reason why it's useful to support email verification.
      You can't prevent the user from entering a wrong email address but you can limit the access and support that you provide them until they verify their address.

      If they did enter their address wrong and subsequently forget their password then the only recourse is to create a new account.

      Delete
    2. I think what you were probably asking is "what if there is some failure to send the message"

      In that case as long as an exception is thrown the message will go back on the queue for redelivery (as long as the Spring Integration queue is backed by a persistent store).

      You can also automate a process that emails users that have not registered periodically to remind them of the benefits of verifying their account.

      Delete
  7. Hello,

    I'm trying to run the code in production mode but I have some issues while following the instructions from the Readme.md hosted on GitHub.

    Point 2 refers to src/main/resources/properties/application.properties which doesn't initially exist but is generated after running "./gradlew tomcatRun" for the first time (maybe its a note which can be added to Readme.md).

    In your project "rest-java", the Spring profile (mainly dev, local, and production) mode was set in gradle.properties, which doesn't seem to be used anymore. What is the way used now to set the profile mode (I'd like to run the test suite in production mode)?

    Point 8 says to use the argument "-Dspring.profiles.active=production" to run the war file in production mode. How can I run the project in production mode using gradlew ("./gradlew tomcatRun -Dspring.profiles.active=production" throws the following exception, too long to be included in this 4096 char-limited message)?

    Thanks for your help and time

    ReplyDelete
    Replies
    1. Just to avoid confusion, are you referring to the ReadMe for this project: https://github.com/iainporter/oauth2-provider or https://github.com/iainporter/rest-java?

      There are a few changes between projects in how they use properties and profiles.

      I was able to execute ./gradlew tomcatRun -Dspring.active.profiles=production

      Can you send me the first few lines of the root exception in the stacktrace

      Delete
    2. Yes, I was referring to the Readme of this project (oauth2-provider).

      I don't know what changed since then by I can now successfully run the project in production mode. :)

      ./gradlew tomcatRun -Dspring.active.profiles=production

      However, I'm still unable to receive email after requesting a password reset.

      app-production.properties:
      email.services.sender.host=smtp.gmail.com
      email.services.sender.port=587
      email.services.sender.username=
      email.services.sender.password=

      When I add a new user, the following message is displayed in the console:
      The Server is running at http://localhost:8080/oauth2-provider
      logbak: 12:20:30.711 com.porterhead.user.UserService - Validating user request.
      logbak: 12:20:30.831 com.porterhead.user.UserService - User does not already exist in the data store - creating a new user [clement.schaffter@gmail.com].

      However the console doesn't print anything when I submit an email address to which an email must be sent to reset the password.

      Here is a partial log which may be of interest. By the way, there is no reference to app-production.properties in the entire log.

      logbak: 12:29:15.400 o.s.c.a.ConfigurationClassBeanDefinitionReader - Skipping bean definition for [BeanMethod:name=healthCheckResource,declaringClass=com.porterhead.configuration.SupportConfiguration]: a definition for bean 'healthCheckResource' already exists. This top-level bean definition is considered as an override.
      logbak: 12:29:15.400 o.s.c.a.ConfigurationClassBeanDefinitionReader - Skipping bean definition for [BeanMethod:name=verificationTokenService,declaringClass=com.porterhead.configuration.UserConfiguration]: a definition for bean 'verificationTokenService' already exists. This top-level bean definition is considered as an override.
      logbak: 12:29:15.401 o.s.c.a.ConfigurationClassBeanDefinitionReader - Skipping bean definition for [BeanMethod:name=userResource,declaringClass=com.porterhead.configuration.UserConfiguration]: a definition for bean 'userResource' already exists. This top-level bean definition is considered as an override.
      logbak: 12:29:15.401 o.s.c.a.ConfigurationClassBeanDefinitionReader - Skipping bean definition for [BeanMethod:name=passwordResource,declaringClass=com.porterhead.configuration.UserConfiguration]: a definition for bean 'passwordResource' already exists. This top-level bean definition is considered as an override.
      logbak: 12:29:15.402 o.s.c.a.ConfigurationClassBeanDefinitionReader - Skipping bean definition for [BeanMethod:name=verificationResource,declaringClass=com.porterhead.configuration.UserConfiguration]: a definition for bean 'verificationResource' already exists. This top-level bean definition is considered as an override.
      logbak: 12:29:15.402 o.s.c.a.ConfigurationClassBeanDefinitionReader - Skipping bean definition for [BeanMethod:name=meResource,declaringClass=com.porterhead.configuration.UserConfiguration]: a definition for bean 'meResource' already exists. This top-level bean definition is considered as an override.
      logbak: 12:29:15.715 o.s.c.a.ConfigurationClassEnhancer - @Bean method PropertiesConfiguration.getProperties is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as @Autowired, @Resource and @PostConstruct within the method's declaring @Configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see @Bean Javadoc for complete details
      ...
      logbak: 12:29:15.767 o.s.c.s.PropertySourcesPlaceholderConfigurer - Loading properties file from file [/home/tschaffter/devel/java/oauth2-provider]
      logbak: 12:29:15.770 o.s.c.s.PropertySourcesPlaceholderConfigurer - Could not load properties from file [/home/tschaffter/devel/java/oauth2-provider]: (No such file or directory)

      Delete
    3. Yes, the readme of oauth2-provider.

      I don't know what changed but now I can run the project in production mode with your command. However, I still don't receive any email when requesting a password reset.

      email.services.sender.host=smtp.gmail.com
      email.services.sender.port=587
      email.services.sender.username=EMAIL_ADDRESS
      email.services.sender.password=PASSWORD

      After submitting the password reset request, nothing is displayed in the terminal running tomcatRun.

      By the way, there is no reference to app-production.properties being loaded in the entire log displayed when starting tomcatRun. Is that fine?

      Delete
  8. EDIT: I didn't forget to set username and password of the mail account

    ReplyDelete
    Replies
    1. ah, I often get the syntax wrong.

      It should be -Dspring.profiles.active=production not -Dspring.active.profiles=production

      also you will need to pass the mongoDB info as such:

      ./gradlew tomcatRun -Dspring.profiles.active=production -DMONGODB_HOST=localhost -DMONGODB_PORT=27017

      Delete
    2. That was it. The provider now works fine in production mode!

      Delete
  9. Hi, Thanks so much for your helpful article! It is getting me started with a User accounts system on Spring Framework, very interesting.
    I have imported your User functionality into an existing maven project of mine, everything works fine. When I try to deploy however, I am getting errors caused by the /email-services-context.xml file. I think the cause is due to an incorrect Jar version on the classpath, but I am not sure and I cannot find anything online.

    The errors are below:
    Line 18: Element 'int:gateway' must have no character or element information item [children], because the type's content type is empty.
    Line 21: Attribute 'message-store' is not allowed to appear in element 'int:queue'.
    Line 25: Attribute 'expression' is not allowed to appear in element 'int:router'.
    Line 26: Attribute 'fixed-rate' is not allowed to appear in element 'int:poller'.
    Line 29: Invalid content was found starting with element 'int:mapping'. One of '{"http://www.springframework.org/schema/beans":bean}' is expected.

    Any input is greatly appreciated as I am still a beginner in using Spring!
    Once again thank you so much for your helpful blog.

    ReplyDelete
    Replies
    1. It has been a while since I touched that project.
      I have updated the Spring dependencies.
      Pull the code and try again.

      Delete
    2. Thanks for your quick reply, it was to do with my maven dependencies. Jersey-server.jar includes spring-3.0.0 so I had to exclude those in my pom.xml. Is there a reason this project has gone untouched for a while? I would be interested in any suggestions you have of new(er) frameworks.

      Converting this project into a Maven project is becoming quite an arduous task, now I believe the data-context.xml is not being correctly implemented on server start up. I am seeing an error in tomcat logs :

      Error creating bean with name 'userRepository': Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: Validation failed for query for method public abstract user.domain.User user.UserRepository.findByUuid(java.lang.String)!

      Any suggestions would be greatly appreciated as I am lacking any support apart from stackoverflow for this project. Thanks again!

      Delete
    3. A lot of the the code in that project is still useful but has been superseded by this project: https://github.com/iainporter/oauth2-provider

      If you have ported this project over to your own maven project then the exceptions may be down to the wrong version of Spring data. I would need to see your pom file to comment further.

      Delete
  10. This comment has been removed by a blog administrator.

    ReplyDelete