Jun 26, 2016

Google recently has deprecated the Google+ Sign in and process of obtaining oauth access tokens viaGoogleAuthUtil.getToken API. Now, they reccomend a single entry point via new Google Sign-In API. The major reasons for doing so are 1. It enhances user experience and 2. It improves security, more here. Also starting with android 6.0, the GET_ACCOUNTS permission has to be requested at runtime and if you implement this API, it eliminates the need for requiring this permission.
The feature that is really exciting is that it introduces new silentSignIn API, which allows for cross device silent sign in (essentially if a user has signed into your application on another platform, he won't be shown the sign in prompt) provided that the requested scopes are same, so this improves the user experience. In addition, you don't have to use the GoogleAuthUtil.getToken API to obtain the tokens as they are granted on the initial sign-in.
So if you have an android application in which you had previously implemented the Google+ sign in and used other Google plus features and want to migrate your android applications to new Google Sign in implementation, this post explains how to do so. Depending upon whether you choose to automate the lifecycle for GoogleAPIClient (Use enableAutoManage, this approach is recommended as it avoids boilerplate code) or manage the lifecycle for GoogleAPIClient by implementing the ConnectionCallbacks interface, the code might slightly differ. But, as the latter approach requires a bit more code, I will explain the process using it.

What needs to be changed

  • Replace mGoogleApiClient.connect() with mGoogleApiClient.connect(GoogleApiClient.SIGN_IN_MODE_OPTIONAL), this is basically required to allow the client to transition between authenticated and unauthenticated states and for use with GoogleSignInApi.
  • Build a GoogleSignInOptions instance. While building the instance, request the additional scopes via requestScopes method,( this is where you can request scopes such as SCOPE_PLUS_LOGIN andSCOPE_PLUS_PROFILE). Also, if you need to authenticate the user with the backend and want to obtain the authorization token to access the API's using your backend use requestIdToken(serverToken) andrequestServerAuthCode(serverToken) methods. Here unlike Google plus sign in the serverToken is just the clientId of the web application.
  • Build the GoogleApiClient instance, use the addApi method to add the Auth.GOOGLE_SIGN_IN_API andPlus.API.
  • In the onStart method connect the client usingmGoogleApiClient.connect(GoogleApiClient.SIGN_IN_MODE_OPTIONAL) and in onStop method disconnect the client. (You may do this in onResume and onPause methods also).
  • After the client is connected, first attempt the silentSignIn and if it fails with code SIGN_IN_REQUIRED, attempt to do a fresh sign in for the user.
  • After the sign in is completed, then you can invoke the Plus.PeopleApi with user accountId to obtain users google profile information.
  • To Sign out the user use Auth.GoogleSignInApi.signOut method and to revoke access useAuth.GoogleSignInApi.revokeAccess method.
  • Remove the android.permission.GET_ACCOUNTS permission from android manifest.

Here's the relevant code.



 @Override
  protected void onCreate(Bundle savedInstanceState) {
  ..
  //Here unlike Google plus sign in the serverToken is just the clientId of the web application
  GoogleSignInOptions gso = new GoogleSignInOptions.Builder(GoogleSignInOptions.DEFAULT_SIGN_IN)
          .requestIdToken(serverToken).requestServerAuthCode(serverToken).
                  requestEmail().
                  requestScopes(Plus.SCOPE_PLUS_LOGIN, Plus.SCOPE_PLUS_PROFILE)
          .build();
  mGoogleApiClient = new GoogleApiClient.Builder(this)
          .addConnectionCallbacks(this)
          .addOnConnectionFailedListener(this).
                  addApi(Auth.GOOGLE_SIGN_IN_API, gso).
                  addApi(Plus.API)
          .build();
  }

  protected void onResume() {

      super.onResume();

      //Here isConnected is just a flag that checks whether user is connected to internet.
      /**To avoid execution of this block you can check whether user previously signedIn on this device by storing a userIdToken and checking whether user needs to be signedIn automatically or not. */
      if (!mGoogleApiClient.isConnecting() && !mGoogleApiClient.isConnected() && isConnected) {
              mGoogleApiClient.connect(GoogleApiClient.SIGN_IN_MODE_OPTIONAL);
          } //Here isSignedIn is a boolean flag that tracks whether the user is signedIn or not.
          else if(isConnected&&mGoogleApiClient.isConnected()&&!isSignedIn){
              signInUsingNewAPI();
          }

  }
  private void signInUsingNewAPI() {
    if (!isSignedIn&&isConnected) {
               attemptSilentSignIn();

           }
  }

  private void attemptSilentSignIn(){
      OptionalPendingResult<GoogleSignInResult> opr = Auth.GoogleSignInApi.silentSignIn(mGoogleApiClient);
      if (opr.isDone()) {
          // If the user's cached credentials are valid, the OptionalPendingResult will be "done"
          // and the GoogleSignInResult will be available instantly.
          Log.d(TAG, "Got cached sign-in");
          GoogleSignInResult result = opr.get();
          handleSignInResult(result);
      } else {
          // If the user has not previously signed in on this device or the sign-in has expired,
          // this asynchronous branch will attempt to sign in the user silently.  Cross-device
          // single sign-on will occur in this branch.
          showProgressDialog();
          opr.setResultCallback(new ResultCallback<GoogleSignInResult>() {
              @Override
              public void onResult(GoogleSignInResult googleSignInResult) {
                  hideProgressDialog();
                  handleSignInResult(googleSignInResult);
              }
          });
      }
  }

  private void handleSignInResult(GoogleSignInResult result){
    if (!result.getStatus().isSuccess()) {
          isSignedIn = false;
          mIntentInProgress = false;
          if(result.getStatus().hasResolution()||result.getStatus().getStatusCode()== CommonStatusCodes.SIGN_IN_REQUIRED){
                freshSignIn(); //Rather than using startResolutionForResult, we invoke our method which attempts to do a fresh sign in and if there is error it is handled in onActivityResult method.
              }

      }
      else {
        mIntentInProgress = false;
        isSignedIn = true;
        final GoogleSignInAccount account = result.getSignInAccount();
        //Maybe save this result.
        SharedPreferences.Editor editor = preferences.edit();
        editor.putString("client_id_token", account.getIdToken());
        editor.putString("auth_code",account.getServerAuthCode());
        editor.apply();
        //You can pass these credentials to your server from here.
        //Invoke the GPlus People API
        Plus.PeopleApi.load(mGoogleApiClient, account.getId()).setResultCallback(new ResultCallback<People.LoadPeopleResult>() {
                  @Override
                  public void onResult(@NonNull People.LoadPeopleResult loadPeopleResult) {
                      Person person = loadPeopleResult.getPersonBuffer().get(0);
                      //Method that obtains the userInfo
                      getProfileInfo(person, account.getEmail());
                  }
              });

      }

  }
  private void freshSignIn(){
      Intent signInIntent = Auth.GoogleSignInApi.getSignInIntent(mGoogleApiClient);
      showProgressDialog();
      startActivityForResult(signInIntent, RC_SIGN_IN);
  }
  @Override
  protected void onActivityResult(int requestCode, int responseCode,
                                  Intent intent) {

      if(requestCode==RC_RESOLVE_ERROR){
          mIntentInProgress = false;
          if (responseCode != RESULT_OK) {
             isSignedIn = false;
             //Maybe show a dialog to user ?
             return;
        }
        //Attemp connection again.
        if (!mGoogleApiClient.isConnecting()) {
              mGoogleApiClient.connect();
          }
        }
        else if (requestCode == RC_SIGN_IN) {
          hideProgressDialog();
          GoogleSignInResult result = Auth.GoogleSignInApi.getSignInResultFromIntent(intent);
          handleSignInResult(result,false);
      }
    }
  // Connection callbacks
  @Override
  public void onConnected(Bundle bundle) {
      if(!isSignedIn) {
          signInUsingNewAPI();
      }
  }

  @Override
  public void onConnectionSuspended(int i) {
    isSignedIn = false;
    if (!isSignedIn&&isConnected) {
    mGoogleApiClient.connect(GoogleApiClient.SIGN_IN_MODE_OPTIONAL);
  }
  }
  @Override
  public void onConnectionFailed(ConnectionResult result) {

      if (!result.hasResolution()) {
          GoogleApiAvailability.getInstance().getErrorDialog(
                  this, result.getErrorCode(), RC_SIGN_IN).show();
          return;
      }

      if (!mIntentInProgress) {
          // Store the ConnectionResult for later usage
          mConnectionResult = result;

          if (!isSignedIn) {
              // The user has already clicked 'sign-in' so we attempt to
              // resolve all
              // errors until the user is signed in, or they cancel.
              resolveSignInError();
          }
      }
  }
/**
 * Method to resolve any signin errors
 */
private void resolveSignInError() {
    if (mConnectionResult.hasResolution()) {
        try {
            mIntentInProgress = true;
            mConnectionResult.startResolutionForResult(this, RC_RESOLVE_ERROR);
        } catch (IntentSender.SendIntentException e) {
            mIntentInProgress = false;
            mGoogleApiClient.connect(GoogleApiClient.SIGN_IN_MODE_OPTIONAL);
        }
    }
}

//Sign out and revoke methods
/**
  * Sign-out from google
  */
 public void signOutFromGoogle() {

     if (mGoogleApiClient.isConnected()) {
         Auth.GoogleSignInApi.signOut(mGoogleApiClient).setResultCallback(
                 new ResultCallback<Status>() {
                     @Override
                     public void onResult(Status status) {
                       isSignedIn = false;
                       //do other stuff here.
                       mGoogleApiClient.disconnect();
                       //Builds a fresh instance of GoogleApiClient
                       buildGoogleApiClient();
                     }
             });
  }
  }

  /**
   * Revoking access from google
   */
  public void revokeGplusAccess() {
      if (mGoogleApiClient.isConnected()) {
          Auth.GoogleSignInApi.revokeAccess(mGoogleApiClient).setResultCallback(
                  new ResultCallback<Status>() {
                      @Override
                      public void onResult(Status status) {
                        isSignedIn = false;
                        //do other stuff here.
                        mGoogleApiClient.disconnect();
                        //Builds a fresh instance of GoogleApiClient
                        buildGoogleApiClient();
                        //You can inform your server of this change
                    }
               });

   }
   }
 //Other utility methods
 private void hideProgressDialog() {
     if (mProgressDialog != null && mProgressDialog.isShowing()) {
         mProgressDialog.hide();
     }
 }

 private void showProgressDialog() {
  if (mProgressDialog == null) {
      mProgressDialog = new ProgressDialog(this);
      mProgressDialog.setMessage("Signing In");
      mProgressDialog.setIndeterminate(true);
  }

  mProgressDialog.show();
 }
Hope this code is helpful in helping you move to the new Google sign in implementation.
app_google_sign_in_image

Posted on Sunday, June 26, 2016 by Unknown

Hive or Impala ?

Hive and Impala both support SQL operation, but the performance of Impala is far superior than that ofHive. Although now with Spark SQL engine and use of HiveContext the performance of hive queries is also significantly fast, impala still has a better performance. The reason that impala has better performance is that it already has daemons running on the worker nodes and thus it avoids the overhead that is incurred during the creation of map and reduce jobs.
The query that I will mention later ran almost 10X faster on impala than on Hive (61 seconds vs around 600 seconds): Impala is known to give even better performance.

Schema on read vs Schema on write

Schema on read differs from schema on write as data is not validated till it is read. Although schema on read offers flexibility of defining multiple schemas for the same data, it can cause nasty runtime errors. As an example Hive and Impala are very particular about the timestamp format that they recognize and support, one workaround to avoid such bad records is to use a trick where rather than specifying the data type as timestamp, you specify the datatype as String and then use the cast operator to transform the records to timestamp format, this way bad records are skipped and the query does not error out.

cast(field_name as timestamp)

Window Functions, Top-N Queries, PL/SQL

Hive and Impala do not support update queries, but they do support select * from insert intooperation. Hive and impala also support window functions. The latter makes life easier because both Impala and Hive do not support PL/SQL procedures.
In the example below, I am using the dataset of NYC Yellow Taxi from the month of January 2015. The query below filters out invalid timestamp records and selects first 500 records per hour for 1st january 2015.

/**Top-N Subquery selects first 500 records per hour for a day*/
insert into nyc_taxi_data_limited  select VendorID, tpep_pickup_datetime , tpep_dropoff_datetime , passenger_count ,trip_distance ,pickup_longitude ,pickup_latitude,RateCodeID ,store_and_fwd_flag  ,dropoff_longitude ,dropoff_latitude ,payment_type ,fare_amount ,extra,mta_tax ,tip_amount,tolls_amount,improvement_surcharge,total_amount from ( select *,
row_number() over (partition by trunc(cast(tpep_pickup_datetime as timestamp), 'HH') order by trunc(cast(tpep_pickup_datetime as timestamp), 'HH') desc)
as rownumb from nyc_taxi_data where cast(tpep_pickup_datetime as timestamp) between cast('2015-01-01 00:00:00' as timestamp) and cast('2015-01-01 23:59:59' as timestamp)
) as q  where rownumb<=500;

Note the use of window function row_number and ordering by truncated timestamp, and cast operator to avoid invalid records.

What's the catch ?

Given the benefits of Impala why would one ever use Hive ? The answer lies in the fact that impala queries are not fault tolerant.

Conclusion

Although, Impala and Hive do not offer entire repertoire of functionality supported by traditional RDBMS's, they are closest wrt to functionality offered by traditional RDBMS's in the world of distributed systems and offer scalable and large scale data analysis capability.

Posted on Sunday, June 26, 2016 by Unknown

The Idea

Java 8 introduced functional programming support, this is a powerful feature which was missing from earlier versions. One of the benefits of functional programming is that it can be used to implement decorator pattern easily. One common requirement is to implement some kind of rate limiting for web services. Now, ideally you would want separation of concerns between the actual business logic and rate limitation logic. With Java 8, we can use function references to implement this separation of concerns and implement the decorator pattern.

The code

The code fragment below shows the implementation of the pattern. It is an example of integration with the Lyft API. The full source code is available here.

/**
 * Generic method which can invoke any function without applying rate limit
 *
 * @param method    the function to invoke or apply the each map input to
 * @param inputList The list of Maps, each of which contains the key value pair of service parameters
 * @param <R>       Generic Return object type in the list
 * @param <K>       Type of Key in Map
 * @param <V>       Type of Value in Map
 * @return A list with object type <V>
 */
private <R, K, V> List<R> invokeWithoutRateLimit(Function<Map, R> method, List<Map<K, V>> inputList) {
    List<R> returnList = new ArrayList<>();
    inputList.stream().forEach(m -> {
        returnList.add(method.apply(m));
    });
    return returnList;
}

/**
 * Generic method which can invoke any function with applying rate limit
 * It uses RxJava and Blocking invocation
 *
 * @param method    the function to invoke or apply the each map input to
 * @param inputList The list of Maps, each of which contains the key value pair of service parameters
 * @param <R>       Generic Return object type in the list
 * @param <K>       Type of Key in Map
 * @param <V>       Type of Value in Map
 * @return A list with object type <V>
 */
private <R, K, V> List<R> invokeWithRateLimit(Function<Map, R> method, List<Map<K, V>> inputList) {
    List<R> returnList = new ArrayList<>();
    Observable.zip(Observable.from(inputList),
            Observable.interval(RATE_LIMIT, TimeUnit.SECONDS), (obs, timer) -> obs)
            .doOnNext(item -> {
                        R result = method.apply(item);
                        returnList.add(result);
                    }
            ).toList().toBlocking().first();
    return returnList;
}

/**
 * This method accepts a list of coordinates and returns the estimated
 * fare for different lyft rides
 *
 * @param costRequestList     The list of coordinates
 * @param invokeWithRateLimit Apply Rate limiting
 * @return A list of Prices per request
 */
public List<CostEstimates> getCostEstimates(List<Map<String, Float>> costRequestList, boolean invokeWithRateLimit) {
    if (invokeWithRateLimit) {
        return invokeWithRateLimit(this::getCostEstimate, costRequestList);
    } else {
        return invokeWithoutRateLimit(this::getCostEstimate, costRequestList);
    }


}
The highlighted code above shows how to pass method reference to methods invokeWithRateLimit() andinvokeWithoutRateLimit(), each of these methods then adds some custom preprocessing logic (like rate limitation using RxJava) after which it invokes the supplied method by using the apply() method. This implementation of the decorator pattern is much easier to grasp, than going via the inheritance route.
You can use the following link to view the entire code on Github repository.
Lyft-Client on Github.

Posted on Sunday, June 26, 2016 by Unknown

You might run into a scenario where you might require conditional authentication with Retrofit 2.0.
This post provides an example of integration with the Lyft API. In case of the Lyft API, first we need to authenticate with and query the oauth/token endpoint to obtain the OAUTH token, and then use thisaccessToken in other service calls. Also, such access tokens have an expiry time(1 hour), so ideally there should be a mechanism to handle this scenario.
One lazy (tends out to be perfect) solution is to use interceptors and compare the HTTP Response codefrom the service to see whether the code is 401. If the code is 401, you can assume that the token has either expired or was never obtained initially, either way you would need to re-authenticate and query the endpoint to obtain the accessToken.
The code block below shows how this is done. To access the entire source code you can visit Lyft-Client on Github.

/**
 * This method initializes the retrofit clients
 * a) One for the initial authentication end point
 * b) Other for other service requests
 */
private void initializeRetrofitClients() {
    OkHttpClient.Builder builder = new OkHttpClient().newBuilder();
    OkHttpClient clientNormal;
    OkHttpClient clientAuthenticated;
    builder.interceptors().add(new Interceptor() {
        @Override
        public okhttp3.Response intercept(Chain chain) throws IOException {
            Request originalRequest = chain.request();
            Request.Builder builder = originalRequest.newBuilder().header("Authorization:Bearer ", accessToken).
                    method(originalRequest.method(), originalRequest.body());
            okhttp3.Response response = chain.proceed(builder.build());
            /**
            implies that the token has expired
            or was never initialized
             */
            if (response.code() == 401) {
                tokenExpired = true;
                logger.info("Token Expired");
                getAuthenticationToken();
                builder = originalRequest.newBuilder().header("Authorization:Bearer ", accessToken).
                        method(originalRequest.method(), originalRequest.body());
                response = chain.proceed(builder.build());
            }
            return response;
        }
    });
    clientAuthenticated = builder.build();
    retrofitAuthenticated = new Retrofit.Builder().client(clientAuthenticated)
            .baseUrl(API_ENDPOINT)
            .addConverterFactory(GsonConverterFactory.create())
            .build();
    OkHttpClient.Builder builder1 = new OkHttpClient().newBuilder();
    builder1.authenticator(new Authenticator() {
        @Override
        public Request authenticate(Route route, okhttp3.Response response) throws IOException {
            String authentication = Credentials.basic(CLIENT_ID, CLIENT_SECRET);
            Request.Builder builder = response.request().newBuilder().addHeader("Authorization", authentication);
            return builder.build();
        }
    });
    clientNormal = builder1.build();
    retrofit = new Retrofit.Builder().client(clientNormal).
            baseUrl(API_ENDPOINT).
            addConverterFactory(GsonConverterFactory.create()).build();
}
/**
 * Is invoked only when the access token is required
 * Or it expires
 */
private void getAuthenticationToken() {
    LyftService lyftService = this.retrofit.create(LyftService.class);
    Call<OAuthResponse> authRequestCall = lyftService.getAccessToken(oAuthRequest);
    Response<OAuthResponse> response = null;
    try {
        response = authRequestCall.execute();
        if (response.isSuccessful()) {
            accessToken = response.body().getAccessToken();
        }
    } catch (IOException e) {
        logger.error("Exception occurred due to ", e);
    }

}
As can be seen in the above code example, we build two OkHttpClient objects, the clientNormal object is configured to use HTTP basic authentication and is used by retrofit object to query the getAccessTokenendpoint to obtain the access token, this accessToken is required by other Lyft service endpoints. TheclientAuthenticated object uses a interceptor to set the Authorization:Bearer header with the value of theaccessToken, which is required for all other service endpoints.
In the method initializeRetrofitClients it can be seen that initially, we just invoke the service endpoint by using the value of accessToken (by calling chain.proceed) and if we see that the response code is 401, we invoke getAuthenticationToken followed by another call to chain.proceed with the new value of accessToken. For subsequent calls, the interceptor will use the stored value of accessToken. This lazy invocation to obtain the access token is better because this way the logic for deciding when to obtain accessToken is not hardcoded. In addition, this keeps the code simple by avoiding unncessary checks.
Hope this post was helpful in clearing the use of interceptors for conditional authentication.
You can use the following link to view the entire code on Github repository.
Lyft-Client on Github.

Posted on Sunday, June 26, 2016 by Unknown

Mar 11, 2016

Well blogger does not have support for latex and the windows live writer is being redeveloped. So, in the meanwhile, I have written a few posts on the pelican blog and thought, I might as well link to them here.

Why you should prefer to use the square root of Gini Index: This post examines the advantages of using the Gini Index as the criteria for building decision trees.
http://orastack.com/why-you-should-use-square-root-of-gini-index.html

Do tweets have predictive power. This post examines whether tweets have an effect on opening weekend revenue of box office movies.
http://orastack.com/scikit-learn_tweet_classifier.html

Posted on Friday, March 11, 2016 by Unknown

Aug 5, 2015

Earlier, I had covered an example here, which showed how to dynamically create users and map the application roles to enterprise groups. In this post, the sample application is extended to show how you can query the application roles from the application stripe(application specific policies).
To query the application specific roles, you need to access the application's policy from policystore and then you can either directly invoke searchRoles(String roleName) or searchRoles(String attributeToSearchRolesBy,String attributeValue,String equalityOrInequalityFlag). The response from the method is a List<AppRoleEntry>. The snippet below shows the code for doing so. Note that although there is another much more flexible method to search across application stripes by using policyStore.getAppRoles(StoreAppRoleSearchQuery obj), It is not implemented for the embedded policy store and throws UnsupportedOperationException.
    /**
     * Given a application stripe name and rolename this can be used to search for a role name
     * This method performs a wildcard search also.
     * @param roleName the rolename to search
     * @param applicationStripeName the application stripe name 
     */
    public List<AppRoleEntry> searchAppRoleInApplicationStripe(String roleName,
                                                               String applicationStripeName) {
        JpsContext ctxt = IdentityStoreConfigurator.jpsCtxt;

        PolicyStore ps = ctxt.getServiceInstance(PolicyStore.class);
        ApplicationPolicy policy;

        try {
            policy = ps.getApplicationPolicy(applicationStripeName);
            return policy.searchAppRoles(ApplicationRoleAttributes.NAME.toString(),roleName,false );
           

        } catch (PolicyStoreException e) {
            throw new RuntimeException(e);
        } 
       
    }

......


    private static final class IdentityStoreConfigurator {
        private static final JpsContext jpsCtxt = initializeFactory();


        private static JpsContext initializeFactory() {
            String methodName =
                Thread.currentThread().getStackTrace()[1].getMethodName();
            JpsContextFactory tempFactory;
            JpsContext jpsContext;
            try {
                tempFactory = JpsContextFactory.getContextFactory();
                jpsContext = tempFactory.getContext();
            } catch (JpsException e) {
                DemoJpsLogger.severe("Exception in " + methodName + " " +
                                     e.getMessage() + " ", e);
                throw new RuntimeException("Exception in " + methodName + " " +
                                           e.getMessage() + " ", e);
            }
            return jpsContext;
        }


    }
....


You can also do various other operations on the policy store and alter the application specific policies, once you have access to those operations. A key thing to note is these operations require specific PolicyStoreAccessPermissions to be granted in the jazn-data.xml. The steps to do so are mentioned below.



  1. Define a resource type of permission class PolicyStoreAccessPermission and the neccessary actions that you want to grant access to (In this example,I am granting access to all operations, signified by *). The snippet is shown below:-



    <resource-type>
      <name>PolicyStorePermission</name>
      <matcher-class>oracle.security.jps.service.policystore.PolicyStoreAccessPermission</matcher-class>
      <actions>*</actions>
    </resource-type>



  2. Next, create resources that are to be granted permissions. In this case, I have created two of them, the first one is the superset that allows access to all the application stripes and the next one grants access to only this applications's stripe. The snippet is shown below:-



    <resources>
     <resource>
            <name>context=APPLICATION, name=*</name>
            <type-name-ref>PolicyStorePermission</type-name-ref>
          </resource>
        
          <resource>
            <name>context=APPLICATION,name=DemoAppSecurity#V2.0</name>
            <type-name-ref>PolicyStorePermission</type-name-ref>
          </resource>
    </resources>



  3. In the last step you have to assign these resources to the application roles or groups.



custom_permission application_stripe_search

The enterprise identity store provider being used here is the embedded weblogic ldap, to run the application properly you will need to configure a password for it in weblogic and set the password in jps-config.xml as shown in the screenshot below.

image_jps_emb_ldap


To run the application, the username/password combination is john/oracle123. To view the search roles screen, either run the SearchRoles.jspx or click on the Search Roles link in the left navigation bar. The link to download the application is mentioned below:-

Download the application.

Posted on Wednesday, August 05, 2015 by Unknown

Jun 11, 2015

You might be aware that there are different content encoding formats for encoding the text. Generally, it is safe to use UTF encoding, but at least you would expect that the websites would specify the encoding format in the response. Alas, you might find certain sites , which just send the content without specifying the content encoding that they are using. So to detect content encoding for such cases, you need a FSM (Finite State Machine). Initially, you just split the input into individual characters and then pass them onto different state machines, each of which uses a different encoding scheme.  For each character that is passed to the state machine, it can either immediately identify a character that is unique to its encoding format, continue, or error out.  At, the end of operation, you would generally expect a specific content encoding format or if insufficient input is available, return the default encoding format.  You can read more about this here Universal Charset Detection. 

Mozilla already has a library for this and there is a java port available for it. Mozilla’s library for this is universalchardet and the java port is juniversalchardet.

It’s really simple to use this as can be seen below:-

public static String detectCharset(InputStream is) throws IOException {

UniversalDetector detector = new UniversalDetector(null);

String encoding=null;

byte[] buf = new byte[1000];
// (2)
int nread;
while ((nread = is.read(buf)) > 0 && !detector.isDone()) {
detector.handleData(buf, 0, nread);
}
// (3)
detector.dataEnd();

// (4)
encoding = detector.getDetectedCharset();
if (encoding != null) {
logger.info("Detected encoding = " + encoding);
} else {
encoding = "ISO-8859-1";
logger.info("No encoding detected.");
}

// (5)
detector.reset();
return encoding;

}


This can be really useful, if you are planning to create a full text parser and the websites don’t return the content encoding that they are using.  I used this while designing the full text parser for my application.



Hope this is useful for others out there.

Posted on Thursday, June 11, 2015 by Unknown

Apr 10, 2015

In this post I will share an example of analyzing memory leak in an Android application. I recently tried to integrate a popular android library ShowcaseView which can be used for first run demos in your android application. The thing is while testing the library, I noticed severe memory leaks. This was occurring because references to ShowcaseView class were being kept. And, as bitmaps are created internally by this class to overlay the content, so each instance was holding a large chunk of memory.

Analyzing the issue 

For heap analysis, I used VisualVM. First, thing you need to realize is that the heap dump taken from android cannot be directly analyzed with VisualVM or other tools such as eclipse MAT. You need to convert it to a format that these tools can analyze. For this you have to run the following command.

hprof-conv Snapshot_2015.04.10_19.37.29.hprof heapissue


Here the first parameter is the android heap dump and the second is the output file.



After loading the file into VisualVM for analysis, I could see the references of ShowCaseView class were being retained as it had GC roots, thus live references, which meant that the instances could not be collected by the JVM Garbage collector.



heap_issue_1



Simple OQL analysis revealed live paths to the instances.



Query:



select  heap.livepaths(u,false) from com.github.amlcurran.showcaseview.ShowcaseView u 




heap_issue_2



As you can see phone decor view was holding references to the view which was added by ShowCaseView.  Thus, the decorview was holding reference to ShowCaseView instances.



Now, Ideally whenever the view has been displayed it should be destroyed, especially,  if it is holding such large chunks of memory.  but, instead due to oversight from the developer instead of removing the view from decorview, he was just setting the views property to View.Gone, which only makes the view invisible and does not remove the view from the ViewGroup. 



The Solution



The problem was then clearly with the fact that the added views should be removed after the ShowcaseView was hidden.  So, I just modified the hide() and hideImmediate() methods to remove the views that were added on top of the decorView.



I have mentioned one of those method below.



@Override
public void hide() {
clearBitmap();
// If the type is set to one-shot, store that it has shot
shotStateStore.storeShot();
mEventListener.onShowcaseViewHide(this);
fadeOutShowcase();
getViewTreeObserver().removeOnPreDrawListener(draw);
if(Build.VERSION.SDK_INT>15){
getViewTreeObserver().removeOnGlobalLayoutListener(globalLayout);
}
else {
getViewTreeObserver().removeGlobalOnLayoutListener(globalLayout);
}
// removeView(ShowcaseView.this);
((ViewGroup)mActivity.getWindow().getDecorView()).removeView(ShowcaseView.this);
}


 



Post, implementing these changes the library works as it is supposed to work, that is just fine. 



heap_issue_fine_1



heap_issue_fine_2



showcaseview



Below, I have mentioned the Gist that contains the entire source code for the modified ShowcaseView. So, happy coding :-)


Posted on Friday, April 10, 2015 by Unknown