Mar 11, 2016

Well blogger does not have support for latex and the windows live writer is being redeveloped. So, in the meanwhile, I have written a few posts on the pelican blog and thought, I might as well link to them here.

Why you should prefer to use the square root of Gini Index: This post examines the advantages of using the Gini Index as the criteria for building decision trees.
http://orastack.com/why-you-should-use-square-root-of-gini-index.html

Do tweets have predictive power. This post examines whether tweets have an effect on opening weekend revenue of box office movies.
http://orastack.com/scikit-learn_tweet_classifier.html

Posted on Friday, March 11, 2016 by Ramandeep Singh Nanda

Aug 5, 2015

Earlier, I had covered an example here, which showed how to dynamically create users and map the application roles to enterprise groups. In this post, the sample application is extended to show how you can query the application roles from the application stripe(application specific policies).
To query the application specific roles, you need to access the application's policy from policystore and then you can either directly invoke searchRoles(String roleName) or searchRoles(String attributeToSearchRolesBy,String attributeValue,String equalityOrInequalityFlag). The response from the method is a List<AppRoleEntry>. The snippet below shows the code for doing so. Note that although there is another much more flexible method to search across application stripes by using policyStore.getAppRoles(StoreAppRoleSearchQuery obj), It is not implemented for the embedded policy store and throws UnsupportedOperationException.
    /**
     * Given a application stripe name and rolename this can be used to search for a role name
     * This method performs a wildcard search also.
     * @param roleName the rolename to search
     * @param applicationStripeName the application stripe name 
     */
    public List<AppRoleEntry> searchAppRoleInApplicationStripe(String roleName,
                                                               String applicationStripeName) {
        JpsContext ctxt = IdentityStoreConfigurator.jpsCtxt;

        PolicyStore ps = ctxt.getServiceInstance(PolicyStore.class);
        ApplicationPolicy policy;

        try {
            policy = ps.getApplicationPolicy(applicationStripeName);
            return policy.searchAppRoles(ApplicationRoleAttributes.NAME.toString(),roleName,false );
           

        } catch (PolicyStoreException e) {
            throw new RuntimeException(e);
        } 
       
    }

......


    private static final class IdentityStoreConfigurator {
        private static final JpsContext jpsCtxt = initializeFactory();


        private static JpsContext initializeFactory() {
            String methodName =
                Thread.currentThread().getStackTrace()[1].getMethodName();
            JpsContextFactory tempFactory;
            JpsContext jpsContext;
            try {
                tempFactory = JpsContextFactory.getContextFactory();
                jpsContext = tempFactory.getContext();
            } catch (JpsException e) {
                DemoJpsLogger.severe("Exception in " + methodName + " " +
                                     e.getMessage() + " ", e);
                throw new RuntimeException("Exception in " + methodName + " " +
                                           e.getMessage() + " ", e);
            }
            return jpsContext;
        }


    }
....


You can also do various other operations on the policy store and alter the application specific policies, once you have access to those operations. A key thing to note is these operations require specific PolicyStoreAccessPermissions to be granted in the jazn-data.xml. The steps to do so are mentioned below.



  1. Define a resource type of permission class PolicyStoreAccessPermission and the neccessary actions that you want to grant access to (In this example,I am granting access to all operations, signified by *). The snippet is shown below:-



    <resource-type>
      <name>PolicyStorePermission</name>
      <matcher-class>oracle.security.jps.service.policystore.PolicyStoreAccessPermission</matcher-class>
      <actions>*</actions>
    </resource-type>



  2. Next, create resources that are to be granted permissions. In this case, I have created two of them, the first one is the superset that allows access to all the application stripes and the next one grants access to only this applications's stripe. The snippet is shown below:-



    <resources>
     <resource>
            <name>context=APPLICATION, name=*</name>
            <type-name-ref>PolicyStorePermission</type-name-ref>
          </resource>
        
          <resource>
            <name>context=APPLICATION,name=DemoAppSecurity#V2.0</name>
            <type-name-ref>PolicyStorePermission</type-name-ref>
          </resource>
    </resources>



  3. In the last step you have to assign these resources to the application roles or groups.



custom_permission application_stripe_search

The enterprise identity store provider being used here is the embedded weblogic ldap, to run the application properly you will need to configure a password for it in weblogic and set the password in jps-config.xml as shown in the screenshot below.

image_jps_emb_ldap


To run the application, the username/password combination is john/oracle123. To view the search roles screen, either run the SearchRoles.jspx or click on the Search Roles link in the left navigation bar. The link to download the application is mentioned below:-

Download the application.

Posted on Wednesday, August 05, 2015 by Ramandeep Singh Nanda

Jun 11, 2015

You might be aware that there are different content encoding formats for encoding the text. Generally, it is safe to use UTF encoding, but at least you would expect that the websites would specify the encoding format in the response. Alas, you might find certain sites , which just send the content without specifying the content encoding that they are using. So to detect content encoding for such cases, you need a FSM (Finite State Machine). Initially, you just split the input into individual characters and then pass them onto different state machines, each of which uses a different encoding scheme.  For each character that is passed to the state machine, it can either immediately identify a character that is unique to its encoding format, continue, or error out.  At, the end of operation, you would generally expect a specific content encoding format or if insufficient input is available, return the default encoding format.  You can read more about this here Universal Charset Detection. 

Mozilla already has a library for this and there is a java port available for it. Mozilla’s library for this is universalchardet and the java port is juniversalchardet.

It’s really simple to use this as can be seen below:-

public static String detectCharset(InputStream is) throws IOException {

UniversalDetector detector = new UniversalDetector(null);

String encoding=null;

byte[] buf = new byte[1000];
// (2)
int nread;
while ((nread = is.read(buf)) > 0 && !detector.isDone()) {
detector.handleData(buf, 0, nread);
}
// (3)
detector.dataEnd();

// (4)
encoding = detector.getDetectedCharset();
if (encoding != null) {
logger.info("Detected encoding = " + encoding);
} else {
encoding = "ISO-8859-1";
logger.info("No encoding detected.");
}

// (5)
detector.reset();
return encoding;

}


This can be really useful, if you are planning to create a full text parser and the websites don’t return the content encoding that they are using.  I used this while designing the full text parser for my application.



Hope this is useful for others out there.

Posted on Thursday, June 11, 2015 by Ramandeep Singh Nanda

Apr 10, 2015

In this post I will share an example of analyzing memory leak in an Android application. I recently tried to integrate a popular android library ShowcaseView which can be used for first run demos in your android application. The thing is while testing the library, I noticed severe memory leaks. This was occurring because references to ShowcaseView class were being kept. And, as bitmaps are created internally by this class to overlay the content, so each instance was holding a large chunk of memory.

Analyzing the issue 

For heap analysis, I used VisualVM. First, thing you need to realize is that the heap dump taken from android cannot be directly analyzed with VisualVM or other tools such as eclipse MAT. You need to convert it to a format that these tools can analyze. For this you have to run the following command.

hprof-conv Snapshot_2015.04.10_19.37.29.hprof heapissue


Here the first parameter is the android heap dump and the second is the output file.



After loading the file into VisualVM for analysis, I could see the references of ShowCaseView class were being retained as it had GC roots, thus live references, which meant that the instances could not be collected by the JVM Garbage collector.



heap_issue_1



Simple OQL analysis revealed live paths to the instances.



Query:



select  heap.livepaths(u,false) from com.github.amlcurran.showcaseview.ShowcaseView u 




heap_issue_2



As you can see phone decor view was holding references to the view which was added by ShowCaseView.  Thus, the decorview was holding reference to ShowCaseView instances.



Now, Ideally whenever the view has been displayed it should be destroyed, especially,  if it is holding such large chunks of memory.  but, instead due to oversight from the developer instead of removing the view from decorview, he was just setting the views property to View.Gone, which only makes the view invisible and does not remove the view from the ViewGroup. 



The Solution



The problem was then clearly with the fact that the added views should be removed after the ShowcaseView was hidden.  So, I just modified the hide() and hideImmediate() methods to remove the views that were added on top of the decorView.



I have mentioned one of those method below.



@Override
public void hide() {
clearBitmap();
// If the type is set to one-shot, store that it has shot
shotStateStore.storeShot();
mEventListener.onShowcaseViewHide(this);
fadeOutShowcase();
getViewTreeObserver().removeOnPreDrawListener(draw);
if(Build.VERSION.SDK_INT>15){
getViewTreeObserver().removeOnGlobalLayoutListener(globalLayout);
}
else {
getViewTreeObserver().removeGlobalOnLayoutListener(globalLayout);
}
// removeView(ShowcaseView.this);
((ViewGroup)mActivity.getWindow().getDecorView()).removeView(ShowcaseView.this);
}


 



Post, implementing these changes the library works as it is supposed to work, that is just fine. 



heap_issue_fine_1



heap_issue_fine_2



showcaseview



Below, I have mentioned the Gist that contains the entire source code for the modified ShowcaseView. So, happy coding :-)


Posted on Friday, April 10, 2015 by Ramandeep Singh Nanda

Mar 21, 2015

SSL handshake errors can occur due to various reasons such as Self Signed certificate, unavailability of protocol or cipher suite requested by client or server, etc.  Recently I faced this issue where I was connecting to third party server using HttpClient library.  Here’s what I did to identify the cause:-

Firstly, I enabled the debug flag for SSL,handshake and failure on  javax.net packages.

-Djavax.net.debug=ssl,handshake,failure


On examining the logs, I could see that the third party site was expecting a cipher key of 256 bits and the only supported keys in my glassfish server were of 128 bits length.  As it happens,  this occurs because OOTB java 6, 7 or 8 support only 128 bit encryption keys. To enable 256 or higher bit key length , you need to download the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files which essentially contains two jars i.e US_export_policy.jar and local_policy.jar and place them in <JRE_HOME>/lib/security/ directory and restart the server to enable higher bit encryption keys.



The above step will enable  256 bit or higher bit encryption keys and will ensure that you do not face SSL Handshake errors due to key strength.



You can download the Policy files from the following links.



JCE Unlimited for java 6



JCE Unlimited for java 7



JCE Unlimited for java 8

Posted on Saturday, March 21, 2015 by Ramandeep Singh Nanda

If you use ProGuard for obfuscating your code and happen to use Retrofit in your application, you will need to configure ProGuard to exclude certain Retrofit files from being obfuscated. Also you must note that if you are using GSON for conversion from JSON to POJO representation, you must ignore those POJO classes from being obfuscated, this is required as if those POJO class  field names are obfuscated, conversion to POJO’s from JSON would fail because POJO  field names are inferred from JSON response.   So to keep it brief you should use the following configuration.

-keep class com.squareup.** { *; }
-keep interface com.squareup.** { *; }
-dontwarn com.squareup.okhttp.**
-keep class retrofit.** { *; }

-keepclasseswithmembers class * {
@retrofit.http.* <methods>;
}

-keep interface retrofit.** { *;}
-keep interface com.squareup.** { *; }
-dontwarn rx.**
-dontwarn retrofit.**


#Here include the POJO's that have you have created for mapping JSON response to POJO for example
com.blogspot.ramannanda.apps.xyz.FeedlyResponse {*;}


Here FeedlyResponse is just a POJO class that maps to JSON fields returned by Feedly feed search API.

Posted on Saturday, March 21, 2015 by Ramandeep Singh Nanda

Mar 4, 2015

I recently reviewed this title and found it short on a few important consideration such as cross client authorization. This is definitely a book for developers who are beginning android application development, but isn’t comprehensive.

I discuss about what each chapter covers and then offer suggestions later on how this book can be improved further.

Chapter 1: Android Security Issues

  1. Talks about the different security compliance standards 
  2. What are the common problems in android applications
  3. How one can easily re-engineer your applications code.

Chapter 2: Protecting your code
Here the author talks about why you should obfuscate your code. It starts by explaining how easy it is to re-engineer the code, if the code is not obfuscated. Obfuscation tools are then covered to show how to obfuscate your applications code. The author then talks about disassemblers to show that even though obfuscation might deter someone from looking at your code, It might not truly prevent someone from hacking your application code.

Chapter 3: Authentication
Here the author talks about different authentication schemes username/password, facebook login etc.

Chapter 4: Network communication
Talks about asymmetric public key encryption, Why you should use SSL security and demonstrates the Man in the middle attack. It also explains why your application should validate ssl certificates.

Chapter 5: Databases
Talks about general database best practices such as encryption and preventing SQL injection.

Chapter 6: Web Server Attacks
Talks about securing web services, XSS attack etc. Here, I feel the author should have covered authentication and authorization challenges that one usually faces with android applications, as one generally needs to implement validations of requests from mobile devices. For example, A user can easily know your service endpoint as the code is deployed on the client side and send a request to that URL from their application as well, So you need to differentiate between the request from your application and other applications.  (I personally use Google plus sign in API's, along with server side token validation to ensure that any back-end requests are originating from within my application and are from the correct individual)

Chapter 7: Third party library integration
Mentions that you should be aware of the permissions that you are granting to the third party libraries.

Chapter 8:Device Security
Talks about device security issues and why you should enable encryption. It then talks about how device security is enforced on Kitkat. The author then discusses some android version specific exploits and offers certain solutions.

Chapter 9: The Future
This chapter covers Intent hijacking and how to deal with it in your android application. The chapter then covers devices such as android wear and the extended ecosystem of android devices and its impact on security considerations. Furthermore, the chapter covers tools which expose security vulnerability in your application.

Conclusions:
The book covers a lot of common security vulnerabilities that developers expose while writing the android applications and has a lucid prose and demonstrates these vulnerabilities practically by showing examples. It also offers solutions to those problems. For a developer who is beginning application development with android, having knowledge about these issues is important.  However, most of these issues would be known to experienced developers. I feel the detailed coverage of topics such as securing back-end services unobtrusively, OAuth, OWSM,  etc could have added value to the book. Maybe its just me but I expect these topics to be covered in detail, as most of the android applications would be using some form of back-end service to offload heavy processing. I rate it 3.5 for the content it has covered. 

Posted on Wednesday, March 04, 2015 by Ramandeep Singh Nanda

Feb 26, 2015

In this post, I will explain how you can use apache chemistry API’s to query the enterprise content management systems. Apache Chemistry project provides client libraries for you to easily implement integration with any of the content management products that support or implement CMIS standard. As you might be aware that there are multiple standards such as JCR and CMIS  for interacting with the content repositories. Although with JCR 2 support for SQL-92 is now available,  I really prefer the conciseness and wider adoption of the  CMIS standard, and the fact that Apache chemistry API’s really make it easy to interact with the content repository. 

I am sharing an example that you can readily test without any special environment setup.  Alfresco, is one of the vendors that provides a ECM product, which supports the CMIS standard. Alfresco, offers a public repository for you to play with.  I will cover this example on a piecemeal basis.

  1. Connecting to the repository
    SessionFactory sessionFactory = SessionFactoryImpl.newInstance();
    Map<String,String> parameter = new HashMap<String,String>();
    parameter.put(SessionParameter.USER, "admin");
    //the binding type to use
    parameter.put(SessionParameter.BINDING_TYPE, BindingType.ATOMPUB.value());
    parameter.put(SessionParameter.PASSWORD, "admin");
    //the endpoint
    parameter.put(SessionParameter.ATOMPUB_URL, "http://cmis.alfresco.com/s/cmis");
    parameter.put(SessionParameter.BINDING_TYPE, BindingType.ATOMPUB.value());
    //fetch the list of repositories
    List<Repository> repositories = sessionFactory.getRepositories(parameter);
    //establish a session with the first ?
    Session session = repositories.get(0).createSession();


    To connect to the repository, you require a few basic parameters such as the username, password, endpoint url (In this case the REST AtomPub service) and a binding type to specify which type of endpoint is it (WEBSERVICES, ATOMPUB, BROWSER, LOCAL, CUSTOM ) are the valid binding types. After we have this information , we still need a repository id to connect to. In this case, I am using the first repository from a list of repositories to establish the session. Now, let’s create a query statement to search for the documents.






  2. Querying the repository: 
    //query builder for convenience
    QueryStatement qs=session.createQueryStatement("SELECT D.*, O.* FROM cmis:document AS D JOIN cm:ownable AS O ON D.cmis:objectId = O.cmis:objectId " +
    " where " +
    " D.cmis:name in (?)" +
    " and " +
    " D.cmis:creationDate > TIMESTAMP ? " +
    " order by cmis:creationDate desc");
    //array for the in argument
    String documentNames[]= new String[]{"Project Objectives.ppt","Project Overview.ppt"};
    qs.setString(1, documentNames);
    Calendar now = Calendar.getInstance();
    //subtract 5 year for viewing documents for last 5 year
    now.add(Calendar.YEAR, -5);
    qs.setDateTime(2, now);
    //get the first 50 records only.
    ItemIterable<QueryResult> results = session.query(qs.toQueryString(), false).getPage(50);


    Here I have used createQueryStatement  method to build a query just for convenience, you could also directly specify a query string(not recommended). The query is essentially a join between objects. This sample code shows, how to specify the date (Line 14) and an array (Line 10) for the in clause as parameters.  Line 16 assigns the searched values to an Iterable interface, where each QueryResult is a record containing the selected columns.




  3. Iterating the results:

    for(QueryResult record: results) {
    Object documentName=record.getPropertyByQueryName("D.cmis:name").getFirstValue();
    logger.info("D.cmis:name " + ": " + documentName);

    Object documentReference=record.getPropertyByQueryName("D.cmis:objectId").getFirstValue();
    logger.info("--------------------------------------");
    logger.info("Content URL: http://cmis.alfresco.com/service/cmis/content?conn=default&id="+documentReference);
    }

    As explained above, we get a Iterable result-set to iterate over the individual records. To fetch the first value from the record (as there might be multiple valued attributes), I am using the getFirstValue method of the PropertyData interface.  Note Line 7 as it contains the actual URL of the resource, which is just a base URL to which the object id of the matched document is appended.


  4. Closing the connection ? As per the chemistry javadoc, there is no need to close a session, as it is purely a client side concept, which makes sense as we are not holding a connection here.



Viewing the results: To view the actual documents just use the URL’s generated by the log statement in the browser.



Building the code: Add the following dependency to maven for building the sample.



    <dependency>
<groupId>org.apache.chemistry.opencmis</groupId>
<artifactId>chemistry-opencmis-client-impl</artifactId>
<version>0.12.0</version>
</dependency>


Wrapping up: I have just covered one example of the CMIS Query API  and Apache chemistry to query for the documents. Kindly refer to the documentation links provided in reference section for other usages.  Below, is the gist that contains the entire sample code.






References:


CMIS_Query_Language Java Examples for Apache Chemistry

Posted on Thursday, February 26, 2015 by Ramandeep Singh Nanda