Fact/Opinion Classification using the Naive Bayes Classifier and the Iterative Hyperlink-Induced Topic Search Algorithm

In this project, I replicated the key results from the paper, “A Novel Two-stage Framework for Extracting Opinionated Sentences from News Articles” (Pujari, Rajkumar. Desai, Swara. Ganguly, Niloy. Goyal, Pawan).

The project entails using a combination of the Naive Bayes classifier and the Hyperlink-Induced Topic Search (HITS) algorithm to carry out fact/opinion classification of the sentences in a given corpus.

(Project’s Github Link)

Choosing a Classifier from SKLearn

Gaussian Naive Bayes
A simple algorithm based on bayes rule. The “naive” aspect of this classifier is that it assumes independence between every pair of features (hardly ever true in practice). 
For the adjacent formula, y is a given class variable and the x variables represent the features. P(y) is the probability of observing class y in the training set. P(x_vector | y) is the probability of observing the specific x_vector given the class y. Note that the product over all conditionals P(x_i|y) is only possible because of our naive assumption. The final equation indicates how the classifier finds the class to predict.
Advantages-> (1) work pretty well in practice despite naive assumption, (2) can estimate the necessary parameters with relatively small amount of training data, (3) “can be extremely fast compared to more sophisticated methods” (source)
Disadvantages-> (1) more sophisticated models that are better suited for data which are trained well, can outperform NB models, (2) “known to be a bad estimator, so the probability outputs are not taken too seriously” (source) (3) can be particularly ineffective compared to more sophisticated models when there are significant dependent relations between pairs of features.
Example Real World Application->(1) document classification, (2) spam filtering
Continue reading

Using Pandas for Evaluating Classifiers

I’m working on evaluating the performance of a few classifiers on a certain data set using python. Recording the basics of the python code I used here for future reference.

Pandas Data Frames are two dimensional labeled data structures which columns. Its the perfect data structure to represent a data set- with each column representing a different feature.

Continue reading

Dealing with Despair: Existentialism and Education

For a greater part of my later years in high school and my initial years of college I suffered from a bout of existential despair. There were protracted periods of time when questioning my world view and my way of life resulted in an overwhelming sense of futility and insignificance. I was so overwhelmed as an angst ridden teen that I made desperate attempts at suspending introspection. Addictive gaming. Binge watching shows and movies. A couple of months of aggressive smoking. A couple of months of aggressive drinking. Obsessing over a girl. The usual suspects of teenage waywardness.

YouthKiAwaaz Link: If You’re Dealing With An Existential Crisis This Post is Definitely For You

Continue reading

Income Inequality in the United States

ECON PROJET IMAGE 2Income inequality in the United States is at historic heights (Pew Research, 2013). Unless the underlying structural drivers of income inequality are addressed, an increasingly restive populace could fundamentally undermine the efficacy of the institutions that promote peace, prosperity and progress in American society. For instance, (1) increasing disillusionment with the political establishment could compromise the delicate balance of power between despotism and anarchy through the rise of authoritarian or populist forces (Hamilton, 1781), (2) associating the pursuit of knowledge as an exploitative attribute of the economic elite could result in a skepticism for education that stymies the nation’s economic potential (Lyngar, 2015), (3) directing the frustration compounded by inequality towards hard-working and productive immigrants could result in the compromise of humanitarian values and the nation’s human capital (Dustmann, 2004).  In short, the growing resentment with the economic and political elite as a consequence of increasing income inequality could result in the adoption of measures that exacerbate various negative social and economic trends. There is thereby a pressing need to understand and address the structural drivers of income inequality. Unfortunately, rather than proactively address the key structural drivers of income inequality, policy makers tend to reactively appease the public subject to the negative effects of income inequality. This paper specifically analyzes one such significant case of appeasement in the recent past where policy makers increased access to housing as a response to voters on the wrong side of the rising income inequality trend (Rajan, 2010).

Continue reading

A Brief Note on John Adams (The HBO TV Miniseries)

The HBO miniseries John Adams (2008) was one of the two TV shows that I binge-watched in the Fall 2014 semester. The show’s evident unadulterated creative process, engaging story telling and commitment to historical accuracy kept me hooked to the computer screen right through it’s seven episodes.

The series, which was directed by Tom Hooper (who went on to direct The King’s Speech and Les Miserables), rightfully received critical acclaim (winning four golden globes and thirteen Emmy awards). I have been trying to understand the events and ideas of the American Revolution for sometime now, and through this show, I was able to develop a basic understanding for the life of one of the centrally important founding fathers of the United States. There were many interesting ideas and events that I made a note to follow-up on as I watched the show (the Boston Massacre, John and Abigail Adams’ amazing relationship, Thomas Jefferson’s emphasis on small Government, Alexander Hamilton setting up the First Bank of the United States etc etc). I have included a small fraction of those notes in this post.

Continue reading

Addressing India’s Immense Healthcare Challenges


An article I recently read in the Wall Street Journal titled, “The Ailing Health of a Growing Nation” describes how the public healthcare system in India is in an abysmal state. According to the article, a senior official in the Indian health ministry said that the country has too many competing social priorities (like education and infrastructure) which cause an extremely inadequate investment in public healthcare by the Indian Government (1 – 1.4 % of GDP). Upon reading the article and conducting some cursory research about the state of public healthcare in India, it was evident that the egregious healthcare problems/crises that India faces will simply not be tackled through Government programs and healthcare alone.

Continue reading