From Wakapon
Jump to: navigation, search
Line 67: Line 67:
 
* "ate", a verb at the past tense
 
* "ate", a verb at the past tense
 
* "the mouse", a definite object or target
 
* "the mouse", a definite object or target
 +
 +
  
 
Let's put '''A''' is the symbol for "cat", '''B''' is the symbol for "eat" and '''C''' is the symbol for "mouse".
 
Let's put '''A''' is the symbol for "cat", '''B''' is the symbol for "eat" and '''C''' is the symbol for "mouse".
Line 72: Line 74:
 
Finally, let's define the "p" subscript for "past" or "preterit".
 
Finally, let's define the "p" subscript for "past" or "preterit".
  
We could write the sentence as :
+
We could then write the sentence as :
  
 
<math>\mathbf{A_d} \to \mathbf{B_p} \to \mathbf{C_d}</math>
 
<math>\mathbf{A_d} \to \mathbf{B_p} \to \mathbf{C_d}</math>
 +
 +
 +
That kind of symbolic representation of sentences is valid in any language, all you need is a symbol database for nouns, verbs, adjectives, adverbs and all these stuff. Automatic contextual translation of texts has made incredible leaps forward these last few years, as shown by Google in their excellent presentation video of Wave where their [http://andrewhitchcock.org/?post=322 automatic translation tool] performed quite brightly.

Revision as of 04:16, 12 February 2010

Okay, let me try and explain what this great idea is about. S13.gif

First, let's see some basic politics as an introduction to what I will expose later.

Politics

First, you should know I'm a fucking social anarchist S2.gif. And as such, I don't like the inherent hierarchy of the powers in place and their ugly scheming to get to the top. I just can't stand politics and corruption. I just can't grasp the concept of lust for power and money. And I can't even begin to understand why someone who has enough money to buy a small country just needs even more.

Let me be clear on these thoughts : I don't want to blow everything up, shoot everyone and make a revolution. Our capitalist system is obviously far from perfect but I believe it can be "mended" in many ways so we achieve more equality in revenues and so a huge part of the world isn't left aside like junk. I would have liked not to quote the obvious here, like the richest 2% owning half the wealth of the planet, or that the cost of the war in Iraq itself would have permitted to buy all the weapons we're so afraid of, or even that it would cost 40 billion $ annually to feed the hungry (the budget of the G8 summit where "important" people discuss of this matter costing $600 million on its own) but I write these small facts here as a memento for some other time.


Keeping that in mind, after spending many years being angry at everything, we need to focus on finding ways to change things and make the system more equal.


Politicians, whether they are left-wing liberal democrats or right-wing conservative republicans, all want the same thing : power. They usually have a short-term vision of things essentially because of their equally short-term mandates. Often laws and amendments get voted to be overruled 1 or 2 presidential mandates later, yielding a brownian-motion-like status quo. Also, these politicians are almost always issued from bourgeoisie and aristocracy, they are formed and taught in high-standard schools whose diplomas always guarantee a successful career. These people are NOT your friends, they are not of your class and don't know the cost or the difficulties and precarity of life, yet here they are trying to solve your problems they don't have a clue about. They only are theoreticians of life.

Also, it is my intimate conviction that politicians have no real power anymore and don't rule their country as they used to : multi-national corporations do through lobbying and economic pressure. Politicians can only limit the damages caused to their countries by these corporations (when they are willing to do so) by applying mere patches and solving neighborhood-range crises, when they are not altogether at the mercy of such corporations through either economic blackmail or mere corruption.

Politicians have become CEOs of their countries they now run like mere corporations. We're the employees. Revolving doors between government positions and private sector companies work 24-7. Conflicts of interest are now showing blatantly in the open and are part of the system.

Another one of my convictions is that the economic system in place is anti-human in all its forms. I mean it's not in the interest of the market, ever, that people are happy and in good health ! If people were all well fed, all had shelter, all were in good health and were all happy with simple facts of life instead of pursuing "happiness" through consumption, then the market would collapse.


What we need is a way to make the people in place take their job seriously. We need a way to monitor what they are doing, to understand what their agendas really are and what possible conflicts of interest they are in : we need to find a way to make them do the work they were elected for. The "affairs" newspapers sometimes leak are mere accidents, I'm sure there are hundreds of these affairs we never hear of and that's a shame. If we ever had a way to somewhat automatically find relations between people, trace their life and monitor their quotes and achievements then we would have a tool to actually "measure" the honesty and value of these people.

They are, after all, public persons elected by the public. It's only fair to assume they should be accountable to the public !


What's the Relation with Semantic Analysis ?

What I'm proposing here is a tool to help people monitor public persons.

The idea is quite simple really : we need to create a program that automatically analyses all possible documents (newspapers mainly, proceedings, reports, bulletins) that pertain to the public life of public persons and build a huge facts database or FDB. This is not spying on these people but merely collecting data on them through quotations of existing documents.

In the end, the FBD should contain a pretty amazing summary of the career of public people. Also, it should contain very useful information on the relationships and collaboration between people. And when I say people, I also mean corporation CEOs and their companies as well (which are now accountable as moral persons according to the law).

Using a simple system of scoring for relationships and public affairs, it should be fairly easy to give "grades" to the public persons, companies or to the facts themselves ranging from "truthful" to "very doubtful". As an example, if we somehow managed to find a connection between a scientific report about the utility of OGMs written by someone who used to work for a company that was at some point commissioned by Monsanto for a project, it would be quite difficult to give a "truthful" grade to that report. There would be a clear conflict of interest here, but it would only be made clear by the program really, an investigation journalist could do that too but that would be a lot of work and journalists are not always free of interest either.

Now you're starting to understand where I'm going.


Program Description

To achieve this, we need to separate the program in several stages :

  1. Bots that will be used to collect and update data from known sources, mainly online newspaper archives and "trusted" sources
  2. A Lexical Analyser that will be used to verify the lexical validity of a text prior feeding it for translation and semantic analysis
  3. A Translator Module that will be used to format the text in a universally readable format so the semantic analyser can be independent of the source language
  4. The Semantic Analyser that will perform the semantic analysis of the language-independent text and that will basically tie facts to names
  5. The Facts Database that will store facts and their relation to people, brands or companies
  6. The Query Engine that will be able to answer user queries and display usable information


Bots

The bots will need to be written specifically for the target site to harvest the data as they are provided by the target site. The main code that grabs the text will be the same for all sites but the part that posts requests to the site will have to be specific to the site itself.

Also, if the site changes presentation or access permissions, the bot should handle failures elegantly and warn us that the code needs to be changed to fit the new site requirements.

The bots should also be able to determine if the text is part of a group of texts as newspapers often choose to write several articles on the same topic and these texts should then be marked as treating of the same subject.

Lexical Analyser

This part is language-specific and should be used to verify the validity of the text. It should do some basic checks like syntax and spelling, punctuation and pre-formatting so the text is ready for translation.

Translator Module

This module is language-specific. It's one of the most important parts of the program as it's responsible for translating any sentence of any language into a language-independent symbolic language form.

For example, the sentence "The cat ate the mouse" contains :

  • "the cat", a definite subject
  • "ate", a verb at the past tense
  • "the mouse", a definite object or target


Let's put A is the symbol for "cat", B is the symbol for "eat" and C is the symbol for "mouse". Let's also define the "d" subscript for "definite" ("the", as opposed to "a" or "some"). Finally, let's define the "p" subscript for "past" or "preterit".

We could then write the sentence as :

<math>\mathbf{A_d} \to \mathbf{B_p} \to \mathbf{C_d}</math>


That kind of symbolic representation of sentences is valid in any language, all you need is a symbol database for nouns, verbs, adjectives, adverbs and all these stuff. Automatic contextual translation of texts has made incredible leaps forward these last few years, as shown by Google in their excellent presentation video of Wave where their automatic translation tool performed quite brightly.