Alex Rivera | Logout

How do I tell what language is a plain-text file written in?

Asked 2010-02-24T12:51:43.103
27

Suppose we have a text file with the content: "Je suis un beau homme ..."

another with: "I am a brave man"

the third with a text in German: "Guten morgen. Wie geht's ?"

How do we write a function that would tell us: with such a probability the text in the first file is in English, in the second we have French etc?

Links to books / out-of-the-box solutions are welcome. I write in Java, but I can learn Python if needed.

My comments

  1. There's one small comment I need to add. The text may contain phrases in different languages, as part of whole or as a result of a mistake. In classic litterature we have a lot of examples, because the aristocracy members were multilingual. So the probability better describes the situation, as most parts of the text are in one language, while others may be written in another.
  2. Google API - Internet Connection. I would prefer not to use remote functions/services, as I need to do it myself or use a downloadable library. I'd like to make a research on that topic.
Edit
Report

1 Answer

2

Although a more complicated solution than you are looking for, you could use Vowpal Wabbit and train it with sentences from different languages.

In theory you could get back a language for every sentence in your documents.

http://hunch.net/~vw/

(Don't be fooled by the "online" in the project's subtitle - that's just mathspeak for learns without having to have whole learning material in memory)

answered 2010-02-24T13:23:57.937

Your Answer