Natural language processing (NLP)

learn more… | top users | synonyms (1)

2
votes
1answer
18 views

How is wordnet curated?

I use Princeton's WordNet in NLP applications. I always read that it is "human-curated." But how is it curated? Who decides that words belong in a synset? How does the process work? Wikipedia does not ...
2
votes
3answers
134 views

Does programming language detection need more input than natural language detection?

I wonder which one of the two needs a larger input to achieve a decent accuracy: programming language detection or natural language detection? More details: Definition of Language detection: ...
2
votes
0answers
48 views

What are recent, high-quality surveys on NLP topics?

In particular POS tagging, dependency and constituent parsing. This is not really my field of study but I would really like to be able to make informed claims on what precisions current top systems ...
1
vote
2answers
142 views

What are the definitions of syntax and semantics?

For a formal language $L \subseteq \Sigma^*$ over an alphabet $\Sigma$. From https://proofwiki.org/wiki/Definition:Syntax The syntax of a formal language is its structure, and is specified by a ...
2
votes
1answer
42 views

Generative Machine Learning algorithms on tree structure

I'm looking into PCFG sentence grammar dependency structure parsing using StanfordNLP PCFG parser. It generates tree structures represented as a string like this: ...
1
vote
0answers
46 views

Two classes of documents. Find weighted relations between them

I have an NLP problem and a potential solution, but I’m a bit green here, so I’m looking for some validation or alternative suggestions. Background I have two types of documents: one is a set of ...
1
vote
0answers
45 views

Translation of natural language to logic [closed]

Given a statement in natural language, what can be said about how many possible translations there are in first-order-logic? What happen if we take a more complex logic like second-order logic and ...
3
votes
2answers
58 views

How do learn the most important nodes in a tree?

I have a list of 20000 words and how often they appeared in a set of 500 newspaper articles. I am trying to build a stemmer which chops off suffuxes from each words, so ...
3
votes
0answers
38 views

What are some of the methods that NLP practitioners use to automatically learn linguistic features from text? [closed]

I am learning about NLP, with an eye to starting some practical NLP projects. I see that many of the algorithms for relation extraction and named entity recognition require you to identify linguistic ...
2
votes
0answers
17 views

What are ways that you might analyze relations extracted from text [closed]

In the coursera NLP course, Dan Jurofsky says you might want to extract relations from text to serve as input to other algorithms. What are examples of ways that researchers have used or might use ...
3
votes
0answers
34 views

Document clustering for summarization

I am curious as to what steps one would reasonably need to take to perform an extraction-based text summarizer. I've taken a look at some papers I've found on Google such as this one, which explains ...
0
votes
0answers
22 views

Is there a paper/implementation/tool on automatically finding/suggesting similar questions for a given question?

I would like to know whether there is a method to find similar questions for a given question, just like this stackexchange.com does. Is there any paper or tools? I tried keywords such as ...
0
votes
0answers
53 views

NLP: in practice, what should I choose as Start and End symbol?

I'm writing a small program that implements a trigram HMM model and apply it by using Viterbi Algorithm. It's very standardized. However, I have never written anything like this before, and I get ...
1
vote
0answers
22 views

In NLP, tokens not seen in training sample, but you know or don't know what they are

In NLP, do you distinguish tokens that you don't observe in a training sample and still expect that they may occur in a test sample, between those you know what they are, and those you don't what ...
2
votes
4answers
181 views

What aspects of linguistics are necessary or good for natural language processing? [closed]

What aspects of linguistics are necessary or good to know for natural language processing? What references do you recommend for studying those aspects? Thanks!
2
votes
1answer
57 views

Solving the part-of-speech tagging problem with HMM

There is a famous part-of-speech tagging problem in Natural Language Processing. The popular solution is to use Hidden Markov Models. So that, given the sentence $x_1 \dots x_n$ we want to find the ...
1
vote
1answer
26 views

Trying to tag parts of a word and keep track of any changes that happen to those parts

I'm a researcher working with a language that has gone through phonological changes through time. I would like to tag parts of a word (i.e. prefix, stem, suffix) and then apply those phonological ...
2
votes
1answer
69 views

Using interval graphs to find authorship disputes

The first chapter of the book "Graphs and their uses" by Oystein Ore says that interval graphs can be used to resolve authorship disputes, but I couldn't find any details. How does this work? What ...
2
votes
0answers
351 views

Next-Word Prediction, Language Models, N-grams

I was looking into how a next-word prediction engine like swift key or XT9 can be implemented. Here's what I did. I read about n-grams here - en.wikipedia.org/wiki/N-gram and ...
1
vote
0answers
270 views

Next Word Prediction using n-gram & Tries

I am studying the following paper for understanding next-word prediction using n-gram & trie: - http://nlp.cs.berkeley.edu/pubs/Pauls-Klein_2011_LM_paper.pdf Before this, I did some brief study on ...
3
votes
1answer
191 views

Determining interests from Twitter text using Latent Dirichlet Allocation

I would like to determine interests or hobbies that people have given their Twitter timeline data. Their timeline is their historical collection of tweets. The result I am trying to achieve is to ...
2
votes
1answer
100 views

Automatic speech correction

When we are trying to learn a new language without any help from native speakers, the first thing we need is something/someone to correct our grammatical mistakes/improper word selection. So is there ...
2
votes
2answers
78 views

Hidden Markov Model in Tagging Problem

I try to understand the details regarding using Hidden Markov Model in Tagging Problem. The best concise description that I found is the Course notes by Michal Collins. The goal is to find a ...
2
votes
1answer
128 views

Estimate entropy, based upon observed frequency counts

Suppose I have $n$ independent observations $x_1,\dots,x_n$ from some unknown distribution over a known alphabet $\Sigma$, and I want to estimate the entropy of the distribution. I can count the ...
0
votes
1answer
90 views

short text categorization with spelling correction

I am building a classifier for short texts in a chat system. My features are words and pairs of words. Naturally, the sentences contain spelling mistakes. If a particular wrong spelling of a certain ...
0
votes
0answers
63 views

Advantages of knowing foreign languages for natural language processing

I wonder about cases in which knowing several languages can lead a researcher to interesting results in natural language processing. For example, knowledge of foreign languages can without doubt ...
2
votes
0answers
66 views

The grammar of the GeoQuery language

GeoQuery is a dataset used for benchmarking semantic parsers. It contains 880 queries about USA geography. The queries are in Prolog format, for example: ...
3
votes
1answer
66 views

Evaluation metric for an ordering algorithm

There is a sentence with N words. The words are randomly shuffled. I have a heuristic algorithm that tries to restore the original order. I want to evaluate my algorithm on a dataset of several ...
0
votes
2answers
2k views

How is Natural language processing related to Artificial Intelligence [closed]

What is the role of AI in Natural language processing ? For example Siri. Is Ai used to make siri get better at understanding the users voice ?
20
votes
9answers
2k views

Are programming languages becoming more like natural languages?

Can we study programming languages in the context of linguistics? Do programming languages evolve naturally in similar ways to natural languages? Although full rationality, and mathematical ...
1
vote
0answers
56 views

Help understanding an audio processing algorithm

I'm reading “Speech segmentation without speech recognition” by Dong Wang, Lie Lu and Hong-Jiang Zhang. The algorithm I'm looking at is a V/C/P (Vowel/Consonant/Pause) classification algorithm on a ...
2
votes
1answer
71 views

Analyzing rules of articles in languages

I'm interested in finding a solution for the following problem: problem space: any language that has more than 1 article let's take German language as example. So the articles in German are "der", ...
2
votes
4answers
244 views

Optimal algorithm for finding all ngrams from a pre-defined set in a text

I'm thinking about the optimal algorithm for the following problem: Input data: a text, say it's an article about 5-50 pages. a set of ngrams (ngram strings, n>2), of arbitrary length, could be ...
3
votes
0answers
212 views

Has someone seen this structure before?

I am working 1 with a certain structure, and I wonder if someone has seen it before. I am no mathematician, so all I can say is that I will do my best to describe this structure. It is actually very ...
5
votes
2answers
287 views

Algorithm to find the probability of a given text to be about a large topic

I want the conditional probability for each topic (being the word that we give as input). For example, the text being have seen and reviewed your requirements you posted here. If you can give ...
4
votes
1answer
163 views

Semantic similarity in text

Is there a relatively simple way of telling if two pieces of text are semantically similar? Some assumptions that are valid: It is all english I have a list of all the important nouns Are there ...
3
votes
1answer
776 views

Semantic clustering

I have a very specific question about semantic clustering. I have a list of words/phrases. I want to run an intelligent semantic clustering algorithm on this list. Please let me know what the ...
17
votes
4answers
825 views

Finding interesting anagrams

Say that $a_1a_2\ldots a_n$ and $b_1b_2\ldots b_n$ are two strings of the same length. An anagramming of two strings is a bijective mapping $p:[1\ldots n]\to[1\ldots n]$ such that $a_i = b_{p(i)}$ ...
4
votes
2answers
105 views

When did commercial Speech Recognition first begin using grammar (sentence structure) for prediction?

It seems as though modern speech recognition (e.g., through Android, iOS phones) make use of grammar or sentence structure. (e.g., it might have a tough time distinguishing between "grammar" and ...
11
votes
2answers
192 views

Identifying events related to dates in a paragraph

Is there an algorithmic approach to identify that dates given in a paragraph correlate to particular events (phrases) in the paragraph? Example, consider the following paragraph: In June 1970, ...
12
votes
1answer
356 views

Compression of domain names

I am curious as to how one might very compactly compress the domain of an arbitrary IDN hostname (as defined by RFC5890) and suspect this could become an interesting challenge. A Unicode host or ...