Natural language processing (NLP)

learn more… | top users | synonyms (1)

-1
votes
0answers
14 views

Trouble extracting only related text from the pdf file? [closed]

The use of Beautiful Soup is not needed here. I have created pdf file and now trying to use pdfminer to extract only text. The problem is how to remove additional text which is the part of ...
2
votes
1answer
23 views

Specific case of a ranking model

So the problem I'm solving is this: I have a list of conversations of 3 messages each (for eg. "hi", " how are you", "remind me to fix this bug" is one conversation, and my problem will have many of ...
0
votes
0answers
20 views

Measure semantic similarity between source codes

Currently, I have a system who measure the similarity between two answers (source codes) analizing the similarity of the codes and the outputs. I need to make it more accurate. A good idea from were ...
0
votes
0answers
20 views

How can I change the text style?

There are algorithms to change the forms of words, to define the subject of the text, as well as the definition of his style (literary, business, scientific, etc.). Do you have any ready ideas or ...
0
votes
0answers
22 views

What is the difference between the Lexical and Character features?

I was reading about Natural Language Processing, I found that there are lexical features and character features that can be extracted of a text. I understood that the lexical features are more focus ...
2
votes
1answer
85 views

How to calculate an accurate estimated reading time of text?

I suppose the calculation should not be done by only two factors (average reading speed/words per minute, and word count). But at least by a third parameter, that in my opinion should measure the ...
2
votes
1answer
33 views

What is the Best and easiest way to create a Classifer for Sentiment Analysis [closed]

Sentiment analysis using Machine Learning is a hot topic. In the present situation when a person doesn't have a problem in having the training data set then which way should we create the classifier ...
2
votes
0answers
17 views

How to determine agreement between two sentences?

A common Natural Language Processing (NLP) task is to determine semantic similarity between two sentences. Has the question of agreement/disagreement between two sentences been covered in NLP or other ...
0
votes
0answers
4 views

How are the ideas from the paper “Modelling Linguistic Expressions Using Fuzzy Relations” currently used in NLP?

The paper written by De Cock, Bodenhofer, and Kerre. A link to the paper: Modelling Linguistic Expressions Using Fuzzy Relations
0
votes
0answers
20 views

Examples for speech recognition systems and spoken dialogue systems

I am collecting material for a MOOC about speech technology. My aim is that students also have examples to try rather than just watching the lecture and some complimentary youtube videos. So the idea ...
10
votes
1answer
529 views

Implementation of Naive Bayes

I am implementing a Naive Bayes algorithm for text categorization with Laplacian smoothing. The problem I am having is that the probability approaches zero because I am multiplying many small ...
3
votes
0answers
43 views

What is the state-of-the-art in machine translation on Chinese/CJK? [closed]

Sorry for my unfamiliarity with the field. Hopefully this question is on-topic here. Currently the machine translation from western European languages to English is arguably quite robust, with Google ...
0
votes
0answers
36 views

Can valid sub-parses be added to an earley chart after a successful parse has been recognized?

I'm implementing the Earley algorithm and am trying to build a parse forest from the output of the recognizer. However in the process of doing so I got this output from an ambiguous CFG, where a non-...
4
votes
1answer
85 views

Proving that English is not a regular language

I am wanting to try and prove that the English language is not regular. The alphabet is the set of all words in the English dictionary. Looking at sentences, I was able to use this pattern of ...
1
vote
1answer
27 views

Question about bigram model

I am trying to build a bigram letter model. I obtain a sequence of words in a form of ['hello','I','am','Johnny']. Firstly, I lower all the words to obtain : ['hello','i','am','johnny']. I am ...
1
vote
1answer
28 views

NLP Classification: rule-based approach providing high precision but low recall

In NLP, rule-based/linguistic approaches are usually said to provide relatively high precision but low recall compared to for example machine learning approaches. I think this is often true for almost ...
0
votes
0answers
35 views

How can feature structures be applied to text summarization?

I'm looking for innovative ways to create text summaries and I was wondering whether feature structures can be used for such a task. Is it possible to create subsume and unification methods that ...
1
vote
0answers
108 views

Is English Recursively enumerable? [closed]

The title says it all. I've tried digging up debate on this issue to see a proof one way or the other but it doesn't look like anyone is able to say whether or not it is. Clearly there are recursive ...
0
votes
1answer
39 views

How to extract best illustrative sentences from documents for a given word? [closed]

I want to extract "best" sentences from the given (html) documents. Let me illustrate this with an example. I have a word "determine". Now I have hundreds of html files having paragraphs and sentences ...
1
vote
1answer
47 views

What's this type of semantic analysis called in NLP?

I'm speculating that this is very much known in NLP, but as I've not studied NLP, I don't know what concepts are related to this: Basically I have entities that are connected by some strings. I ...
0
votes
1answer
30 views

In NLP, does the lexer have to tag the tokens before the parser?

In NLP, does the lexer have to tag the tokens before the parser? I.e. does the lexer have to classify the tokens to morphological categories before the parser? I'm thinking yes, but is this also the ...
0
votes
1answer
76 views

Where can I find a reasonable NLP CFG or CSG for English?

I'm looking to do natural language parsing and looking for how the CFG (or CSG) should be defined for English. Surely one'd expect to find one from the internet already, but do you know where? It ...
0
votes
0answers
25 views

Common models for inferring semantics (truth/factuality of statements)?

What are common models for creating rules for inferring meanings (e.g. truth values) of natural language statements such as if one wanted to infer the truth value of the input statement John ...
-1
votes
1answer
22 views

Context-free grammar for “member of”/set membership?

I'm wondering whether context-free grammars (or what else) can be used to implement "member of" structures, which are structures for describing that something is a member of something. E.g. I want to ...
0
votes
1answer
27 views

How to detect plurals in English sentences?

I'm wondering how to detect English plurals in natural language sentences. Such as: fish computers houses ... Using Earley parsing I'm able to get to the s or no-s, but I have difficulties in ...
1
vote
0answers
36 views

Parsing based on logical connectives and quantifiers?

What techniques exist for parsing sentences based on logical connectives and quantifiers? That is, for parsing sentences that are structured "around" logical connectives and quantifiers. E.g. [A] ...
1
vote
0answers
28 views

Algorithm to change article reading level

I was wondering, are there any famous algorithms in NLP or NLG that can take some text (say a news article online) and transform it to a lower reading level?
2
votes
1answer
66 views

Language grammar correction with supervised learning

I want to work on automatic grammar correction using machine learning (possibly using recurrent or deep neural networks). The algorithm will be supplied with both corrected and initial documents for ...
1
vote
1answer
31 views

NLP: Automatically categorize words semantically into non-predefined classes

I am looking for a method to categorize a given word into semantically into classes. A simple example would be: chair -> furniture The most reasonable way to ...
8
votes
0answers
78 views

What are some efficient ways to find the differences between two large corpuses of text that have similar, but differently ordered content?

I have two large files containing paragraphs of English text: The first text is about 200 pages long and has about 10 paragraphs per page (each paragraph is 5 sentences long). The second text ...
2
votes
1answer
64 views

Scientific soundness of computer science papers [closed]

First of all, this question relates to an issue that affects several areas of science, but since StackExchange doesn't have a meta science section I'll make it specific to computer science, which is ...
1
vote
2answers
99 views

Computing probability of sentence using N-grams

I have implemented N-grams by constructing a tree (or a trie, technically) that stores frequencies of each N-gram. Each path in the tree represents an N-gram and its frequency: the path consists of N ...
2
votes
0answers
18 views

Standard training and test corpuses for POS tagging

I'm wondering if there is a standard corpus that people use to evaluate their part-of-speech taggers. I am implementing my own tagger, and would like a meaningful way to measure its performance, ...
3
votes
0answers
38 views

Infer probabilities, for concatenation of words

Fix an alphabet $\Sigma$, and a set of words, $W = \{w_1,\dots,w_n\} \subseteq \Sigma^*$. I have a randomized model that works like this: Alice generates a random sequence of words, using some ...
4
votes
2answers
164 views

Are syntax and semantic just 2 structures such that one is a model of the other?

The syntax of a language is a structure. The semantic of a language is a structure. The semantic of a language is a model of its syntax. And that's all ? The duality syntax/semantic is just model ...
0
votes
0answers
173 views

Most frequently auto-tagging algorithms for text documents

Please if it's possible introduce the newest and most frequently auto-tagging algorithms for auto-tag text document. I want to get one text document and export tag for that. I also saw TF-IDF but it ...
2
votes
1answer
45 views

How to compute the TF-IDF scores for a handful of documents, but without my own corpus?

I have a small number of documents (I could probably get 4 or 5 documents) and want to assign terms in these documents a score based on its importance to the document. I want to find which terms are ...
2
votes
0answers
27 views

How to store linguistic data?

I'm both interested in practical and theoretical aspects of the problem. How do you store linguistic data. Suppose you have stemmed, parsed, tagged and, perhaps even did some preliminary analysis, ...
3
votes
1answer
80 views

Choice of machine learning algorithms frequency of parts of speech

I'm new to machine learning. I have text, and I tag the text according to their parts of speech tag ie walk is tagged as verb, etc. I tag entire sentences, and then convert them into a vector based on ...
2
votes
1answer
37 views

How do IR researchers evaluate the ranks of documents?

I am developing a new IR system in a specialized context. I understand that a traditional IR system (like a search engine) should rank documents in terms of their relevance for a query. The most ...
2
votes
1answer
63 views

Most frequently tools or programming language for implementation text processing and nlp algorithms in academic papers and journals [closed]

I want to prototype and try some idea (some algorithm) in the field of text processing and nlp and if the results was good I want to publish some paper or journal article about that. I am familiar ...
1
vote
2answers
301 views

Detect if a sentence is in passive voice

Given an English sentence, I am looking for a programmatic way to tell whether the sentence is written in passive voice. Currently, I just check if there is a was ...
2
votes
1answer
52 views

Looking for a paper comparing NLP methods with Data Mining techniques

I've recently attended a conference, where one of the participants mentioned a recent paper published by a Google employee, which showed that using data mining techniques in application to NLP might ...
1
vote
1answer
48 views

NLP - Deep belief neural networks or logic?

The NLP progresses seems to me that has split in two big group of thoughts: Deep learning have a neural network with multiple layers that has been trained and learned to represent knowledge and ...
1
vote
1answer
21 views

Where can I find a study on the amount of different word frequencies in a corpus? [closed]

I need to know how many different values the frequency of a certain word in a corpus can there be for a natural language processing problem. Is there any study or site that has such estimation?
2
votes
1answer
92 views

Natural Language Parser that can handle syntactic and lexical errors

I have some background in natural language processing and I know that all parsers (top down or bottom up, or mix), at least when I studied just about a few years ago, cannot handle any error. A small ...
3
votes
1answer
72 views

Natural language query processing

I am trying to implement a natural language query preprocessing module which would, given a query formulated in natural language, extract the keywords from that query and submit it to an IR (...
6
votes
3answers
615 views

Complexity of natural language processing problems [closed]

Which natural language processing problems are NP-Complete or NP-Hard? I've searched the natural-lang-processing and complexity-theory tags (and related complexity tags), but have not turned up any ...
2
votes
1answer
25 views

How is wordnet curated?

I use Princeton's WordNet in NLP applications. I always read that it is "human-curated." But how is it curated? Who decides that words belong in a synset? How does the process work? Wikipedia does not ...
2
votes
3answers
298 views

Does programming language detection need more input than natural language detection?

I wonder which one of the two needs a larger input to achieve a decent accuracy: programming language detection or natural language detection? More details: Definition of Language detection: ...