Skip to main content
10 votes

any hope for a universal automatic parser?

You might be interested in learning about grammar induction: given a set of examples of strings from a context-free language, there are algorithms to learn a context-free grammar that generates those ...
D.W.'s user avatar
  • 162k
10 votes

Data Science vs Operations Research

While both Operations Research and Data Science both cover a large amount of topics and areas, I'll try to give my perspective on what I see as the most representative and mainstream parts of each. ...
mhum's user avatar
  • 2,146
6 votes

Data Science vs Operations Research

This isn't a full answer, since mhum's is quite good in contrasting the differing aims of OR vs DS. Rather, I want to address this comment of yours: I was wondering if, for example, one could use ...
A. G.'s user avatar
  • 261
6 votes

Detecting events in time series data

Your first step is to characterize what effect you expect an event to have on your signals. Does it change the mean? Increase the mean? Change the variability? The more you can say about the type ...
D.W.'s user avatar
  • 162k
6 votes

What kind of logarithm does $\log$ mean in Wikipedia articles?

It depends entirely on context. In mathematics as a whole, $\log$ usually denotes the natural logarithm (base $\mathrm{e}$). In computer science, the situation isn't as clean because we often ...
David Richerby's user avatar
4 votes

Measuring the information of a document?

Assuming that your data comes from a Markovian source, you can estimate the entropy of the source using an optimal compression algorithm such as Lempel–Ziv, whose theoretical version (without limiting ...
Yuval Filmus's user avatar
4 votes
Accepted

Computer vision: object detection with labels that are single coordinates

The state of the art in such problems is done these days via deep neural networks. Among others, two popular and recent approaches for solving the problem of detection and localization of objects are ...
Nathaniel Bubis's user avatar
4 votes

How to generate response variable during machine learning?

No. You can't. You need ground truth. You're asking "if I don't know which claims are fraudulent, can an algorithm somehow determine that for me?" The answer of course is no: the algorithm doesn't ...
D.W.'s user avatar
  • 162k
3 votes

Data Science vs Operations Research

As a strategist, I've had the opportunity to work with both sides of the discipline. In trying to explain what OR and DS are to a qualitative MBA executive, my (overly) simplistic one line ...
user88056's user avatar
3 votes

Data Searching from a large data set without reading each element

Say you have $n$ entries, $m$ distinct letters, and the $i$-th letter occurs $k_i$ times. You can find the last occurrence of the $i$-th letter using a variant of binary search: Double your step size ...
adrianN's user avatar
  • 5,961
3 votes

Machine Learning: Identify Patterns in Time-Series Data

Yes, your data is "time-series data", since it's a sequence of measurements of the same variable collected over time. Time-series data can be collected continuously or at discrete intervals. Your ...
rphv's user avatar
  • 1,624
3 votes
Accepted

In Data Mining, what does it mean to be greedy?

I'm not a Data Mining expert, but from what I understand, it means the same thing as it does outside of data mining: to improving the solution in a locally optimal way, rather than one that is ...
Joey Eremondi's user avatar
3 votes

Word Frequency with Ordering in O(n) Complexity

The gathering of occurrence counts is O(n), so the trick is really only finding the top k occurrence counts. A heap is a common way to aggregate the top k values, although other methods can be used (...
KWillets's user avatar
  • 1,274
3 votes

Gramatical evolution doesn't result on creating good constants

In Grammatical Evolution there are three well known approaches to the problem of constant creation: expression based (the "traditional" approach). This is what you're using (arithmetic operators are ...
manlio's user avatar
  • 2,062
3 votes

Gramatical evolution doesn't result on creating good constants

It sounds like you're saying that your approach is able to effectively find the shape of the expression, but it takes it a lot longer to find the right constants. One possible approach is to generate ...
D.W.'s user avatar
  • 162k
3 votes
Accepted

Why does the BFR (Bradley, Fayyad and Reina) algorithm assume clusters to be normally distributed around its centroid?

Roughly, the algorithm needs to estimate the probability to assign a point the correct cluster. So the algorithm add P to a cluster if it is very unlikely that, after all the points have been ...
user1315621's user avatar
3 votes

Why is it not always possible to compute the centroid of feature vectors?

A metric space consists of a set $X$ of "points" and a metric $d\colon X \times X \to \mathbb{R}_{\geq 0}$ (giving the "distance" between any two points) which satisfies the following constraints: ...
Yuval Filmus's user avatar
2 votes

How to generate response variable during machine learning?

Let me know of any other methods to generate response variable when there is none Anomaly detection techniques could be what you're looking for (but you don't "generate response variable when there ...
manlio's user avatar
  • 2,062
2 votes
Accepted

About the MNIST data-set

The short answer is: I don't know. Visualizations There's been lots of work on visualizing the MNIST data set, in 2 dimensions, and even in 3 dimensions. For instance, here's an embedding into 2D ...
D.W.'s user avatar
  • 162k
2 votes

Machine Learning - Support Vector Machines

In addition to listed references, I would recommend the following: A. Nefedov. Support Vector Machines: A Simple Tutorial, 2016 N. Cristianini, J. Shawe-Taylor. An Introduction to Support Vector ...
Leo's user avatar
  • 121
2 votes

A decision algorithm to choose what partner to use

Techniques developed for the multi-armed bandit problem might be useful for discovering the partner with the best acceptance rate and dealing with the changing acceptance rate over time. Those ...
D.W.'s user avatar
  • 162k
2 votes
Accepted

Finding (and possibly extracting) source code in heterogenous text data set

There are many possible ways you might do this. I'll suggest one. I suggest you train a classifier to recognize whether a sequence of characters is code or text. Here, the goal is to build a ...
D.W.'s user avatar
  • 162k
2 votes

Data Science vs Operations Research

I obtained my Operations Research degree from a military institution and have been using it for military applications for about 15 years. In that timeframe, Data Science has grown into a main-stream ...
Xavier Lugo's user avatar
2 votes
Accepted

What kind of logarithm does $\log$ mean in Wikipedia articles?

I think it depends on the situation. For a mathematician, $\log n$ probably means the natural logarithm. For a computer scientist, $\log n$ probably denotes the base $2$ logarithm, etc. I think ...
zdm87's user avatar
  • 68
2 votes
Accepted

Find plane within margin of error of >50% of points

One reasonable approach is to use RANSAC. Pick three points. Find the plane that goes through them. Iterate through all the points and keep the points that are near that plane (the inliers) and ...
D.W.'s user avatar
  • 162k
2 votes
Accepted

In a set of sentences, how could I determine the fewest sentences that contains all characters?

This is an instance of set cover. It is NP-hard, so there is no efficient algorithm to find the absolute optimal solution. The problem has been studied and there are many techniques out there, which ...
D.W.'s user avatar
  • 162k
2 votes

Machine learning and test split for time series data

In most contexts, random shuffling is absolutely required. If you do not randomly split the dataset, you might get misleading results. For instance, imagine that we have a dataset of patients, and ...
D.W.'s user avatar
  • 162k
1 vote
Accepted

Bias or not when finding patterns using data mining techniques?

In data mining, is it okay to start from expectations (bias) as to which patterns could be present and actually do statistics to see if this is actually the case Sure. This is called hypothesis ...
Martin Thoma's user avatar
  • 2,360
1 vote

How to figure out whether two texts refer to the same object or event

This is a document clusterization problem. A general solution is to define some sort of distance between documents and apply a standard clusterization algorithm, such as k-means or expectation ...
Maxim's user avatar
  • 640
1 vote

List count of occurrences pairs, triplets, etc. from sets

This is the itemset mining problem. Use any standard algorithm for that problem. Follow the links to learn more about standard algorithms for that task.
D.W.'s user avatar
  • 162k

Only top scored, non community-wiki answers of a minimum length are eligible