1
$\begingroup$

I have thousands of data points associated with users. So a single user can have 2000-10000 data points. These data points are identified by contiguous numbers (e.g. all numbers from 0 to 2000). Each data point can have 3 states: true, false, unknown.

What could I do to group these data points so that they don't take too much space?

My initial plan is to basically cluster them. Something like:

  • unknown: [1-23, 25, 27]
  • true: [24, 26]
  • false: [28-30]

Where m-n represents number from m to n.

Improve this question
$\endgroup$
7
  • $\begingroup$ A practical and simple solution is to use a three bits times 2000 per user. For a million users, this would take 750 megabytes. It's not hard to see you can compress this further too, but how many users do you have? How little space you require? $\endgroup$
    – Juho
    Mar 17, 2021 at 12:45
  • $\begingroup$ I'm probably over-engineering here. But it's not the space on the DB but the data sent to the user that has me thinking about this. I'll need to generate a chart with the data points. Probably just having a structure with each point and relying on the zipping functionality of the browser is the way to go. $\endgroup$
    – Artur Carvalho
    Mar 17, 2021 at 13:16
  • 1
    $\begingroup$ What is your question? Are you asking how to store them, or asking how to cluster them? If you're asking how to store them, why is it relevant to this question that you plan to cluster them? If you're asking how to cluster them, you will probably need to define some notion of similarity. Do you want to use Hamming distance on the sequences of states for the two users? Do you want to do some kind of alignment between the two sequences? Something else? I'm not clear on the requirements. $\endgroup$
    – D.W. ♦
    Mar 19, 2021 at 5:59
  • $\begingroup$ en.wikipedia.org/wiki/Run-length_encoding, en.wikipedia.org/wiki/Lossless_compression $\endgroup$
    – D.W. ♦
    Mar 19, 2021 at 6:01
  • $\begingroup$ @D.W. Let's say I want to send to an end users' browser 10K data points. Each data point can have 3 states: true, false, unknown. I want to make this operation fast for the end user. Because if a user has 10K data points with the state "on" I could just send an object {true:{from:0, to: 9999 }} instead of sending something like an array [true, true, ...] 10000 times. $\endgroup$
    – Artur Carvalho
    Mar 19, 2021 at 10:50

1 Answer 1

Reset to default
0
$\begingroup$

The solution was to use run-length encoding.

Let's say that I have:

  • The first 10 questions true
  • The next 24 false
  • The next 34 unknown

I'd transform it in 10T24F34U. You can find more info on Wikipedia's Run-length encoding as provided by D.W. in the comments.

The main reason for this solution is to send data via an API. I'm not sure if I'm going to use this algorithm or just rely on the typical compression when sending data to a browser. So that is another possible solution.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.