I have a data set that is similar to this:

  • 10,000 jobs with 200,000 applicants which are linked to a job.

I'm looking to cluster the shared jobs based on applicants that they share, am I reinventing the wheel looking to build this or is there a common solution?

Most of the example I found all require a "distance" but I can't see how I could compute one in this case?

Edit: To add, the clustering criteria would be fairly straightforward more than X applicants (say 5) in common, with only a single cluster per job.

  • $\begingroup$ Can you provide more detail on the criteria to cluster the jobs? In general, a question is expected to articulate enough detail so that an adequate answer could be identified or upvoted. $\endgroup$
    – John L.
    Mar 20 '19 at 21:26
  • $\begingroup$ Thanks @Apass.Jack I've added some more context - essentially I want to group where the job applicants intersect by more than 5 across jobs. $\endgroup$ Mar 20 '19 at 21:39

You could use the Jaccard distance to compare two jobs: i.e., the distance between two jobs is

$$d(A,B) = {|A \cap B| \over |A \cup B|},$$

where $A$ is the set of applicants for the first job and $B$ is the set of applicants for the second job. Then, use any standard clustering algorithm with this distance measure.

  • $\begingroup$ Thanks, so I'd execute that against all jobs and build a matrix from that then is then fed into a clustering algorithm? $\endgroup$ Mar 20 '19 at 21:40
  • $\begingroup$ @williamvicary, yes (all pairs of jobs, not all jobs). $\endgroup$
    – D.W.
    Mar 20 '19 at 22:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.