0
$\begingroup$

I have a medical dataset, which is something like this

| Disease | Symptoms |
| -------- | -------- |
|  Disease 1  |  ['Symptom1', 'Symptom2', 'Symptom3', ...]  |
|  Disease 2  |  ['Symptom2', 'Symptom11', 'Symptom7', ...]  |
|  Disease 3  |  ['Symptom55', 'Symptom91', 'Symptom65', ...]  |

I want a generate a model that when given a set of symptoms as input will be able to relate those symptoms to symptoms in the dataset and then give n-nearest diseases for the given input.

I have tried to run KNN on it, by converting the symptoms into feature vectors with encoding but it does not solve my problem as the input set can contain symptoms that are not the same logically but they mean the same.

For reference the example dataset.

$\endgroup$
3
  • $\begingroup$ "not the same logically but they mean the same": what ?? $\endgroup$
    – user16034
    Jun 7, 2023 at 10:57
  • $\begingroup$ @YvesDaoust the model should be able to go through a user input, extract the symptoms, and then use them to classify a disease for the symptoms in a user dialogue. But when in the input a symptom is not mentioned in the same way as a string in the db. The model should understand the meaning of phrases or words from the input and then should be able to figure out if a symptom is present or not even though the wording does not have the same in short not the same string value. $\endgroup$
    – Ashrjz
    Jun 8, 2023 at 13:48
  • $\begingroup$ I guess that the qualifier "logically" is not appropriate then. $\endgroup$
    – user16034
    Jun 8, 2023 at 14:06

2 Answers 2

0
$\begingroup$

I think what you could do is do a semantic relation first, such as find similarities between the terms first then do the encoding. So if you use a pre-trained model such as BERT or GPT, then you can find the semantic similarities of the symptom terms first. Then after finding the similarities you can group them together to encode the similar symptoms together then feed them into the KNN-classifier.

$\endgroup$
0
$\begingroup$

IMO, the various symptom descriptions should be classified by a human and regrouped. Later, when an unknown expression is fed to the system, it should be scrutinized by an expert. For every group, you can choose a canonical representative or just a number. Better use the Hamming distance than a geometric metric, as the numbers do not relate to proximity/similarity.

I would not rely on automatic synonym or paraphrase analysis, especially as regards health issues.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.