Thank you for contacting us. We're currently working with Dr. Stacy Johnson to automate the process of identifying PE and DVT patient cohort. For the PE task, we want to determine whether a patient has PE or not, if so, whether the PE is acute or chronic and its artery level. For the DVT task, we want to determine whether a patient has VT or not, superficial or deep, proximal or distal, acute or chronic, left or right, upper limbs or lower limbs. For both tasks, we also want to determine if the report impression is certain or uncertain. He is very active to push the project forward and has made a lot of annotations for NLP training and evaluation.
For the DVT task, I exported 400 DVT reports with preannotations (conclusions from preliminary …show more content…
But that data is not completely reliable due to report id mismatched issue. We've gotten apparently good performance on visit level. To make sure the pipeline has good performance on document level, I exported another 400 chest CT reports for him to review. Haven't gotten them back yet.
For the rest of project, I think there are some works that you can help, but Dr. Johnson and Reed would be the ones to decide where are we heading. If Dr. Johnson hasn't finished the 400 PE reports review, I think you can work on the rest.
If we are focusing on the operational side, then this project is almost close to the end. However, scientifically, there is an issue that we haven't addressed before. That's the reliability of the annotated data, because we only have one expert annotated the data. Because there are some cases that different experts may have different opinions, we often need multiple experts to make the annotated data, so that we can estimate how much percentage of the total data may result such disagreement using the interannotator agreement (IAA). Then we can more reliably estimate how NLP performs. For instance, if the IAA is around 80%, but NLP performance appears to be above 95%, we cannot trust that performance report for sure. On the contrary, if IAA is around 95%, and NLP performance is obviously lower than that, then we are sure that there is a room for NLP to
For the DVT task, I exported 400 DVT reports with preannotations (conclusions from preliminary …show more content…
But that data is not completely reliable due to report id mismatched issue. We've gotten apparently good performance on visit level. To make sure the pipeline has good performance on document level, I exported another 400 chest CT reports for him to review. Haven't gotten them back yet.
For the rest of project, I think there are some works that you can help, but Dr. Johnson and Reed would be the ones to decide where are we heading. If Dr. Johnson hasn't finished the 400 PE reports review, I think you can work on the rest.
If we are focusing on the operational side, then this project is almost close to the end. However, scientifically, there is an issue that we haven't addressed before. That's the reliability of the annotated data, because we only have one expert annotated the data. Because there are some cases that different experts may have different opinions, we often need multiple experts to make the annotated data, so that we can estimate how much percentage of the total data may result such disagreement using the interannotator agreement (IAA). Then we can more reliably estimate how NLP performs. For instance, if the IAA is around 80%, but NLP performance appears to be above 95%, we cannot trust that performance report for sure. On the contrary, if IAA is around 95%, and NLP performance is obviously lower than that, then we are sure that there is a room for NLP to