Skip to content

Commit

Permalink
Rewrites partitioned vid restoration in Perl
Browse files Browse the repository at this point in the history
  • Loading branch information
netj committed Jan 13, 2017
1 parent ec70d36 commit d1fe52a
Show file tree
Hide file tree
Showing 3 changed files with 9 additions and 6 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -196,12 +196,7 @@ if .deepdive_.execution.processes | has("process/grounding/combine_factorgraph")
for pid in $DEEPDIVE_FACTORGRAPH_SHARDS; do
cd \"$DEEPDIVE_RESULTS_DIR\"/$pid
cat inference_result.out.text |
# restoring shard ID to the vids
python3 -c "
import sys
for line in sys.stdin:
vid, cid, prob = line.split(); print('\t'.join([str(int(vid) + ($SHARD_BASE << 48)), cid, prob]))
" |
pid=$pid deepdive env restore_partitioned_vids |
DEEPDIVE_LOAD_FORMAT=tsv \\
deepdive load \(deepdiveInferenceResultVariablesTable | @sh) /dev/stdin
done
Expand Down
7 changes: 7 additions & 0 deletions inference/restore_partitioned_vids
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#!/usr/bin/perl -w
use strict;
my $SHARD_BASE = $ENV{pid} << 48;
while (<>) {
my ($vid, @rest) = split(" ");
print join("\t", ($SHARD_BASE | $vid), @rest), "\n";
}
1 change: 1 addition & 0 deletions stage.sh
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ generate-wrapper-for-libdirs "$STAGE_DIR"/util/sampler-$cmd \
done
stage inference/deepdive-model util/
stage inference/run-sampler util/
stage inference/restore_partitioned_vids util/

# Stanford CoreNLP utilities
stage util/nlp/deepdive-corenlp util/
Expand Down

0 comments on commit d1fe52a

Please sign in to comment.