Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mitigate model load diff calculation issues on neptune #149

Closed
dlamoris opened this issue Jan 23, 2024 · 0 comments · Fixed by #151
Closed

mitigate model load diff calculation issues on neptune #149

dlamoris opened this issue Jan 23, 2024 · 0 comments · Fixed by #151
Assignees

Comments

@dlamoris
Copy link
Contributor

During model load diff generation at https://github.com/Open-MBEE/flexo-mms-layer1-service/blob/v0.1.1/src/main/kotlin/org/openmbee/flexo/mms/routes/gsp/ModelLoad.kt#L220, ?srcGraph as a var can cause neptune to stall.
Example:

PREFIXES...
insert {
    graph m-graph:Transactions {
        mt:diff a mms:Transaction ;
            mms:created "2024-01-23T02:51:56.654793433Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
            mms:serviceId "mms5-layer1-0" ;
            mms:requestPath "/orgs/jama/repos/msr3/branches/master/graph" ;
            mms:requestMethod "POST" ;
            mms:authMethod ?__mms_authMethod ;
            mms:appliedGroupId ?__mms_groupId ;
            # mms:requestBody "" ;
            mms:requestBodyContentType "text/turtle" ;
            mms-txn:srcGraph ?srcGraph ;
            mms-txn:dstGraph mor-graph:Load.53b47e3b-4b6c-4044-a03f-ee721496770f ;
            mms-txn:insGraph ?insGraph ;
            mms-txn:delGraph ?delGraph ;
            mms:user mu: ;
            mms:org mo: ;
            mms:repo mor: ;
            mms:branch morb: ;
    
            .
    }
    graph m-graph:AccessControl.Policies {
        m-policy:AutoDiffOwner.6b6cda4d-8528-44d0-bb6c-439fc23e88aa a mms:Policy ;
            mms:subject mu: ;
            mms:scope mord: ;
            mms:role mms-object:Role.AdminDiff  ;
            .
    
    }
    graph ?insGraph {
        ?ins_s ?ins_p ?ins_o .    
    }
    
    graph ?delGraph {
        ?del_s ?del_p ?del_o .
    }
    
    graph mor-graph:Metadata {
        ?diff a mms:Diff ;
            mms:id ?diffId ;
            mms:createdBy mu: ;
            mms:srcCommit ?srcCommit ;
            mms:dstCommit <https://mms5.jpl.nasa.gov/orgs/jama/repos/msr3/commits/53b47e3b-4b6c-4044-a03f-ee721496770f> ;
            mms:insGraph ?insGraph ;
            mms:delGraph ?delGraph ;
            .
    }
    
    
    
}
where {
    graph mor-graph:Metadata {
            # select the latest commit from the current named ref
            mor-branch:master mms:commit ?srcCommit .
    
            # get the latest snapshot associated with the source commit
            ?srcCommit ^mms:commit/mms:snapshot ?srcSnapshot .
            {
                # prefer the model snapshot
                ?srcSnapshot a mms:Model ;
                    mms:graph ?srcGraph  .
            } union {
                # settle for staging...
                ?srcSnapshot a mms:Staging ;
                    mms:graph ?srcGraph .
    
                # ...if model is not available
                filter not exists {
                    ?srcCommit ^mms:commit/mms:snapshot/a mms:Model .
                }
            }
        }
    
    
    bind(
        sha256(
            concat(str(<https://mms5.jpl.nasa.gov/orgs/jama/repos/msr3/commits/53b47e3b-4b6c-4044-a03f-ee721496770f>), "\n", str(?srcCommit))
        ) as ?diffId
    )
    
    bind(
        iri(
            concat(str(<https://mms5.jpl.nasa.gov/orgs/jama/repos/msr3/commits/53b47e3b-4b6c-4044-a03f-ee721496770f>), "/diffs/", ?diffId)
        ) as ?diff
    )
    
    bind(
        iri(
            concat(str(mor-graph:), "Diff.Ins.", ?diffId)
        ) as ?insGraph
    )
    
    bind(
        iri(
            concat(str(mor-graph:), "Diff.Del.", ?diffId)
        ) as ?delGraph
    )
    
    
    {
        # delete every triple from the source graph...
        graph ?srcGraph {
            ?del_s ?del_p ?del_o .
        }
    
        # ... that isn't in the destination graph 
        filter not exists {
            graph mor-graph:Load.53b47e3b-4b6c-4044-a03f-ee721496770f {
                ?del_s ?del_p ?del_o .
            }
        }
    } union {
        # insert every triple from the destination graph...
        graph mor-graph:Load.53b47e3b-4b6c-4044-a03f-ee721496770f {
            ?ins_s ?ins_p ?ins_o .
        }
    
        # ... that isn't in the source graph
        filter not exists {
            graph ?srcGraph {
                ?ins_s ?ins_p ?ins_o .
            }
        }
    } union {}
}

This can cause Neptune to think for 30 min and times out

Replacing ?srcGraph with a definite graph lets the update finish in under 2 minutes

I think we'll have to do something similar to model query where the srcGraph is queried first.

@dlamoris dlamoris added this to Flexo Jan 23, 2024
@github-project-automation github-project-automation bot moved this to New in Flexo Jan 23, 2024
@dlamoris dlamoris moved this from New to Ready in Flexo Jan 23, 2024
@dlamoris dlamoris linked a pull request Jan 24, 2024 that will close this issue
@dlamoris dlamoris moved this from Ready to In Progress in Flexo Jan 30, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in Flexo Jan 31, 2024
@dlamoris dlamoris mentioned this issue Jan 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants