Hi Jordan,

Do you see an easy way to assign ROI labels to edges in the neuprint executor? Let's say I want to query something like this:

A -> B [weight > 20, ROI == "CX"]
A -> B [weight > 30, ROI == "CRE(L)"] 

So basically, there are two things here—multigraphs, which you address already in the docs, and encoding edge ROIs. I wonder if that's rather a hard thing to do or not. The data should be there as neuprint-python fetch_synapse_connections returns something like this

    bodyId_pre  bodyId_post roi_pre roi_post  x_pre  y_pre  z_pre  x_post  y_post  z_post  confidence_pre  confidence_post
0    792368888    754547386  PED(R)   PED(R)  14013  27747  19307   13992   27720   19313           0.996         0.401035
1    792368888    612742248  PED(R)   PED(R)  14049  27681  19417   14044   27662   19408           0.921         0.881487
2    792368888   5901225361  PED(R)   PED(R)  14049  27681  19417   14055   27653   19420
...

According to this issue it looks like it's possible. My observation is that the physical location of a connection between two neurons is an important feature of a motif. Looking forward to hearing what you say.

EDIT: Maybe an indirect way to support multiple edges between two nodes is by grouping edge attributes. Does something like this seem plausible. You are doing smth similar in the multigraph docs already: A -> B [synapse_count > 2]. But what exactly is synapse_count?

A -> B [[weight >= 20, ROI == "CX"], [weight > 30, ROI == "CRE(L)"]]

Best, Jakob

Tried to run the query from the tutorial:

motif = Motif("""
# My Awesome Motif

Nose_Cell -> Brain_Cell
Brain_Cell -> Arm_Cell
""")

But got this error:

FileNotFoundError                         Traceback (most recent call last)
<ipython-input-1-3a88159c0a0c> in <module>
----> 1 import dotmotif
      2 import networkx
      3 
      4 motif = Motif("""
      5 # My Awesome Motif

~\anaconda3\lib\site-packages\dotmotif\__init__.py in <module>
     24 from networkx.algorithms import isomorphism
     25 
---> 26 from .parsers.v2 import ParserV2
     27 from .validators import DisagreeingEdgesValidator
     28 

~\anaconda3\lib\site-packages\dotmotif\parsers\v2\__init__.py in <module>
     11 
     12 
---> 13 dm_parser = Lark(open(os.path.join(os.path.dirname(__file__), "grammar.lark"), "r"))
     14 
     15 

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\xxxx\\anaconda3\\lib\\site-packages\\dotmotif\\parsers\\v2\\grammar.lark'

Hey There, I'm using dotmotif to query the neuPrint dataset and have found some of the neurons have properties that aren't accepted in the query string format e.g. 'AVLP(R)': True,

Is there a way to still query w/ these params? I tried adding directly to the _node_constraints but that doesn't seem to work either e.g.

motif._node_constraints['A']['AVLP(R)'] = {}
motif._node_constraints['A']['AVLP(R)']['='] = [True]

Variable `R` not defined (line 2, column 83 (offset: 156))
"    WHERE B.status = "Traced" AND A.status = "Orphan" AND A.INP = True AND A.AVLP(R) = True"

The edge attribute in the neuprint executor threw an error with the new JSON feature implementation. I also made the neuprint executor tests more rigorous.

In grandiso v1.1.0 and above, there is an optional limit argument to the find_motifs call which short-circuits motif counting if a certain number of valid mappings are found.

Right now, NetworkX and GrandIso executors implement the dotmotif limit parameter by finding all motifs and then downselecting, which is super inefficient and lame. We could pretty substantially improve performance by supporting the GrandIso limit arg.

A notable challenge: We perform an additional downselect after running grandiso (to double-check attribute filters). So we may need to store a list of mappings temporarily in order to backfill the results list if candidate mappings are filtered out.

Ingesting a NetworkX graph with integer ids results in an error: ValueError: Could not export graph: unsupported operand type(s) for +: 'int' and 'str'. It should be straightforward to handle integers, though A node can be any hashable Python object except None. Maybe just cast with repr.

Currently, the parser overwrites previous operators if it's redefined:

A -> B [value<=5, value<=2]

...will yield a constraint operator of

{ "value": { "lte": 2.0 } }

(i.e. overwriting the first rule).

Proposed syntax concepts:

Nodes

Inline maplike:

Node1 { type="GABA", z<12 } -> Node2

Pros:

• Succinct

Cons:

• Possible duplication or conflicting attributes if map is included on multiple lines for the same node

Postfix where-like:

Node1 -> Node2 | Node1.type = "GABA", Node1.z < 12

Pros:

• Succinct

Cons:

• Possible duplication or conflicting attributes if attrs are included on multiple lines for the same node

Footnote constraints

Node1 -> Node2

Node1.type = "GABA"
Node1.z < 12

Pros:

• Reduces possibility of conflicting constraints
• Clear syntax; can be standalone in its own macro

Cons:

• Linecount verbose
• Decouples attributes from connectivity clauses

Edges

Inline maplike:

A ->{type: "excitatory", neurotransmitter: "ACh"} B

Pros:

• Inline

Cons:

• Reduces clarity of language

Postfix where-like:

A -> B | [type: "excitatory", neurotransmitter: "ACh"]

Pros:

• Inline

Cons:

• Reduces clarity of language

Infix maplike:

A -[type: "excitatory", neurotransmitter: "ACh"]> B

Pros:

• Inline

Cons:

• Reduces clarity of language

This adds support for complex edge constraints in macros:

decreasing_edge_weights(a, b, c) {
    a -> b as ab
    b -> c as bc
    ab.weight > bc.weight
}

...

In increasing levels of challengingness:

• [x] Add support for simple (edge-value) edge constraints in macros
• [x] Add support for dynamic (edge-edge) edge constraints in macros
• [x] Extend support for recursive calls to macros with simple constraints
• [x] Extend support for recursive calls to macros with dynamic constraints
• [x] Update documentation

This fixes #110 and finishes work started in #119.

This PR adds support for edge aliases (first described in #110) and comparisons between edge attributes with values and with other edges.

This enables syntax like this:

A -> B as ab
B -> A as ba

ab.weight > ba.weight

• [x] Add support in the DSL
• [x] Add support in the parser + transformer
• [x] Add support in the executors:
  • [x] GrandIso
  • [x] NetworkX
  • [x] NeuPrint
  • [x] Neo4j

I am going to push macro support in a separate PR, since this one is getting pretty lengthy already!

Adds support for "bracket" syntax for node attributes. An attribute like XYZ(ABC) or ABC DEF used to be disallowed because of illegal characters in the attribute name, particularly when using the "dot-attribute" notation:

# broken:
A -> B
A.ABC DEF > 10

The new syntax uses bracket-attribute notation to "escape" these names:

# working:
import dotmotif
from dotmotif.executors.NeuPrintExecutor import NeuPrintExecutor

HOSTNAME = "neuprint.janelia.org"
DATASET = "hemibrain:v1.2.1"
TOKEN = "[YOUR TOKEN HERE]"

motif = dotmotif.Motif("""
A -> B
A['AVLP(R)'] = True
""")

E = NeuPrintExecutor(HOSTNAME, DATASET, TOKEN)

E.find(motif, limit=2)

Fixes #111.

We should be able to automatically catch things like this:

A.type = 4
A.type != 4

Right now, we'll catch them in certain instances, but not when constraints are inherited from automorphisms (see #118). Getting smarter about this will likely improve runtime considerably.

Anonymous motif participants:

A -> _hidden
_hidden -> B

Anonymous node participants in macros:

two_hop(A, B) {
    A -> _i
    _i -> B
}

two_hop(neuron1, neuron2)