Network simulation tools

Overview

Overview

I'm building my network simulation environments with Vagrant using libvirt plugin on a Ubuntu 20.04 system... and I always hated how boring it was to set up a new topology and initial device configurations. The tools in this repository should make that task easier, and you're most welcome to submit pull requests adding support for other environments or platforms.

An overview of tools:

create-topology : Creates a full-blown network topology, Vagrantfile and Ansible inventory from a simple list of nodes and links. More details

config.ansible : Applies Jinja2 configuration template to network devices

initial-config.ansible : Using topology data generated by create-topology and default device configuration templates configures common device parameters, protocols that should have been enabled (LLDP), enables interfaces, and configures IP addresses on interfaces.

display-neighbors.ansible : Using Ansible fact gathering collect device neighbor data (usually LLDP neighbors) and displays them. Use YAML Ansible output callback for maximum readability.

connect.sh : SSH to a network using device names, management network IP addresses (ansible_host), SSH port, and username/passwords from Ansible inventory. Ideal when you use centralized Vagrant environments and want to connect to the devices while being in playbook development directory.

Libvirt-specific tools

enable-lldp.sh : Given libvirt network name, change group_fwd_mask for the corresponding Linux bridge to enable LLDP passthrough across the Linux bridge.

Comments
  • Default IP MTU

    Default IP MTU

    See https://github.com/ipspace/netsim-tools/issues/171

    I propose the following:

    defaults:
     ip_mtu: 9000
    

    (and then we update all device templates to apply this consistently)

    Do we need separate ipv4/ipv6 values?

    Implemented 
    opened by jbemmel 23
  • Renaming

    Renaming "netsim-tools" into "netlab"

    When I started developing netsim-tools I had absolutely no idea we would get as far as we got. It was meant to be a collection of tools that would help you build networking labs, not a whole-blown system with an abstraction layer supporting over a dozen protocols... so the name netsim-tools seems less appropriate than it did in the past.

    As the CLI interface already uses netlab command to do anything and everything (as opposed to the good old days when you had to run individual Ansible playbooks), it might make sense to rename the whole thing (including Python package) to netlab.

    Thoughts?

    /cc @ssasso @jbemmel @petercrocker (as I can't assign you to this issue for whatever weird reason)

    opened by ipspace 19
  • IBGP local-as sessions (if supported) on Dell OS10, VyOS

    IBGP local-as sessions (if supported) on Dell OS10, VyOS

    A quick glance at the recent code changes shows a new 'localas_ibgp' neighbor type. It is currently only referenced within the bgp module, but it can be present in the input for templates.

    This breaks logic like https://github.com/ipspace/netlab/blob/dev/netsim/ansible/templates/bgp/eos.macro.j2#L11 which assumes there are only 'ibgp' or 'ebgp' values.

    This may be a work in progress - just documenting what I noticed

    opened by jbemmel 16
  • Update eos.vrf.j2

    Update eos.vrf.j2

    For some reason jinja interprets the RD as possibly a sexagesimal number? TIL: https://en.wikipedia.org/wiki/Sexagesimal

    The effect is when ansible reads the host_vars file value rd: 65000:1 it interprets it as 3900001.

    TASK [eos_config: deploying initial from templates/initial/eos.j2] **************************************************************************************
    fatal: [h1xgw1]: FAILED! => changed=false
      data: |-
        rd 3900001
        % Ambiguous command
        localhost(config-s-ansibl-vrf-blue)#
      msg: |-
        rd 3900001
        % Ambiguous command
        localhost(config-s-ansibl-vrf-blue)#
    
    opened by petercrocker 15
  • Redesigning addressing

    Redesigning addressing

    Based on recent fixes and ongoing discussions (cc @jbemmel @ssasso) it's evident we need to redesign the IPv4/IPv6 addressing system. The solutions has to support at least:

    • Single-stack IPv4, single-stack IPv6, and dual-stack setup
    • Address pools, static link addressing, and static node addressing
    • Unnumbered IPv4 and IPv6 LLA (per-protocol unnumbered)
    • Unaddressed (l2only) links

    Ideally, we'd have an easy migration path and retain (temporary) support for existing unnumbered attribute.

    opened by ipspace 13
  • How to choose specific devices for given nodes

    How to choose specific devices for given nodes

    When attempting to implement the Multipath example using SROS/SRLinux:

    [email protected]:~/srlinux/netsim-examples/BGP/Multipath$ netlab create -d srlinux -p clab -s nodes.rr.device=sros -s nodes.m.device=sros
    Traceback (most recent call last):
      File "/home/jeroen/srlinux/netsim-tools/netlab", line 8, in <module>
        netsim.cli.lab_commands()
      File "/home/jeroen/srlinux/netsim-tools/netsim/cli/__init__.py", line 76, in lab_commands
        mod.run(sys.argv[arg_start:])   # type: ignore
      File "/home/jeroen/srlinux/netsim-tools/netsim/cli/create.py", line 67, in run
        read_topology.add_cli_args(topology,args)
      File "/home/jeroen/srlinux/netsim-tools/netsim/read_topology.py", line 98, in add_cli_args
        common.set_dots(topo,k.split('.'),v)
      File "/home/jeroen/srlinux/netsim-tools/netsim/common.py", line 112, in set_dots
        set_dots(b[k_list[0]],k_list[1:],v)
      File "/home/jeroen/srlinux/netsim-tools/netsim/common.py", line 112, in set_dots
        set_dots(b[k_list[0]],k_list[1:],v)
      File "/home/jeroen/srlinux/netsim-tools/netsim/common.py", line 108, in set_dots
        b[k_list[0]] = v
    TypeError: 'NoneType' object does not support item assignment
    
    documentation 
    opened by jbemmel 13
  • Convert link interface data into a list of dictionaries (was: s-hook links)

    Convert link interface data into a list of dictionaries (was: s-hook links)

    Some use cases require links that connect 2 ports of the same node. Due to the representation as a dict, these cannot currently be represented (type "p2p" results in an error)

    Proposal: new link type ("loopback" or "s-hook")

    opened by jbemmel 12
  • VLAN implementation / feature design

    VLAN implementation / feature design

    Starting the feature design discussion with my proposal.

    Some devices used on netsim may need to support VLANS / switchports on links ("L3 switch behaviour"). Some others may only need to support sub-interface tagging ("router behaviour").

    Generically speaking, a "switch node" may have the following attributes:

    nodes:
      switch_node:
        vlans:
        - id: 5
          prefix: 1.2.3.0/24
    

    (the prefix key will trigger an IP assignment to the SVI)

    At the same time, the link needs to have some additional attributes:

    links:
    - switch_node:
        switchport:
          mode: access
          native_vlan: 5
      host_node:
    - switch_node:
        switchport:
          mode: trunk
          vlans: [ 5 ]
          native_vlan: 1
      router_node:
        dot1q:
          vlan: 5
    
    opened by ssasso 11
  • Release 1.1

    Release 1.1

    Release 1.1 should go out in the next few days, potentially over the next weekend. Can't talk about plugins until we have a release that uses the new data structures, but I also want to work some more on the current code hoping to trip a bug or two.

    Here's the current to-do list:

    • Have to add a check of unnumbered capabilities (#107)
    • Would like to add #123 to stress-test configuration templates (in which case it would be great if @ssasso and @jbemmel could do a few tests once I'm done)
    • #84 would be nice to have (cc @petercrocker), but maybe it's worth waiting for #96
    • It would be great if @jbemmel could add #112 and #113 for SR Linux and SR OS
    • Increase code coverage (we added tons of new code)
    • Documentation

    Anything else you'd like to see in 1.1?

    Planned for 1.1.1:

    • Provider-specific device properties (#96)
    • Subnet object (#103)
    • MPLS module (#60)

    I will probably have to take a short break before I start working on 1.1.1 and do some leisurely reading. Fast Reroute and variants comes to mind ;)

    opened by ipspace 11
  • Add support for containerized VyOS.

    Add support for containerized VyOS.

    We can't use the official 1.3-epa2 image from Docker Hub because it's outdated and doesn't work correctly with virtual network devices [1]. The thing is critical for the container environment with veth devices. Hence for now we can use the unofficial image [2] that is based on VyOS nightly ISO.

    1. https://github.com/vyos/vyos-1x/commit/8cf5a4f023c5459cad4c84e93f73a9ddd69be81a
    2. https://github.com/sysoleg/vyos-container
    opened by sysoleg 10
  • If OSPF is used only on VRFs, OSPF deployment is miserably failing for almost all the templates

    If OSPF is used only on VRFs, OSPF deployment is miserably failing for almost all the templates

    Assume a topology similar to this one:

    vrfs:
      common:
        import: [ red, blue, common ]
      blue:
        import: [ blue, common ]
      red:
        import: [ red, common ]
    
    nodes:
      core1:
        module: [ bgp, ospf, vrf ]
        bgp.as: 65000
      aggr1:
        module: [ ospf ]
      aggr2:
        module: [ bgp ]
        bgp.as: 65111
        bgp.originate: [ 100.64.0.0/24 ]
      host1:
        module: []
        device: linux
      host2:
        module: []
        device: linux
      server:
        module: []
        device: linux
    
    links:
    - core1: { vrf: red }
      aggr1:
    - core1: { vrf: blue }
      aggr2:
    - aggr1:
      host1:
      ospf.cost: 22
    - aggr2:
      host2:
    - core1: { vrf: common }
      server:
    

    Where, on core1, all the interfaces are inside a VRF.

    The generated host_var file for Ansible does not have a ospf "root object", which leads to this hard failure:

        The task includes an option with an undefined variable. The error was: 'ospf' is undefined
    

    due to the fact that almost all ospf templates have sth like:

    {% if ospf.af.ipv4 is defined %}
    

    see i.e.,

    • https://github.com/ipspace/netlab/blob/dev/netsim/ansible/templates/ospf/eos.j2#L2
    • https://github.com/ipspace/netlab/blob/dev/netsim/ansible/templates/ospf/cumulus.j2#L1
    • https://github.com/ipspace/netlab/blob/dev/netsim/ansible/templates/ospf/ios.j2#L2

    I think we should always add the root ospf: object, when the ospf module is loaded.

    opened by ssasso 10
  • VRF and VLAN links

    VRF and VLAN links

    I like specifying links in the shorthand format (a-b instead of the full dictionary). That doesn't work well for VLAN and VRF links as they need additional attributes.

    We could solve that by adding links list to global VRFs and VLANs, for example:

    groups:
      routers:
        module: [ vrf ]
        members: [ r ]
    
    nodes: [ r, h1, h2, h3, h4 ]
    
    vrf:
      red:
        links: [ h1-r, h2-r ]
      blue:
        links: [ h3-r, h4-r ]
    

    VRF and VLAN links would

    • be transformed into the usual dictionary format
    • get vrf: X or vlan.access: X attribute
    • be appended to the global links list

    Feedback appreciated ;)

    enhancement 
    opened by ipspace 1
  • Split the topology-defaults file

    Split the topology-defaults file

    Topology-defaults file is becoming way too large (and will get much bigger when we implement #700). We should split it into numerous smaller files and combine them either at package publishing time or at runtime (depending on how time-consuming the whole thing is).

    Implementation is "trivial":

    • Topology defaults can have include attribute anywhere
    • include attribute specifies a glob relative to the topology-defaults.yml file
    • All files matched by the glob are read and inserted as a single key-value pair into the defaults at the level of the include directive. The key is the name of the file (minus the file type), the value is whatever was read from the file.

    For example:

    include: modules/*.yml
    devices:
      include: devices/*.yml
    providers:
      include: providers/*.yml
    

    Caveats:

    • This process would work only for package topology-defaults and would not be documented outside of developer documentation
    • Included files cannot have further include attributes. We have to stop somewhere, plus this nicely stops infinite recursion ;)

    Packaging optimization:

    • If needed, we'd have a script that would generate fully-evaluated topology-defaults.yml (or even a JSON file -- JSON is much faster to read) to be included in published Python packages.
    enhancement 
    opened by ipspace 2
  • More thorough attribute checking framework

    More thorough attribute checking framework

    Right now we have a generic framework that checks the validity of attribute names, the values of attributes are checked throughout the code. We could get rid of most of the value checks if we describe attributes better.

    Right now, the valid global/node/vlan/vrf/module... attributes are specified as lists of valid attribute names, for example:

    bgp:
      attributes:
        global: [
          af, as, next_hop_self, rr_cluster_id, rr_list, ebgp_role, as_list, sessions, activate,
          advertise_loopback, advertise_roles, community, replace_global_as ]
        node: [
          af, as, next_hop_self, rr, rr_cluster_id, originate, advertise_loopback, sessions, activate,
          community, router_id, local_as, replace_global_as ]
        vrf: [ router_id ]
        node_copy: [ local_as, replace_global_as ]
        link: [ advertise ]
        interface: [ local_as, replace_global_as ]
    

    Instead of using attribute lists, we could use dictionaries and (optionally) describe attributes with sub-dictionaries, for example (ignoring the question of whether we're supporting 4-byte ASN or not):

    bgp:
      attributes:
        vrf: 
          router_id: { type: 'ipv4' }
        link:
          advertise: { type: 'bool' }
        interface:
          local_as: { type: 'int', min_value: 1, max_value: 65535, value_error: 'BGP AS number should be between 1 and 65535' }
    

    We would need some shortcuts, like user-defined type (doesn't make sense to repeat 'what is AS' everywhere an AS is used) and simple inheritance (local_as can appear on node or interface).

    Migration process: attributes specified in the new format would go through the thorough value tests, attributes specified as lists would get their names checked. We could thus gradually migrate individual attributes or modules to the new framework.

    enhancement 
    opened by ipspace 0
  • Multi-provider topologies

    Multi-provider topologies

    Wouldn't it be nice if you could run a few nodes in VMs and a few others in containers? That's what I'll try to do with multi-provider topologies. Obviously we'll be limited in what can be done. I can see these easy combos:

    • libvirt + containerlab
    • anything + external (happily ignoring connectivity issues)

    Configuration would be extremely easy: add provider attribute to nodes to specify which provider a node should use. The main provider (top-level topology attribute) would have a list of allowed secondary providers so we'll stop the unsupported combinations.

    Using external provider with another one is a piece of cake (conceptually, still needs to be done): call the proper netlab up/down hooks at the proper time and move on.

    Combining libvirt and clab providers will be more fun.

    netlab create (or equivalent) will have to:

    • Set link type to bridge for any mixed-provider segment - libvirt provider hook
    • Create Vagrantfile
    • Modify the lab topology (remove libvirt nodes, clean up links...) - libvirt provider hook
    • Create prototype clab.yml - potentially from within the same libvirt provider hook

    netlab up will have to:

    • Execute vagrant up
    • Collect Linux bridge names from libvirt bridge names and modify the prototype clab.yml file
    • Start the containerlab part of the lab (potentially just execute containerlab deploy command)

    netlab down will have to:

    • Execute containerlab destroy command to bring down the containers
    • Execute vagrant destroy -f to bring down the VMs
    • Execute cleanup
    enhancement 
    opened by ipspace 1
  • Composite topologies

    Composite topologies

    Time to implement the discussion we had in #151:

    • A new top-level attribute components would allow you to define reusable lab components.
    • Each component would have links and nodes plus any valid top-level module attribute
    • Links within a component could refer to nodes within the component or to global nodes (not recommended)
    • Nodes within the component would have the usual nodes structure

    Components would be used within nodes dictionary. The key (node name) would be the prefix used to include the component (see below), the value would have to have an include attribute that would refer to a valid component. The value could also include any valid node-level attribute.

    Including a component would include all component nodes and links into the main nodes dictionary. Node names and references in links would be prefixed with the included component prefix. You could then use the prefixed names in top-level links to connect a component to the outside world.

    Example: two leaf-and-spine clusters connected with a superspine. IBGP+OSPF within each cluster, EBGP with superspine.

    defaults.device: frr
    
    components:
       pod:
          nodes: [ l1, l2, s1, s2 ]
          module: [ ospf, bgp ]
          bgp.as: 65000
          links: [ l1-s1, l1-s2, l2-s1, l2-s2 ]
    
    nodes:
      c1:
        bgp.as: 65100
      c2:
        bgp.as: 65100
      pod_1:
        include: pod
        bgp.as: 65101
      pod_2:
        include: pod
        bgp.as: 65102
    
    links:
    - c1-pod_1_s1
    - c1-pod_2_s1
    - c2-pod_1_s2
    - c2-pod_2_s2
    
    enhancement 
    opened by ipspace 1
Releases(release_1.1.3)
Owner
Ivan Pepelnjak
Ivan Pepelnjak
FTX auto lending bot with python

FTX auto lending bot Get the API key Check my article for step by step + screenshots Setup & Run Install python 3 Install dependency pip install -r re

Patompong Manprasatkul 1 Dec 24, 2021
A lightweight, dependency-free Python library (and command-line utility) for downloading YouTube Videos.

24 July 2020 Actively soliciting contributers! Ping @ronncc if you would like to help out! pytube pytube is a very serious, lightweight, dependency-fr

pytube 7.9k Jan 02, 2023
Analyzed the data of VISA applicants to build a predictive model to facilitate the process of VISA approvals.

Analyzed the data of Visa applicants, built a predictive model to facilitate the process of visa approvals, and based on important factors that significantly influence the Visa status recommended a s

Jesus 1 Jan 08, 2022
LEC_Ditto is a bot that tracks the follows and unfollows of Twitter accounts

✨ LEC_Ditto ✨ I'm Ditto, and I'm a bot 🤖 . Getting Started | Installation | Usage Getting Started LEC_Ditto is a bot that tracks the follows and unfo

2 Mar 30, 2022
Visual Weather api. Returns beautiful pictures with the current weather.

VWapi Visual Weather api. Returns beautiful pictures with the current weather. Installation: sudo apt update -y && sudo apt upgrade -y sudo apt instal

Hotaru 33 Nov 13, 2022
WhatsApp Web API client with multi-device support

Tauros WhatsApp Web client for multi-device in python Free software: MIT Documentation: https://tauros.readthedocs.io Features TODO Credits This packa

Manjit Pardeshi 0 Jan 20, 2022
Lol qq parser - A League of Legends parser for QQ data

lol_qq_parser A League of Legends parser for QQ data Sources This package relies

Tolki 3 Jul 13, 2022
Yes, it's true :revolving_hearts: This repository has 301 stars.

Yes, it's true! Inspired by a similar repository from @RealPeha, but implemented using a webhook on AWS Lambda and API Gateway, so it's serv

511 Dec 30, 2022
Telegram Google Translater Bot

Google-Translater-Bot Hey Mo Tech, I am simple Google Translater Bot. I can translate any language to you selected language Team Mo Tech Deploy To Her

21 Dec 01, 2022
RChecker - Checker for minecraft servers

🔎 RChecker v1.0 Checker for Minecraft Servers 💻 Supported operating systems: ✅

Pedro Vega 1 Aug 30, 2022
An elegant mirai-api-http v2 Python SDK.

Ariadne 一个适用于 mirai-api-http v2 的 Python SDK。 本项目适用于 mirai-api-http 2.0 以上版本。 目前仍处于开发阶段,内部接口可能会有较大的变化。 安装 poetry add graia-ariadne 或 pip install graia

Graia Project 259 Jan 02, 2023
The world's first public V2ray manager Telegram bot

📌 DarkV2ray-Manager-Bot 0.1 UPDATE 11/11/2021 Telegram bot v2ray Test user expired date data limit paylode && sni usage user on/off heroku bot hostin

@Dk_king_offcial 1 Nov 11, 2021
Benachrichtigungs-Bot für das niedersächische Impfportal / Notification bot for the lower saxony vaccination portal

Ein kleines Wochenend-Projekt von mir. Der Bot überwacht die REST-API des niedersächsischen Impfportals auf freie Impfslots und sendet eine Benachrichtigung mit deinem bevorzugtem Service. Ab da gilt

sibalzer 37 May 11, 2022
A Python script to create customised Spotify playlists using the JSON, Spotipy Library and Spotify Web API, based on seed tracks in your history.

A Python script to create customised Spotify playlists using the JSON, Spotipy Library and Spotify Web API, based on seed tracks in your history.

Youngseo Park 1 Feb 01, 2022
A mass account list editor for python

Account-List-Editor This is an mass account list editor Usage Run the editor.py file with python (python3 ./editor.py) Press a button (1/2) and drag &

ExtremeDev 1 Dec 20, 2021
ResolveURL - Fork of UrlResolver by eldorados, tknorris and jsergio123

ResolveURL Fork of UrlResolver by eldorados, tknorris and jsergio123 I am in no

gujal 60 Jan 03, 2023
Archive tweets and make them searchable

Tweeter Archive and search your tweets and liked tweets using AWS Lambda, DynamoDB and Elasticsearch. Note: this project is primarily being used a tes

Kamil Sindi 8 Nov 18, 2022
Pincer-ext-commands - A simple, lightweight package for pincer prefixed commands

pincer.ext.commands A reimagining of pincer's command system and bot system. Ins

Vincent 2 Jan 11, 2022
Reverse engineering multi-device WhatsApp Web.

whatsapp-web-multi-device-reveng In this repository, the research for reverse engineering multi-device WhatsApp Web takes place, see here for a descri

84 Jan 01, 2023
Automatically send commands to send Twitch followers to any Twitch account.

Automatically send commands to send Twitch followers to any Twitch account. You just need to be in a Twitch follow bot Discord server!

Thomas Keig 6 Nov 27, 2022