Openvswitch create queue can't reach bit rate limitation - sdn

all:
I'm new to SDN and also new to stackoverflow.
Here is my topology:
When I setting the queue for mytopo, I notice that when the maximum bit lower than 300000, the iperf value become strange. Cant't reach the maximum bit rate setting.
let max-rate= ppp
setting order:
ovs-vsctl -- set Port s1-eth1 qos=#newqos -- \
--id=#newqos create QoS type=linux-htb other-config:max-rate=1000000 queues=0=#q0 -- \
--id=#q0 create Queue other-config:min-rate= ppp other-config:max-rate= ppp
Test ppp = 1000000,800000,600000,400000,300000,250000,200000.
Are there something wrong? or it's ovs's limitation?
ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.0.2
Compiled Dec 9 2015 14:08:11
OpenFlow versions 0x1:0x1
mininet python file:
#!/usr/bin/python
import re
import sys
from mininet.cli import CLI
from mininet.log import setLogLevel, info, error
from mininet.net import Mininet
from mininet.link import TCLink
from mininet.topolib import TreeTopo
from mininet.util import quietRun
from mininet.node import RemoteController
from mininet.topo import Topo
topos = { 'mytopo': ( lambda: MyTopo() ) }
class MyTopo( Topo ):
# "this topo is used for Scheme_1"
def __init__( self ):
"Create custom topo."
# Initialize topology
Topo.__init__( self )
# Add hosts
h1 = self.addHost( 'h1' , ip="192.168.254.11/24", mac="00:00:00:00:00:01", defaultRoute="via 10.0.0.254")
h2 = self.addHost( 'h2' , ip="192.168.254.12/24", mac="00:00:00:00:00:02", defaultRoute="via 10.0.0.254")
h3 = self.addHost( 'h3' , ip="192.168.254.13/24", mac="00:00:00:00:00:03", defaultRoute="via 10.0.0.254")
# Add switches
s1 = self.addSwitch( 's1' )
# Add links
self.addLink( s1, h1 )
self.addLink( s1, h2 )
self.addLink( s1, h3 )
if __name__ == '__main__':
setLogLevel( 'info' )
info( '*** Creating network\n' )
net = Mininet( topo=MyTopo(),controller=None, link=TCLink)
c0 = RemoteController( 'c0', ip='127.0.0.1', port=6653 )
net.addController(c0)
net.start()
CLI( net )
net.stop()

I miss take the queue usage:
in the doc mentions that :
The port s1-eth1 is the switch port linked to h3.
Running iperf with h3 server, h4 client:
h4 → h3 (client to server) throttled to 4Mbit/s
h3 → h4 (server to client) not throttled
I was miss-leading by the mininnet: iperf h1 h2.(although I don't know what does the number means).

Related

Getting "cannot unpack non-iterable NoneType object" error when trying to run GA optimization

I'm running through the ASE tutorial and am trying to run a GA optimization (https://wiki.fysik.dtu.dk/ase/tutorials/ga/ga_optimize.html). However, when I run the code, I get an unpacking error. I'm not too sure about how to approach the issue.
Code is as follows:
from random import random
from ase.io import write
from ase.optimize import BFGS
from ase.calculators.emt import EMT
from ase.ga.data import DataConnection
from ase.ga.population import Population
from ase.ga.standard_comparators import InteratomicDistanceComparator
from ase.ga.cutandsplicepairing import CutAndSplicePairing
from ase.ga.utilities import closest_distances_generator
from ase.ga.utilities import get_all_atom_types
from ase.ga.offspring_creator import OperationSelector
from ase.ga.standardmutations import MirrorMutation
from ase.ga.standardmutations import RattleMutation
from ase.ga.standardmutations import PermutationMutation
# Change the following three parameters to suit your needs
population_size = 20
mutation_probability = 0.3
n_to_test = 20
# Initialize the different components of the GA
da = DataConnection('gadb.db')
atom_numbers_to_optimize = da.get_atom_numbers_to_optimize()
n_to_optimize = len(atom_numbers_to_optimize)
slab = da.get_slab()
all_atom_types = get_all_atom_types(slab, atom_numbers_to_optimize)
blmin = closest_distances_generator(all_atom_types,
ratio_of_covalent_radii=0.7)
comp = InteratomicDistanceComparator(n_top=n_to_optimize,
pair_cor_cum_diff=0.015,
pair_cor_max=0.7,
dE=0.02,
mic=False)
pairing = CutAndSplicePairing(slab, n_to_optimize, blmin)
mutations = OperationSelector([1., 1., 1.],
[MirrorMutation(blmin, n_to_optimize),
RattleMutation(blmin, n_to_optimize),
PermutationMutation(n_to_optimize)])
# Relax all unrelaxed structures (e.g. the starting population)
while da.get_number_of_unrelaxed_candidates() > 0:
a = da.get_an_unrelaxed_candidate()
a.calc = EMT()
print('Relaxing starting candidate {0}'.format(a.info['confid']))
dyn = BFGS(a, trajectory=None, logfile=None)
dyn.run(fmax=0.05, steps=100)
a.info['key_value_pairs']['raw_score'] = -a.get_potential_energy()
da.add_relaxed_step(a)
# create the population
population = Population(data_connection=da,
population_size=population_size,
comparator=comp)
# test n_to_test new candidates
for i in range(n_to_test):
print('Now starting configuration number {0}'.format(i))
a1, a2 = population.get_two_candidates()
a3, desc = pairing.get_new_individual([a1, a2])
if a3 is None:
continue
da.add_unrelaxed_candidate(a3, description=desc)
# Check if we want to do a mutation
if random() < mutation_probability:
a3_mut, desc = mutations.get_new_individual([a3])
if a3_mut is not None:
da.add_unrelaxed_step(a3_mut, desc)
a3 = a3_mut
# Relax the new candidate
a3.calc = EMT()
dyn = BFGS(a3, trajectory=None, logfile=None)
dyn.run(fmax=0.05, steps=100)
a3.info['key_value_pairs']['raw_score'] = -a3.get_potential_energy()
da.add_relaxed_step(a3)
population.update()
write('all_candidates.traj', da.get_all_relaxed_candidates())
Really confused on what I should try to fix the issue. Can't return the a1 and a2 values either.

packet-in request after idle timeout in sdn switch using ryu controller

I'm working on the Ryu controller to set the idle and hard timeout for flows. I'm assigning the idle timeout to 10s and hard timeout to 30s. On the very first, when I run pingall on mininet this will install the flow rules by generating a packet miss-match request. When a timeout event occurs it will remove the flow rule from the flow table. Now when I again run pingall on mininet it is not generating a packet miss-match request. all the packets are dropped. Please help me with how I can fix this. The code for the Ryu app is given below.
# Copyright (C) 2011 Nippon Telegraph and Telephone Corporation.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from ryu.base import app_manager
from ryu.controller import ofp_event
from ryu.controller.handler import CONFIG_DISPATCHER, MAIN_DISPATCHER
from ryu.controller.handler import set_ev_cls
from ryu.ofproto import ofproto_v1_3
from ryu.lib.packet import packet
from ryu.lib.packet import ethernet
from ryu.lib.packet import ether_types
class SimpleSwitch13(app_manager.RyuApp):
OFP_VERSIONS = [ofproto_v1_3.OFP_VERSION]
def __init__(self, *args, **kwargs):
super(SimpleSwitch13, self).__init__(*args, **kwargs)
self.mac_to_port = {}
#set_ev_cls(ofp_event.EventOFPSwitchFeatures, CONFIG_DISPATCHER)
def switch_features_handler(self, ev):
datapath = ev.msg.datapath
ofproto = datapath.ofproto
parser = datapath.ofproto_parser
# install table-miss flow entry
#
# We specify NO BUFFER to max_len of the output action due to
# OVS bug. At this moment, if we specify a lesser number, e.g.,
# 128, OVS will send Packet-In with invalid buffer_id and
# truncated packet data. In that case, we cannot output packets
# correctly. The bug has been fixed in OVS v2.1.0.
match = parser.OFPMatch()
actions = [parser.OFPActionOutput(ofproto.OFPP_CONTROLLER,
ofproto.OFPCML_NO_BUFFER)]
self.add_flow(datapath, 0, match, actions)
def add_flow(self, datapath, priority, match, actions, buffer_id=None):
ofproto = datapath.ofproto
parser = datapath.ofproto_parser
inst = [parser.OFPInstructionActions(ofproto.OFPIT_APPLY_ACTIONS,
actions)]
if buffer_id:
mod = parser.OFPFlowMod(datapath=datapath, buffer_id=buffer_id,
idle_timeout=10, hard_timeout=30, priority=priority, match=match,
instructions=inst)
else:
mod = parser.OFPFlowMod(datapath=datapath, priority=priority,
idle_timeout=10, hard_timeout=30, match=match, instructions=inst)
datapath.send_msg(mod)
#set_ev_cls(ofp_event.EventOFPPacketIn, MAIN_DISPATCHER)
def _packet_in_handler(self, ev):
# If you hit this you might want to increase
# the "miss_send_length" of your switch
if ev.msg.msg_len < ev.msg.total_len:
self.logger.debug("packet truncated: only %s of %s bytes",
ev.msg.msg_len, ev.msg.total_len)
msg = ev.msg
datapath = msg.datapath
ofproto = datapath.ofproto
parser = datapath.ofproto_parser
in_port = msg.match['in_port']
pkt = packet.Packet(msg.data)
eth = pkt.get_protocols(ethernet.ethernet)[0]
if eth.ethertype == ether_types.ETH_TYPE_LLDP:
# ignore lldp packet
return
dst = eth.dst
src = eth.src
dpid = datapath.id
self.mac_to_port.setdefault(dpid, {})
self.logger.info("packet in %s %s %s %s", dpid, src, dst, in_port)
# learn a mac address to avoid FLOOD next time.
self.mac_to_port[dpid][src] = in_port
if dst in self.mac_to_port[dpid]:
out_port = self.mac_to_port[dpid][dst]
else:
out_port = ofproto.OFPP_FLOOD
actions = [parser.OFPActionOutput(out_port)]
# install a flow to avoid packet_in next time
if out_port != ofproto.OFPP_FLOOD:
match = parser.OFPMatch(in_port=in_port, eth_dst=dst, eth_src=src)
# verify if we have a valid buffer_id, if yes avoid to send both
# flow_mod & packet_out
if msg.buffer_id != ofproto.OFP_NO_BUFFER:
self.add_flow(datapath, 1, match, actions, msg.buffer_id)
return
else:
self.add_flow(datapath, 1, match, actions)
data = None
if msg.buffer_id == ofproto.OFP_NO_BUFFER:
data = msg.data
out = parser.OFPPacketOut(datapath=datapath, buffer_id=msg.buffer_id,
in_port=in_port, actions=actions, data=data)
datapath.send_msg(out)

how to make cross hair mouse tracker on a PlotWidget() promoted in designer-qt5

I am trying to make a cross hair on my pyqtgraph interactive plots, which are embedded in a PyQt5 GUI thanks to the designer-qt5. I found a working
code in the pyqtgraph "examples". A simplified WORKING example is posted below. Now I want the same, but the problem seems to be that I promoted a
QGraphicsView() to a pg.PlotWidget in the designer, instead of pg.GraphicsWindow()? The Code does not work for me because my p1 is "pyqtgraph.widgets.PlotWidget.PlotWidget object" while in the example p1 is
"pyqtgraph.graphicsItems.PlotItem.PlotItem.PlotItem object".
So what should I do to make this example work for me?
import numpy as np
import pyqtgraph as pg
from pyqtgraph.Qt import QtGui, QtCore
from pyqtgraph.Point import Point
pg.setConfigOption('background', '#ffffff')
pg.setConfigOption('foreground', 'k')
pg.setConfigOptions(antialias=True)
app = QtGui.QApplication([])
win = pg.GraphicsWindow()
win.setWindowTitle('pyqtgraph example: crosshair')
label = pg.LabelItem(justify='right')
win.addItem(label)
p1 = win.addPlot(row=1, col=0)
p1.setAutoVisible(y=True)
#create numpy arrays
#make the numbers large to show that the xrange shows data from 10000 to all the way 0
data1 = 10000 + 15000 * pg.gaussianFilter(np.random.random(size=10000), 10) + 3000 * np.random.random(size=10000)
p1.plot(data1, pen="r")
#cross hair
vLine = pg.InfiniteLine(angle=90, movable=False)
hLine = pg.InfiniteLine(angle=0, movable=False)
p1.addItem(vLine, ignoreBounds=True)
p1.addItem(hLine, ignoreBounds=True)
vb = p1.vb
print(p1)
print(vb)
def mouseMoved(evt):
pos = evt[0] ## using signal proxy turns original arguments into a tuple
if p1.sceneBoundingRect().contains(pos):
mousePoint = vb.mapSceneToView(pos)
index = int(mousePoint.x())
if index > 0 and index < len(data1):
label.setText("<span style='font-size: 12pt'>x=%0.1f, <span style='color: green'>y2=%0.1f</span>" % (mousePoint.x(), data1[index]))
vLine.setPos(mousePoint.x())
hLine.setPos(mousePoint.y())
proxy = pg.SignalProxy(p1.scene().sigMouseMoved, rateLimit=60, slot=mouseMoved)
#p1.scene().sigMouseMoved.connect(mouseMoved)
## Start Qt event loop unless running in interactive mode or using pyside.
if __name__ == '__main__':
import sys
if (sys.flags.interactive != 1) or not hasattr(QtCore, 'PYQT_VERSION'):
QtGui.QApplication.instance().exec_()
I am very sorry for the noise!!! I fix it myself!
The important part was:
plot_wg.proxy = proxy
Very simple...
Below is the function, which should work for any PlotWidget:
def cross_hair(self, plot_wg, log=False ):
global fit
################### TETS cross hair ############3
vLine = pg.InfiniteLine(angle=90, movable=False)#, pos=0)
hLine = pg.InfiniteLine(angle=0, movable=False)#, pos=2450000)
plot_wg.addItem(vLine, ignoreBounds=True)
plot_wg.addItem(hLine, ignoreBounds=True)
vb = plot_wg.getViewBox()
label = pg.TextItem()
plot_wg.addItem(label)
def mouseMoved(evt):
pos = evt[0] ## using signal proxy turns original arguments into a tuple
if plot_wg.sceneBoundingRect().contains(pos):
mousePoint = vb.mapSceneToView(pos)
if log == True:
label.setText("x=%0.3f, y1=%0.3f"%(10**mousePoint.x(), mousePoint.y()))
else:
label.setText("x=%0.3f, y1=%0.3f"%(mousePoint.x(), mousePoint.y()))
vLine.setPos(mousePoint.x())
hLine.setPos(mousePoint.y())
#print(mousePoint.x(),mousePoint.y())
plot_wg.getViewBox().setAutoVisible(y=True)
proxy = pg.SignalProxy(plot_wg.scene().sigMouseMoved, rateLimit=60, slot=mouseMoved)
plot_wg.proxy = proxy
proxy = pg.SignalProxy(plot_wg.scene().sigMouseMoved, rateLimit=60, slot=mouseMoved)
plot_wg.proxy = proxy
################### TETS cross hair ############3

Apache Beam job (Python) using Tensorflow Transform is killed by Cloud Dataflow

I'm trying to run an Apache Beam job based on Tensorflow Transform on Dataflow but its killed. Someone has experienced that behaviour? This is a simple example with DirectRunner, that runs ok on my local but fails on Dataflow (I change the runner properly):
import os
import csv
import datetime
import numpy as np
import tensorflow as tf
import tensorflow_transform as tft
from apache_beam.io import textio
from apache_beam.io import tfrecordio
from tensorflow_transform.beam import impl as beam_impl
from tensorflow_transform.beam import tft_beam_io
from tensorflow_transform.tf_metadata import dataset_metadata
from tensorflow_transform.tf_metadata import dataset_schema
import apache_beam as beam
NUMERIC_FEATURE_KEYS = ['feature_'+str(i) for i in range(2000)]
def _create_raw_metadata():
column_schemas = {}
for key in NUMERIC_FEATURE_KEYS:
column_schemas[key] = dataset_schema.ColumnSchema(tf.float32, [], dataset_schema.FixedColumnRepresentation())
raw_data_metadata = dataset_metadata.DatasetMetadata(dataset_schema.Schema(column_schemas))
return raw_data_metadata
def preprocessing_fn(inputs):
outputs={}
for key in NUMERIC_FEATURE_KEYS:
outputs[key] = tft.scale_to_0_1(inputs[key])
return outputs
def main():
output_dir = '/tmp/tmp-folder-{}'.format(datetime.datetime.now().strftime('%Y%m%d%H%M%S'))
RUNNER = 'DirectRunner'
with beam.Pipeline(RUNNER) as p:
with beam_impl.Context(temp_dir=output_dir):
raw_data_metadata = _create_raw_metadata()
_ = (raw_data_metadata | 'WriteInputMetadata' >> tft_beam_io.WriteMetadata(os.path.join(output_dir, 'rawdata_metadata'), pipeline=p))
m = numpy_dataset = np.random.rand(100,2000)*100
raw_data = (p
| 'CreateTestDataset' >> beam.Create([dict(zip(NUMERIC_FEATURE_KEYS, m[i,:])) for i in range(m.shape[0])]))
raw_dataset = (raw_data, raw_data_metadata)
transform_fn = (raw_dataset | 'Analyze' >> beam_impl.AnalyzeDataset(preprocessing_fn))
_ = (transform_fn | 'WriteTransformFn' >> tft_beam_io.WriteTransformFn(output_dir))
(transformed_data, transformed_metadata) = ((raw_dataset, transform_fn) | 'Transform' >> beam_impl.TransformDataset())
transformed_data_coder = tft.coders.ExampleProtoCoder(transformed_metadata.schema)
_ = transformed_data | 'WriteTrainData' >> tfrecordio.WriteToTFRecord(os.path.join(output_dir, 'train'), file_name_suffix='.gz', coder=transformed_data_coder)
if __name__ == '__main__':
main()
Also, my production code (not shown) fail with the message: The job graph is too large. Please try again with a smaller job graph, or split your job into two or more smaller jobs.
Any hint?
The restriction on the pipeline description size is documented here:
https://cloud.google.com/dataflow/quotas#limits
There is a way around that, instead of creating stages for each tensor that goes into tft.scale_to_0_1 we could fuse them by first stacking them together, and then passing them into tft.scale_to_0_1 with 'elementwise=True'.
The result will be the same, because the min and max are computed per 'column' instead of across the whole tensor.
This would look something like this:
stacked = tf.stack([inputs[key] for key in NUMERIC_FEATURE_KEYS], axis=1)
scaled_stacked = tft.scale_to_0_1(stacked, elementwise=True)
for key, tensor in zip(NUMERIC_FEATURE_KEYS, tf.unstack(scaled_stacked, axis=1)):
outputs[key] = tensor

Finding PID's of Virtual Machine in openstack

I am working on openstack and I want to monitor the Virtual Machines cpu usage. For that I want to find their PIDs through the parent (central) openstack instance.
I used
ps aux | grep
and I did receive an output. I however want to confirm if this is correct PID. Is their any way I can check this?
Or is their any other way to find the PID's of the virtual machine?
Update.
This command does not work . It gives me a PID which always change. Its not constant.
Thank you
Well libvirt has some interfaces for this. Here's some python that extracts that data into datastructures for you:
#!/usr/bin/env python
# Modules
import subprocess
import traceback
import commands
import signal
import time
import sys
import re
import os
import getopt
import pprint
try:
import libvirt
except:
print "no libvirt detected"
sys.exit(0)
from xml.dom.minidom import parseString
global instances
global virt_conn
global tick
global virt_exist
def virtstats():
global virt_exist
global virt_conn
global instances
cpu_stats = []
if virt_exist == True:
if virt_conn == None:
print 'Failed to open connection to the hypervisor'
virt_exist = False
if virt_exist == True:
virt_info = virt_conn.getInfo()
for x in range(0, virt_info[2]):
cpu_stats.append(virt_conn.getCPUStats(x,0))
virt_capabilities = virt_conn.getCapabilities()
domcpustats = 0
# domcpustats = virDomain::GetcpuSTATS()
totmem = 0
totvcpu = 0
totcount = 0
vcpu_stats = []
for id in virt_conn.listDomainsID():
dom = virt_conn.lookupByID(id)
totvcpu += dom.maxVcpus()
vcpu_stats.append(dom.vcpus())
totmem += dom.maxMemory()
totcount += 1
dom = parseString(virt_capabilities)
xmlTag = dom.getElementsByTagName('model')[0].toxml()
xmlData=xmlTag.replace('<model>','').replace('</model>','')
for info in virt_info:
print info
for stat in cpu_stats:
print "cpu %s" % stat
for vstat in vcpu_stats:
print "vcpu:\n"
pprint.pprint(vstat)
print "CPU ( %s ) Use - %s vCPUS ( %s logical processors )" % (xmlData, totvcpu, virt_info[2])
sys.exit(0)
def main():
try:
global virt_conn
global virt_exist
virt_conn = libvirt.openReadOnly(None)
virt_exist = True
except:
virt_exist = False
print "OK: not a compute node"
sys.exit(0)
virtstats()
if __name__ == "__main__":
main()
Now what you get from this in terms of usage is cpu time.
The vcpu blocks are basically in this layout:
1st: vCPU number, starting from 0.
2nd: vCPU state.
0: offline
1: running
2: blocked on resource
3rd: CPU time used in nanoseconds
4th: real CPU number
The CPU blocks are obvious once you realize that's what's goin down in libvirt.
Hope that helps!
By using libvirt, python, lxml, and lsof you can recover the pid if your Virtual Instance (domain) has a display output set. (VNC, Spice, ...)
Retrieve display port
Retrieve pid from opened display port
Here is the code:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from lxml import etree
import libvirt
from subprocess import check_output
def get_port_from_XML(xml_desc):
tree = etree.fromstring(xml_desc)
ports = tree.xpath('//graphics/#port')
if len(ports):
return ports[0]
return None
def get_pid_from_listen_port(port):
if port is None:
return ''
return check_output(['lsof', '-i:%s' % port, '-t']).replace('\n','')
conn = libvirt.openReadOnly('')
if conn is None:
print 'Failed to open connection to the hypervisor'
sys.exit(1)
for domain_id in conn.listDomainsID():
domain_instance = conn.lookupByID(domain_id)
name = domain_instance.name()
xml_desc = domain_instance.XMLDesc(0)
port = get_port_from_XML(xml_desc)
pid = get_pid_from_listen_port(port)
print '%s (port:%s) (pid:%s)' % (name, port, pid)
grep "79d87652-8c8e-4afa-8c13-32fbcbf98e76" --include=libvirt.xml /path/to/nova/instances -r -A 2 | grep "<name" | cut -d " " -f 3
allows to find "instance-" which can be mapped to ps aux output of "-name" parameter. so you can map openstack instance id to pid.
The most simple way is using cgroups:
In Ubuntu:
cat /sys/fs/cgroup/cpuset/libvirt/qemu/<machine-name>/tasks