Anything Linux related

Getting My Real VM Server Back Online Part III: Storage, iSCSI, and Live Migrations

After some dubious network configurations (that I should have never configured incorrectly) I finally got multipath working to the main storage server. All of the multipath.conf examples I saw resulted in non-functional iSCSI MPIO, while having no multipath.conf left me with failover MPIO instead of interleaved/round-robin.

A large issue with trying to get MPIO configured was the fact that all the examples I found were either old (and scsi_id works slightly differently in Ubuntu 14.04) or just poor. Yes, I wound up using Ubuntu. Usually I use Slackware for EVERYTHING, but lately I’ve been trying to branch out. Most of the VMs run Fedora, “Pegasus” or VMSrv1 uses Fedora, “Titan” uses Ubuntu.

Before I did anything with multipath.conf (It’s empty on Ubuntu 14.04), I got this:

root@titan:/home/frankd# multipath -ll
size=256G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 13:0:0:0 sde 8:64 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 12:0:0:0 sdd 8:48 active ready running

Note the disks are both round-robin — with only one member each! This works for fail-over, but did nothing for performance. The only thing that wound up working for multipath.conf was this:

defaults {
 user_friendly_names yes
 polling_interval 3
 path_grouping_policy multibus
 path_checker readsector0
 path_selector "round-robin 0"
 features "0"
 no_path_retry 1
 rr_min_io 100

multipaths {
 multipath {
  wwid 1FREEBSD_HTPC1-D1
  alias testLun

The wwid/alias doesn’t work, however. All of the MPIO is just coming from the defaults stanza. I attempted many things with no luck, unfortunately. I’m going to have to delve into this more especially if I want live migrations to work properly with MPIO. As it stands the disk devices are pointing at a single IP (ex /dev/disk/by-path/ip-, I’ll need to point at aliases to get the VMs working with multipath.

The multipath tests themselves were promising though, dd was able to give me a whopping 230MB/s to the mapper device over a pair of GigE connections.

The output from ‘multipath -ll’ now looked more reasonable:

root@titan:/home/frankd# multipath -ll
size=256G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 39:0:0:0 sde 8:64 active ready running
  `- 40:0:0:0 sdg 8:96 active ready running

You can see the drives are both under the same round-robin policy instead of two separate ones.

The storage server also saw some slight changes, including upgrading from one Intel X25-V 40GB for L2ARC to 2xX25-Vs for a total of 80GB. I also added a 60GB Vertex 2 as a SZIL device. I really need to build a machine with more RAM and partition out the SZIL. I’ll likely wind up using my 840Pro 256GB for L2ARC and leave the old X25Vs out of the main array once I get a pair of 10GbE cards for maximum speed (hopefully near-native of the 840Pro — perhaps better with a large amount of ARC) to my workstation.

So we’re at a point where everything appears to be working, although in need of some upgrades! Great! I’m looking at a KCMA-D8 Dual Opteron C32 motherboard as I have a pair of Opteron 4184s (6 core Lisbon, very similar to a Phenom II X6 1055T) laying around, so I could put together a 32GB 12 core machine for under $400 — but as always, budgetary constraints for a hobby squash that idea quickly.

PHP, Screen Scraping & SevOne Deferred Data — or a Network Operator/Engineer That Can Do More Than Networks! (Possible Rant)

This post is semi work related, but I do have SevOne running at home as my NMS for graphing, trending and alerting. There are some statistics that I insert via their ‘deferred data,’ which is done through a (fairly horrible) SOAP API. Before Cablevision I hadn’t written a line of PHP, or used SOAP in any language — but all of the SevOne examples were written in PHP. I picked it up to write some scripts that insert data from CMTS (Cable Modem Termination Systems) that were not available  through SNMP which was really helpful in our monitoring and alerting.  As I do have some kind of programming background it wasn’t difficult to pickup the ‘beginners,’ scripting language. Eventually I wrote a class to encapsulate a bunch of stuff that I do to interface with SevOne as we could really spike the CPU usage if we did things by way of their example scripts.

The entire CMTS script involved me writing a telnet wrapper that uses socket calls and is entirely written in PHP. This was probably a naive approach and a stream to telnet would’ve been much better (as we will be moving to SSH anyway), although it did allow me to flex some critical thought muscles I forgot I had and haven’t been used in some time. After some tuning it’s actually pretty fast and doesn’t present a bottleneck or resource drain. Eventually even the telnet functions got expanded on considerably into a couple of classes for accessing data on IOS and Arris C4. The IOS functions got expanded on to parse data for some critical things for CMTS-specific stuff, then for other pieces of equipment (ie 7600/6500). The base telnet class was later extended to IOS-XR for ASR9K devices. Then NX-OS. A flexible SNMP wrapper was created primarily for polling STB (Set Top Boxes). I got pretty good at writing PHP in a time-efficient manner to parse all kinds of stuff. As a “network operator,” this was great! I could get all kinds of statistics that would normally be “impossible,” or really “time-intensive. The SevOne stuff expanded and got refined as I dug more into my programming background.  Even as I was using a fairly clumsy language without formal training I had a good idea of how to write EFFECTIVE PHP (in my mind anyway). Parts of it might be ugly, but it gets the job done. I became adept at writing regular expressions, I even dynamically generated it based on the output so it would be extremely easy to update in the future and could even deal with a good deal of changes without being updated. I wrote SevOne wrappers to generate (or update) hundreds of threshold alarms at a time that would otherwise take a LOT of man hours to manually input. I wrote scripts to poll SevOne for information so we could figure out how to most effectively utilize our servers, or just to look at devices for certain criteria.. things that we wouldn’t be able to do an effective manner otherwise.

That’s a lot of background, but the end result: I created a great SevOne Deferred Data wrapper that caches a lot of information to cut back on CPU hits to the SevOne servers. It connects to peers dynamically as required to cut back on CPU. It takes a lot of work away from SevOne that might otherwise be (naively) processed on SevOne’s side. I know because I did it. And we pegged CPU cores on our SevOne boxes. We ran into so many issues. Because of that we were unable to do what we needed to do without some of these (admittedly somewhat minor) innovations. For Cablevision it’s amazing. For me, it allows a great amount of work in small amounts of time. All of the SevOne nitty-gritty is abstracted away and done automagically. Things are created on the fly, SOAP calls that might generate an exception (and this happens often) are automatically retried depending on the error. There’s little need for error checking in a script that screen scrapes to collect data. In my mind that’s an example of a great piece of code, it doesn’t make you think about what you’re doing with something. It does what you expect of it (for the most part). When upper management asks for something I look good because I can get it to them extremely quickly. When SevOne says it will take months for their Professional Services division to create something or cost some obscene amount of money, I’m there offering a quick solution which is cheap for them. Maybe that’s not great, but I like doing my job well.

I’m not the best programmer ever. I’m not a scripting pro. I don’t know that many scripting languages, but I can pick them up fairly quickly. I’ve never touched Python. I can whip things up in Ruby (ON RAILS EVEN!) but I’m not a Ruby guru. I can write shell scripts but I’m not extremely proficient in BASH, nevermind TCSH, KSH or any of the others. I’m not really a network engineer. But combining my general purpose background has allowed me to do things that might not otherwise be possible. Certainly your regular CCNP or CCIE carrying network guy is not going to have quite the scripting background to get the data required into an NMS — and your regular programmer is not going to have enough of an idea about network devices or the network in general to have a good understanding of what might need to be collected into the NMS.  At home I want to monitor some things on my VM server that aren’t available through SNMP. I could write scripts and custom OIDs to access them through SNMP but that requires configuration on the SevOne side. It’s much faster for me to use my wrapper to insert the data I need (a good example — CPU wattage).

I realize this post is more of a rant than usual, but feel free to check the wrappers out at SevOne PHP Wrappers



VMs, Linux Software Bridges and 802.1q — What I Learned This Time

When initially setting up the box, I had the idea in my head that I might create several bridges. One for each VLAN. That’s probably one of the best ways to tackle the issue unless you really want the trunk to exist on the VM –which is also fine and valid. But by default it gives every device the option of accessing any VLAN in the trunk, which since we’re in a lab environment is not particularly an issue.

But I like to work reality into lab mockups as much as possible. I have plenty of NIC ports, so even creating a lot of trunks is not an issue. But our VMs will accept a large number of virtual NICs, so this option seemed semi-elegant.

The first issue I ran into was crosstalk between the VLANs, I had created a bunch of 802.1q sub-interfaces (which strip/tag incoming and outgoing frames) via ‘vconfig’ or ‘ip link’. I attached p32p1.10 to br10 and p32p1.1 to br1. I attached tap0 to br10 and tap1 to br1. Everything appeared to be working on the very initial configuration until I saw the output of ‘sh cdp nei’ on the physical Cisco 3550. It saw itself. That meant it was receiving bounceback. So I loaded up tcpdump and watched bridge traffic and examined the macs in the Linux software bridge. There was definitely cross talk — and after a hunch and a little investigation it turns out that QEMU doesn’t do much to separate NIC traffic as I called them with the ‘old’ syntax. After updating my QEMU launch options the problem disappeared and I was happy… until…

Neither OSPF or EIGRP were forming neighborships. Load up tcpdump again, examine traffic. I see the packets hitting br1 and br10 from XRv, and from both the 3550 and the 2821 that I’m currently configuring. That looks good.. but ‘debug ospf packet’ on XRv was not giving me anything aside from what it was sending out. So I aimed tcpdump at the tap interfaces instead, and I saw that the tap interface was not receiving the HELLO packets on either VLAN. (Hint: Here is where I went wrong in my diagnostic chase, I had filtered tcpdump down to only EIGRP/OSPF, had I not the problem would’ve been almost immediately evident)

Thinking for some reason with no basis in fact that it may be a multicast issue with Linux software bridges, I decided to configure neighbors manually. That also resulted in a neighborship not forming with XRv to any other box. Other traffic (ICMP/TCP/UDP) was unaffected, so I thought that was interesting. I started watching the interface again, this time with no filter — and I saw the VM host replying to XRv with an ICMP Unreachable. Pretty clearly a firewall rule problem. Ebtables (iptables for layer 2 stuff if you haven’t seen it) was clear, and I didn’t see anything immediately in iptables.. but it’s always faster to test fixes than to examine things (and in a lab, perfectly acceptable), so a simple iptables -I FORWARD -j ACCEPT while removing the manual neighborships so EIGRP and OSPF would both go back to using multicast resulted in everything working viagra aus holland bestellen. Great! Classic implicit deny caught me.

This is where I usually get annoyed by pre-configured rules. Usually I load up Slackware, but I’ve been using Fedora lately for ease of getting some things up and running with real dependency management. Had I been using Slackware with its default no rules everything would’ve been honky dory, and I would’ve configured some myself when I felt the time was right.

To continue — I tried to make a new bridge, trunk0, with p32p2 in it. I load up tcpdump and notice that there’s no traffic aside from STP on some VLANs that aren’t in active use. Apparently configuring those subinterfaces whisks the frames away from the main interface, so I just deleted all of them off of p32p1, configured another trunk port and added that to trunk0 on the Linux box and voila! Tagged packets from everywhere! I have yet to try a tap interface into trunk0 and a VM, but I have a feeling everything should be all right. Then again, every time I have that feeling is usually when things are about to go terribly wrong.

© 2017 Musings

Theme by Anders NorenUp ↑