...

By default, the SOS controller’s REST API runs at the controller’s IP on TCP port 8080. An HTTP client can be used to leverage the API. The popular utility curl is used throughout this document as an example HTTP client. For more information on curl, please refer to curl’s documentation. The following is a general example on how to use curl:

...

Typical Workflow

In order to use SOS, the network must first be configured. This is an assumed prerequisite to this document.

After the network is configured, the agents should then be added to the controller, information about the transfer to conduct SOS on should be whitelisted in the controller, any SOS parameters should be tuned as desired, and lastly the controller should be checked to ensure the network is ready to handle pending transfer. At this point, the transfer can be initiated.

In the following steps, the controller is assumed to be running at the IP address 192.168.1.1 with its REST API exposed on port 8080.

Add the SOS Agents

Once the network is configured, the SOS controller needs to know about the agents it has at its disposal. One or more agents can be added, although note that SOS must have at least two configured agents in order to function. The following is an example of adding an agent:

curl http://192.168.1.1:8080/wm/sos/agent/add/json -X POST -d '{"ip-address":"10.0.0.1", "control-port":"9998", "data-port":"9877", "feedback-port":"9997"}' | python -m json.tool

which returns:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   189    0    97  100    92   1308   1240 --:--:-- --:--:-- --:--:--  1310
{
    "code": "0",
    "message": "Agent successfully added. It will be available for the next SOS session."
}

The Python module json.tool is used to format the JSON output in human-readable form. This is not strictly required, although it sure makes the JSON string look pretty 

Once an agent is added to the controller, there is no need to add it again in subsequent transfers.

Add a Whitelist Entry

To keep track of the transfers which should have SOS performed, the controller maintains a whitelist, where all transfers listed will use SOS, while all transfers absent will be handled outside of SOS. The default behavior of the controller is to emulate a L2 learning switch for all transfers that are not proactively whitelisted.

The following is an example of adding a whitelist entry:

...

...

Typical Workflow

In order to use SOS, the network must first be configured. This is an assumed prerequisite to this document.

After the network is configured, the agents should then be added to the controller, information about the transfer to conduct SOS on should be whitelisted in the controller, any SOS parameters should be tuned as desired, and lastly the controller should be checked to ensure the network is ready to handle the pending transfer. At this point, the transfer can be initiated.

In the following steps, the controller is assumed to be running at the IP address 192.168.1.1 with its REST API exposed on port 8080.

Add the SOS Agents

Once the network is configured, the SOS controller needs to know about the agents it has at its disposal. One or more agents can be added, although note that SOS must have at least two configured agents in order to function. The following is an example of adding an agent:

curl http://192.168.1.1:8080/wm/sos/agent/add/json -X POST -d '{"ip-address":"10.0.0.1", "control-port":"9998", "data-port":"9877", "feedback-port":"9997"}' | python -m json.tool

which returns:

   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   191189    0   101 97  100    92  90 1308 16248  144781240 --:--:-- --:--:-- --:--:--  168331310
{
    "code": "0",
    "message": "WhitelistEntryAgent successfully added. TheIt entrywill be mayavailable initiatefor the next dataSOS transfersession."
}

Once a transfer is whitelisted, it will persist until explicitly removed. The start and stop values are currently ignored (if supplied).

Tune Transfer Parameters

SOS's performance can be tuned by adjusting the number of parallel connections to use between the agents, the agent application's TCP receive buffer size (in bytes) per connection, and the reordering queue length on the receiving agent. The OpenFlow flow timeouts can also be adjusted, but it is recommended that they be left at their defaults of 0 for the hard timeout and 60 seconds for the idle timeout.

Any changes to these parameters will not impact ongoing transfers and will only apply to future transfers. Any settings will remain in effect until changed at a later time.

To adjust the number of parallel connections to use between the agents, one can do the followingThe Python module json.tool is used to format the JSON output in human-readable form. This is not strictly required, although it sure makes the JSON string look pretty 

Once an agent is added to the controller, there is no need to add it again in subsequent transfers.

Add a Whitelist Entry

To keep track of the transfers which should have SOS performed, the controller maintains a whitelist, where all transfers listed will use SOS, while all transfers absent will be handled outside of SOS. The default behavior of the controller is to emulate a L2 learning switch for all transfers that are not proactively whitelisted.

The following is an example of adding a whitelist entry:

curl http://192.168.1.1:8080/wm/sos/whitelist/configadd/json -X POST -d '{"parallelserver-ip-connectionsaddress":"4096"10.0.0.4", "server-tcp-port":"5001", "client-ip-address":"10.0.0.2"}' | python -m json.tool

which returns:

   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    88191    0    57101  100    3190  16248 9961   541714478 --:--:-- --:--:-- --:--:-- 1140016833
{
    "code": "0",
    "message": "Parallel connections set to 4096WhitelistEntry successfully added. The entry may initiate the data transfer."
}

It is recommended that the number of parallel connections range from 1 to 8,000. The agents will not reliably handle more than 8,000 given RAM utilization. As a general rule, the more parallel connections that are used, the better the performance will be. However, there comes a point where the CPU struggles to keep up with reading from a too-large number of these sockets.

Likewise, to adjust the TCP receive buffer size in the agent applicationOnce a transfer is whitelisted, it will persist until explicitly removed. The start and stop values are currently ignored (if supplied).

Tune Transfer Parameters

SOS's performance can be tuned by adjusting the number of parallel connections to use between the agents, the agent application's TCP receive buffer size (in bytes) per connection, and the reordering queue length on the receiving agent. The OpenFlow flow timeouts can also be adjusted, but it is recommended that they be left at their defaults of 0 for the hard timeout and 60 seconds for the idle timeout.

Any changes to these parameters will not impact ongoing transfers and will only apply to future transfers. Any settings will remain in effect until changed at a later time.

To adjust the number of parallel connections to use between the agents, one can do the following:

curl http://192.168.1.1:8080/wm/sos/config/json -X POST -d '{"bufferparallel-sizeconnections":"700004096"}' | python -m json.tool

...

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    7288    0    4957  100    2331   81059961   38045417 --:--:-- --:--:-- --:--:--  980011400
{
    "code": "0",
    "message": "BufferParallel sizeconnections set to 700004096"
}

It is recommended that the buffer size be at least 30,000 bytes and go no higher than 500,000 bytes. Values above 500,000 have not shown improvements in data transfer speed, while values under 30,000 bytes tend to hinder performance. The "sweet spot" at this point has been determined to be around 60,000-70,000 bytes given a large ~4,000 number of parallel connections.And lastly, to adjust the agent application's reordering queue lengthnumber of parallel connections range from 1 to 8,000. The agents will not reliably handle more than 8,000 given RAM utilization. As a general rule, the more parallel connections that are used, the better the performance will be. However, there comes a point where the CPU struggles to keep up with reading from a too-large number of these sockets.

Likewise, to adjust the TCP receive buffer size in the agent application, one can do the following:

curl http://192.168.1.1:8080/wm/sos/config/json -X POST -d '{"queuebuffer-capacitysize":"570000"}' | python -m json.tool

...

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    7072    0    4849  100    2223   57108105   26173804 --:--:-- --:--:-- --:--:--  60009800
{
    "code": "0",
    "message": "QueueBuffer capacitysize set to 570000"
}

The queue capacity is merely a safety feature that will only take effect if there is packet loss on a given parallel connection or if the agent CPU cannot keep up with the arrival rate of packets. Better performance with high queue capacity values indicates network instability, which implies that perhaps the buffer size and number of parallel connections parameters are either too large or too small.

If you would like to experiment with idle timeouts, you may adjust the idle timeout as followsIt is recommended that the buffer size be at least 30,000 bytes and go no higher than 500,000 bytes. Values above 500,000 have not shown improvements in data transfer speed, while values under 30,000 bytes tend to hinder performance. The "sweet spot" at this point has been determined to be around 60,000-70,000 bytes given a large ~4,000 number of parallel connections.

And lastly, to adjust the agent application's reordering queue length, one can do the following:

curl http://192.168.1.1:8080/wm/sos/config/json -X POST -d '{"idlequeue-timeoutcapacity":"605"}' | python -m json.tool

...

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    6870    0    4748  100    2122   97125710   43392617 --:--:-- --:--:-- --:--:-- 11750 6000
{
    "code": "0",
    "message": "IdleQueue timeoutcapacity set to 605"
}

The idle timeout is by default set to 60 seconds. This means that the last packet of the transfer, on a per-flow basis, will cause the flows to expire and be automatically removed from the switches 60s later. This is a long time, but it is safe in that it allows the agents to clear their buffers and finish transferring data, which might be terminated prematurely given a combination of shorter timeouts and poor parallel connection and buffer size parameter choices. Note that the "feedback-port" key used when adding an agent (above) is going to be used to allow the agents to send explicit notification to the controller when they have completed a particular transfer; this is about 95% complete at this time. It will eliminate the need to estimate idle timeouts.

Changing the hard timeout is supported, but it does not make sense at present (thus I will not include an example  ). Having hard timeouts could serve as a way to kick out transfers that have used more than their allotted share/time, but this is not a feature that is supported. As such, it is recommended that the hard timeout be left at 0 seconds (infinite).

Check Controller Readiness

Before a transfer is to be initiated, the controller should be queried to ensure all systems are ready to react to the transfer and perform SOS. This works well in a single-user environment where the user is performing sequential transfers. As such, it is the model we use at this point; however, a solution is nearing completion that allows the controller to pre-allocate resources for a user during a specific time period (which is more real-world) – these are the unused start and end times indicated in the whitelist REST API above.

To probe the controller to see if it is ready, one can use the status API as follows:

curl http://192.168.1.1:8080/wm/sos/status/json -X GET | python -m json.tool

which returns:

...

...

queue capacity is merely a safety feature that will only take effect if there is packet loss on a given parallel connection or if the agent CPU cannot keep up with the arrival rate of packets. Better performance with high queue capacity values indicates network instability, which implies that perhaps the buffer size and number of parallel connections parameters are either too large or too small.

If you would like to experiment with idle timeouts, you may adjust the idle timeout as follows:

curl http://192.168.1.1:8080/wm/sos/config/json -X POST -d '{"idle-timeout":"60"}' | python -m json.tool

which returns:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    68    0    47  100    21   9712   4339 --:--:-- --:--:-- --:--:-- 11750
{
    "code": "0",
    "message": "Idle timeout set to 60"
}

The idle timeout is by default set to 60 seconds. This means that the last packet of the transfer, on a per-flow basis, will cause the flows to expire and be automatically removed from the switches 60s later. This is a long time, but it is safe in that it allows the agents to clear their buffers and finish transferring data, which might be terminated prematurely given a combination of shorter timeouts and poor parallel connection and buffer size parameter choices. Note that the "feedback-port" key used when adding an agent (above) is going to be used to allow the agents to send explicit notification to the controller when they have completed a particular transfer; this is about 95% complete at this time. It will eliminate the need to estimate idle timeouts.

Changing the hard timeout is supported, but it does not make sense at present (thus I will not include an example  ). Having hard timeouts could serve as a way to kick out transfers that have used more than their allotted share/time, but this is not a feature that is supported. As such, it is recommended that the hard timeout be left at 0 seconds (infinite).

Check Controller Readiness

Before a transfer is to be initiated, the controller should be queried to ensure all systems are ready to react to the transfer and perform SOS. This works well in a single-user environment where the user is performing sequential transfers. As such, it is the model we use at this point; however, a solution is nearing completion that allows the controller to pre-allocate resources for a user during a specific time period (which is more real-world) – these are the unused start and end times indicated in the whitelist REST API above.

To probe the controller to see if it is ready, one can use the status API as follows:

curl http://192.168.1.1:8080/wm/sos/status/json -X GET | python -m json.tool

which returns:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    68    0    47  100    21   9712   4339 --:--:-- --:--:-- --:--:-- 11750
{
    "code": "0",
    "message": "Ready to accept a transfer"
}

if the controller is ready. The return will appear as follows if the user should wait before proceeding:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    68    0    47  100    21   9712   4339 --:--:-- --:--:-- --:--:-- 11750
{
    "code": "6",
    "message": "Not ready to accept a transfer"
}

Initiate Transfer

Once the agents have been added, the transfer has been whitelisted, and the controller indicates the SOS network is ready to handle the transfer, the user can initiate the transfer using any means on the client machine. The assumption is that the underlying transport protocol used is TCP and that the client IP address, server IP address, and server TCP port are those indicated in the whitelisted entry.

It is important to remember that SOS is a transparent service. From a user's perspective, the transfer should be initiated from the client to the server as a normal transfer would be conducted. The transfer should not in any way be directed to/from the agent IP addresses – this will happen automatically for you. In order for SOS to properly detect and perform this redirection, the transfer must be initiated by the client IP to the server IP at the server TCP port as indicated in the whitelist entry configured above.

Check Statistics

To get information about the transfer being performed, such as the agents chosen, the routes being used, etc. The statistics API can be used as follows:

curl http://192.168.1.1:8080/wm/sos/stats/json -X GET | python -m json.tool

which returns, as an example:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3283    0  3283    0     0   126k      0 --:--:-- --:--:-- --:--:--  128k
{
    "active-transfers": [
        {
            "buffer-size": "40000",
            "client": {
                "ip-address": "10.0.0.2",
                "mac-address": "00:60:dd:43:ac:79",
 Dload  Upload   Total   Spent    Left  Speed
100  "tcp-port": "51519"
  68    0    47  100},
   21   9712   4339 --:--:-- --:--:-- --:--:-- 11750
{ "client-side-agent": {
     "code": "0",     "message": "Ready to accept a transfer"active-transfers"
}

if the controller is ready. The return will appear as follows if the user should wait before proceeding:

  % Total: [
      % Received % Xferd  Average Speed   Time    Time  "9b3eb3ab-68f4-41ba-a102-d1498362eb63"
   Time  Current           ],
                "control-port": "9998",
    Dload  Upload   Total   Spent    Left  Speed
100"data-port": "9877",
     68    0    47  100 "feedback-port": "9997",
 21   9712   4339 --:--:-- --:--:-- --:--:-- 11750
{     "codeip-address": "610.0.0.1",
     "message": "Not ready to accept a transfer"
}

Initiate Transfer

Once the agents have been added, the transfer has been whitelisted, and the controller indicates the SOS network is ready to handle the transfer, the user can initiate the transfer using any means on the client machine. The assumption is that the underlying transport protocol used is TCP and that the client IP address, server IP address, and server TCP port are those indicated in the whitelisted entry.

Check Statistics

To get information about the transfer being performed, such as the agents chosen, the routes being used, etc. The statistics API can be used as follows:

curl http://192.168.1.1:8080/wm/sos/stats/json -X GET | python -m json.tool

which returns, as an example:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current           "mac-address": "00:60:dd:43:ac:72"
            },
            "flow-timeout": "10",
            "parallel-connections": "4096",
            "queue-capacity": "5",
            "route-agent-to-agent": {
                "destination-device": {
                    "active-transfers": [
                 Dload  Upload   Total   Spent"9b3eb3ab-68f4-41ba-a102-d1498362eb63"
    Left  Speed 100  3293    0  3293    0 ],
   0   100k      0 --:--:-- --:--:-- --:--:--  100k {     "activecontrol-transfersport": [
"9998",
       {             "bufferdata-sizeport": "400009877",
                    "clientfeedback-port": { "9997",
                    "ip-address": "10.0.0.23",
                    "mac-address": "0090:60e2:ddba:432b:acc9:7995",
                "tcp-port": "51518"},
             },   "route": [
        "client-side-agent": {
                "active-transfers": ["[id=00:00:00:60:dd:43:ac:72, port=local]",
                    "a84adbfb-0dec-4a62-b70f-fee1ad84cf1c"[id=00:00:00:60:dd:43:ac:72, port=1]",
                ],                 "control-port": "9998","[id=00:03:00:01:e8:d7:58:9b, port=25]",
                    "data-port": "9877",[id=00:03:00:01:e8:d7:58:9b, port=4]",
                    "feedback-port": "9997",[id=00:01:00:01:e8:8b:77:db, port=61]",
                    "ip-address": "10.0.0.1",[id=00:01:00:01:e8:8b:77:db, port=47]",
                    "mac-address": "00:60:dd:43:ac:72"
    [id=00:00:90:e2:ba:2b:c9:95, port=1]",
       },             "flow-timeout": "10",
            "parallel-connections": "4096",[id=00:00:90:e2:ba:2b:c9:95, port=local]"
               "queue-capacity": "5", ],
                "route-agent-to-agenttype": {"AGENT_2_AGENT",
                "destinationsource-device": {
                    "active-transfers": [
                        "a84adbfb9b3eb3ab-0dec68f4-4a6241ba-b70fa102-fee1ad84cf1cd1498362eb63"
                    ],
                    "control-port": "9998",
                    "data-port": "9877",
                    "feedback-port": "9997",
                    "ip-address": "10.0.0.31",
                    "mac-address": "9000:e260:badd:2b43:c9ac:95"72"
                }
              },},
            "route-client-to-agent": {
                "routedestination-device": [{
                    "[id=00:00:00:60:dd:43:ac:72, port=local]","active-transfers": [
                    "[id=00:00:00:60:dd:43:ac:72, port=1]",     "9b3eb3ab-68f4-41ba-a102-d1498362eb63"
               "[id=00:03:00:01:e8:d7:58:9b, port=25]"      ],
                    "[id=00:03:00:01:e8:d7:58:9b, port=4]"control-port": "9998",
                    "[id=00:01:00:01:e8:8b:77:db, port=61]"data-port": "9877",
                    "[id=00:01:00:01:e8:8b:77:db, port=47]feedback-port": "9997",
                    "[id=00:00:90:e2:ba:2b:c9:95, port=1]"ip-address": "10.0.0.1",
                    "[id=00:"mac-address": "00:9060:e2dd:ba43:2b:c9:95, port=local]ac:72"
                ]},
                "route-type": "AGENT_2_AGENT", [
                    "[id=00:03:00:01:e8:d7:58:9b, "source-device": {port=23]",
                    "active-transfers": [[id=00:03:00:01:e8:d7:58:9b, port=25]",
                        "a84adbfb-0dec-4a62-b70f-fee1ad84cf1c"
  "[id=00:00:00:60:dd:43:ac:72, port=1]",
                 ],        "[id=00:00:00:60:dd:43:ac:72, port=local]"
           "control-port": "9998",    ],
                "dataroute-porttype": "9877CLIENT_2_AGENT",
                    "feedbacksource-portdevice": "9997",{
                    "ip-address": "10.0.0.12",
                    "mac-address": "00:60:dd:43:ac:7279",
                    }"tcp-port": "51519"
                },
            "route-client-to-agent": "a84adbfb-0dec-4a62-b70f-fee1ad84cf1c" },
            "route-server-to-agent": {
                "destination-device": {
                    "active-transfers": [
                        "a84adbfb9b3eb3ab-0dec68f4-4a6241ba-b70fa102-fee1ad84cf1cd1498362eb63"
                    ],
                    "control-port": "9998",
                    "data-port": "9877",
                    "feedback-port": "9997",
                    "ip-address": "10.0.0.3",
                    "mac-address": "90:e2:ba:2b:c9:95"
                },
                "route": [
                    "[id=00:01:00:01:e8:8b:77:db, port=48]",
                    "[id=00:01:00:01:e8:8b:77:db, port=47]",
                    "[id=00:00:90:e2:ba:2b:c9:95, port=1]",
                    "[id=00:00:90:e2:ba:2b:c9:95, port=local]"
                ],
                "route-type": "SERVER_2_AGENT",
                "source-device": {
                    "ip-address": "10.0.0.4",
                    "mac-address": "02:63:1e:4f:ab:f1",
                    "tcp-port": "5001"
                }
            },
            "server": {
                "ip-address": "10.0.0.4",
                "mac-address": "02:63:1e:4f:ab:f1",
                "tcp-port": "5001"
            },
            "server-side-agent": {
                "active-transfers": [
                    "a84adbfb9b3eb3ab-0dec68f4-4a6241ba-b70fa102-fee1ad84cf1cd1498362eb63"
                ],
                "control-port": "9998",
                "data-port": "9877",
                "feedback-port": "9997",
                "ip-address": "10.0.0.3",
                "mac-address": "90:e2:ba:2b:c9:95"
            },
            "server-side-agent-tcp-port": {
                "length": "2",
                "port": "3728937296"
            },
            "time-init": "Mon Oct 26 14:2939:0144 EDT 2015",
            "time-start": "Mon Oct 26 14:2939:0245 EDT 2015",
            "time-stop": "none",
            "transfer-id": "9b3eb3ab-68f4-41ba-a102-d1498362eb63"
        }
    ],
    "agents": [
        {
            "active-transfers": [
                "a84adbfb9b3eb3ab-0dec68f4-4a6241ba-b70fa102-fee1ad84cf1cd1498362eb63"
            ],
            "control-port": "9998",
            "data-port": "9877",
            "feedback-port": "9997",
            "ip-address": "10.0.0.1",
            "mac-address": "00:60:dd:43:ac:72"
        },
        {
            "active-transfers": [
                "a84adbfb9b3eb3ab-0dec68f4-4a6241ba-b70fa102-fee1ad84cf1cd1498362eb63"
            ],
            "control-port": "9998",
            "data-port": "9877",
            "feedback-port": "9997",
            "ip-address": "10.0.0.3",
            "mac-address": "90:e2:ba:2b:c9:95"
        }
    ],
    "terminated-connections": [],
    "whitelist-entries": [
        {
            "client-ip-address": "10.0.0.2",
            "server-ip-address": "10.0.0.4",
            "server-tcp-port": "5001"
        }
    ]
}