# proxy-verifier
**Repository Path**: mirrors_yahoo/proxy-verifier
## Basic Information
- **Project Name**: proxy-verifier
- **Description**: Proxy Verifier is an HTTP replay tool designed to verify the behavior of HTTP proxies. It builds a verifier-client binary and a verifier-server binary which each read a set of YAML or JSON files that specify the HTTP traffic for the two to exchange.
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 0
- **Forks**: 0
- **Created**: 2020-08-19
- **Last Updated**: 2026-02-21
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
Table of Contents
=================
* [Proxy Verifier](#proxy-verifier)
* [Traffic Replay Specification](#traffic-replay-specification)
* [HTTP Specification](#http-specification)
* [Server Response Lookup](#server-response-lookup)
* [HTTP/2 Specification](#http2-specification)
* [HEADERS and DATA frame](#headers-and-data-frame)
* [RST_STREAM frame](#rst_stream-frame)
* [GOAWAY frame](#goaway-frame)
* [Await](#await)
* [Protocol Specification](#protocol-specification)
* [PROXY protocol support](#proxy-protocol-support)
* [Session and Transaction Delay Specification](#session-and-transaction-delay-specification)
* [Keep Connection Open](#keep-connection-open)
* [Traffic Verification Specification](#traffic-verification-specification)
* [Field Verification](#field-verification)
* [URL Verification](#url-verification)
* [Status Verification](#status-verification)
* [Body Verification](#body-verification)
* [Example Replay File](#example-replay-file)
* [Installing](#installing)
* [Prebuilt Binaries](#prebuilt-binaries)
* [Building from Source](#building-from-source)
* [Prerequisites](#prerequisites)
* [Build](#build)
* [Using Prebuilt Libraries](#using-prebuilt-libraries)
* [ASan Instrumentation](#asan-instrumentation)
* [Debug Build](#debug-build)
* [Statically Link](#statically-link)
* [Running the Tests](#running-the-tests)
* [Unit Tests](#unit-tests)
* [Gold Tests](#gold-tests)
* [Usage](#usage)
* [Required Arguments](#required-arguments)
* [Optional Arguments](#optional-arguments)
* [--format <format-specification>](#--format-format-specification)
* [--keys <key1 key2 ... keyn>](#--keys-key1-key2--keyn)
* [--verbose](#--verbose)
* [--interface <interface>](#--interface-interface)
* [--no-proxy](#--no-proxy)
* [--strict](#--strict)
* [--rate <requests/second>](#--rate-requestssecond)
* [--repeat <number>](#--repeat-number)
* [--run-continuously](#--run-continuously)
* [--thread-limit <number>](#--thread-limit-number)
* [--qlog-dir <directory>](#--qlog-dir-directory)
* [--tls-secrets-log-file <secrets_log_file_name>](#--tls-secrets-log-file-secrets_log_file_name)
* [--send-buffer-size <size>](#--send-buffer-size-size)
* [--poll-timeout <timeout_ms>](#--poll-timeout-timeout_ms)
* [Tools](#tools)
* [Replay Gen replay_gen.py](tools/replay_gen.py)
* [-n,--number <NUMBER>](#-n--number-number)
* [-tl,--trans-lower <TRANS_LOWER>](#-tl--trans-lower-trans_lower)
* [-tu,--trans-upper <TRANS_UPPER>](#-tu--trans-upper-trans_upper)
* [-sl,--sess-lower <SESS_LOWER>](#-sl--sess-lower-sess_lower)
* [-su,--sess-upper <SESS_UPPER>](#-su--sess-upper-sess_upper)
* [-tp,--trans-protocols <TRANS_PROTOCOLS>](#-tp--trans-protocols-trans_protocols)
* [-u,--url-file <URL_FILE>](#-u--url-file-url_file)
* [-o,--output <OUTPUT>](#-o--output-output)
* [-p,--prefix <PREFIX>](#-p--prefix-prefix)
* [-j,--out-json](#-j--out-json)
* [Remap Config to URL List remap_config_to_url_list.py](tools/remap_config_to_url_list.py)
* [-o,--output <OUTPUT_FILE>](#-o--output-output_file)
* [--no-ip](#--no-ip)
* [Contribute](#contribute)
* [License](#license)
# Proxy Verifier
Proxy Verifier is an HTTP replay tool designed to verify the behavior of HTTP
proxies. It builds a verifier-client binary and a verifier-server binary which
each read a set of YAML files (or JSON files, since JSON is YAML) that specify
the HTTP traffic for the two to exchange.
Proxy Verifier supports the HTTP replay of the following protocols:
* HTTP/1.x and HTTP/2
* HTTP and HTTP over TLS
* IPv4 and IPv6
Broadly speaking, Proxy Verifier is designed to address two proxy testing
needs:
* **Traffic correctness testing**: In this context Proxy Verifier can be used
in manual or automated end to end tests to verify correct behavior of the
details of generally a small number of transactions. [Transaction
Box](https://github.com/SolidWallOfCode/txn_box) is an example of a tool that
relies entirely on Proxy Verifier for its automated end to end tests (see its
[autest](https://github.com/SolidWallOfCode/txn_box/tree/master/test/autest)
directory).
* **Production simulation testing**: In this context, Proxy Verifier is used in
an isolated lab environment configured as much like a production environment
as possible. The Verifier client and server are provided replay files as
input that are either auto generated or collected via a tool like [Traffic
Dump](https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/traffic_dump.en.html)
from an actual production environment. Proxy verifier then replays this
production-like traffic against the proxy under test. Proxy Verifier can
replay such traffic at rates over 10k requests per second. Here are some
examples where this use of Proxy Verifier can be helpful:
* Safely testing experimental versions of a patch.
* Running diagnostic versions of the proxy that may not be performant enough
for the production environment. Debug, Valgrind, and ASan builds are
examples of build configurations whose resultant proxy performance may be
impossible to run in production but can be run safely in the production
simulation via Proxy Verifier.
* Performance comparison across versions of the proxy. Since Proxy Verifier
replays the traffic for the same set of replay files consistently across
runs, different versions of the proxy can be compared with regard to their
performance against the same replay dataset. This affords the ability to
compare detect performance improvement or degradation across versions of the
proxy.
It should be noted that when using Proxy Verifier in a production simulation
environment, the proxy will be receiving traffic from the client that looks
like production traffic. Thus, by design, the HTTP request targets and Host
header fields will reference actual production servers. It will be important to
configure the proxy under test to direct these requests not to the actual
production servers but to the Proxy Verifier server or servers. The way this
is achieved will depend upon the proxy. One way to accomplish this is to
configure the production environment with a test DNS server such as
[MicroDNS](https://bitbucket.org/autestsuite/microdns/src/master/) configured
to resolve all host names to the Proxy Verifier server or servers in the
production simulation environment.
## Traffic Replay Specification
### HTTP Specification
Proxy Verifier traffic behavior is specified via YAML files. The behavior for
each connection is specified under the top-most `sessions` node, the value of
which is a sequence where each item in the sequence describes the
characteristics of each connection. For each session the protocol stack can be
specified via the `protocol` node. This node describes the protocol
characteristics of the connection such as what version of HTTP to use, whether
to use TLS and what the characteristics of the TLS handshake should be. In the
absence of a `protocol` node the default protocol is HTTP/1 over TCP. See
[Protocol Specification](#protocol-specification) below for details about the
`protocol` node. Each session is run in parallel by the verifier-client
(although see [--thread-limit <number>](#--thread-limit-number) below for
a way to serialize them).
In addition to protocol specification, each session in the `sessions`
sequence contains a `transactions` node. This itself is a sequence of
transactions that should be replayed for the associated session. Within each
transaction, Proxy Verifier's traffic replay behavior is specified in the
`client-request` and `server-response` nodes. `client-request` nodes are used
by the Proxy Verifier client and tell it what kind of HTTP requests it should
generate. `server-response` nodes are used by the Proxy Verifier server to
indicate what HTTP responses it should generate. Proxy traffic verification
behavior is described in the `proxy-request` and `proxy-response` nodes which
will be covered in [Traffic Verification
Specification](#traffic-verification-specification). For HTTP/1, these
transactions are run in sequence for each session; for HTTP/2, the transactions
are run in parallel.
For HTTP/1 requests, `client-request` has a map as a value which contains each of
the following key/value YAML nodes:
1. `method`: This takes the HTTP method as the value, such as `GET` or `POST`.
1. `url`: This takes the request target as the value, such as `/a/path.asp` or
`https://www.example.com/a/path.asp`.
1. `version`: This takes the HTTP version, such as `1.1` or `2`.
1. `headers`: This takes a `fields` node which has, as a value, a sequence of
HTTP fields. Each field is itself a sequence of two values: the name of the
field and the value for the field.
1. `content`: This specifies the body to send. It takes a map as a value. The
user can specify a `size` integer value in which an automated body of that
size will be generated. Otherwise a `data` string value can be provided in
which the specified body will be sent and an `encoding` taking `plain` or
`uri` to specify that the string is raw or URI encoded.
Here's an example of a `client-request` node describing an HTTP/1.1 POST
request with a request target of `/pictures/flower.jpeg`
and containing four header fields with a body size of 399 bytes:
```YAML
client-request:
method: POST
url: /pictures/flower.jpeg
version: '1.1'
headers:
fields:
- [ Host, www.example.com ]
- [ Content-Type, image/jpeg ]
- [ Content-Length, '399' ]
- [ uuid, 1234 ]
content:
size: 399
```
For convenience, if Proxy Verifier detects a Content-Length header, then the
`content` node stating the size of the body is not required. Thus this can be
simplified to:
```YAML
client-request:
method: POST
url: /pictures/flower.jpeg
version: '1.1'
headers:
fields:
- [ Host, www.example.com ]
- [ Content-Type, image/jpeg ]
- [ Content-Length, '399' ]
- [ uuid, 1234 ]
```
`server-response` nodes indicate how the Proxy Verifier server should respond
to HTTP requests. In place of `method`, `url`, and `version`, HTTP/1 responses
take the following nodes:
1. `status`: This takes an integer corresponding to the HTTP response status,
such as `200` or `404`.
1. `reason`: This takes a string that describes the status, such as `"OK"` or
`"Not Found"`.
Here's an example of an HTTP/1 `server-response` with a status of 200, four
fields, and a body of size 3,432 bytes:
```YAML
server-response:
status: 200
reason: OK
headers:
fields:
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, image/jpeg ]
- [ Transfer-Encoding, chunked ]
- [ Connection, keep-alive ]
content:
size: 3432
```
Observe in this case that the response contains the `Transfer-Encoding:
chunked` header field, indicating that the body should be chunk encoded. Proxy
Verifier supports chunk encoding for both requests and responses and will use
this when it detects the `Transfer-Encoding: chunked` header field. In this
case, the 3,432 bytes will be the size of the chunked body payload without the
bytes for the chunk protocol (chunk headers, etc.).
Finally, here is an example of a response with specific body content sent (YAML
in this case) as opposed to the generated content specified by the
`content:size` nodes above:
```YAML
server-response:
status: 200
reason: OK
headers:
fields:
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, text/yaml ]
- [ Transfer-Encoding, chunked ]
- [ Connection, keep-alive ]
content:
encoding: plain
data: |
### Heading
* Bullet
* Points
```
#### Server Response Lookup
The `client-request` and `server-response` nodes are all that is required to
direct Proxy Verifier's replay of a transaction. The next section will describe
how to specify proxy transaction verification behavior via the `proxy-request` and
`proxy-response` nodes. Before proceeding to that, however, it is valuable to
understand how the Verifier server decides which response to send for any given
request it receives. Consider again the `client-request` node in the preceding
example:
```YAML
client-request:
method: POST
url: /pictures/flower.jpeg
version: '1.1'
headers:
fields:
- [ Host, www.example.com ]
- [ Content-Type, image/jpeg ]
- [ Content-Length, '399' ]
- [ uuid, 1234 ]
content:
size: 399
```
Note the presence of the `uuid` field. A field such as this is sent by the
client and used by the server. When a server receives a request, keep in mind
that all it has to direct its behavior is what it has parsed from the replay
file or files and what it sees in the request. There may be hundreds of
thousands of parsed `server-response` nodes, each describing a unique response
with which the server may reply for any given incoming request. The server does
not talk directly to the client, so it cannot communicate with it and say, "Hey
Verifier client, I just read such and such request off the wire. Which response
would you like me to send for this?" When it receives an HTTP request,
therefore, it only has the contents of that request from which to choose its
response. To facilitate the server's behavior, a unique *key* is associated
with every request. During the YAML parsing phase, the Verifier server
associates each parsed `server-response` with a key it derives from the
associated `client-request` for the transaction. During the traffic replay
phase, when it reads a request off the wire, it derives a key from the HTTP
request using the same process for generating keys from `client-request` nodes
used in the parsing phase. It then looks up the proper YAML-specified response
using that key. By default, Proxy Verifier uses the `uuid` HTTP header field
values as the key, as utilized in the examples in this document, but this is
configurable via the `--format` command line argument. For details see the
section describing this argument below.
Both the client and the server will fail the YAML parsing phase and exit with a
non-zero return if either parses a transaction for which they cannot derive a
key. If during the traffic processing phase the Verifier server somehow
receives a request for which it cannot derive a key, it will return a *404 Not
Found* response and close the connection upon which it received the request.
### HTTP/2 Specification
For HTTP/2, the protocol describes the initial request line values with pseudo
header fields. For that protocol, therefore, the user need not specify the
`method`, `url`, and `version` nodes and would instead specify these values
more naturally as pseudo headers in the `fields` sequence. Here is an example of
an HTTP/2 `client-request` analogous to the HTTP/1 request above (note that the
`Content-Length` header field is optional in HTTP/2 and as such is omitted here):
```YAML
client-request:
headers:
fields:
- [ :method, POST ]
- [ :scheme, https ]
- [ :authority, www.example.com ]
- [ :path, /pictures/flower.jpeg ]
- [ Content-Type, image/jpeg ]
- [ uuid, 1234 ]
content:
size: 399
```
The status code is described in the `:status` pseudo header field.
Also, HTTP/2 does not allow for chunk encoding nor does it use the `Connection`
header field, using instead its framing mechanism to describe the body and
session lifetime. An analogous HTTP/2 response to the HTTP/1 request above,
therefore, would look like the following:
```YAML
server-response:
headers:
fields:
- [ :status, 200 ]
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, image/jpeg ]
content:
size: 3432
```
Trailer response headers are useful for transmitting additional fields after the
message body, providing dynamically generated metadata such as integrity checks
or post-processing status. Proxy Verifier supports sending and receiving trailer
headers, as demonstrated below:
```YAML
server-response:
status: 200
headers:
fields:
# Some fields...
content:
# Some data...
trailers:
encoding: esc_json
fields:
- [ x-test-trailer-1, one ]
- [ x-test-trailer-2, two ]
```
It is also possible to specify everything as a sequence of frames. The available
options for the frame sequence are:
* `DATA`
* `HEADERS`
* `RST_STREAM`
* `GOAWAY`
Here's an example replay file:
```YAML
client-request:
frames:
- HEADERS:
headers:
fields:
- [ :method, POST ]
- [ :scheme, https ]
- [ :authority, www.example.com ]
- [ :path, /pictures/flower.jpeg ]
- [ Content-Type, image/jpeg ]
- [ uuid, 1234 ]
- DATA:
content:
size: 399
- RST_STREAM:
error-code: STREAM_CLOSED
server-response:
frames:
- HEADERS:
headers:
fields:
- [ :status, 200 ]
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, image/jpeg ]
- DATA:
content:
size: 3432
```
#### HEADERS and DATA frame
The `HEADERS` and `DATA` frame nodes are designed to be specified in a way that is consistent
with their HTTP/1 counterparts. From a parsing perspective, this means they simply wrap the
`headers` and `content` nodes that are used for HTTP/1 specification as described above. For
an example, see the `RST_STREAM` frame section below.
Note that multiple `DATA` frames can be specified for requests and responses. The replay follows
the same order the `DATA` frames are listed. See the example below:
```YAML
client-request:
frames:
- HEADERS:
headers:
fields:
- [:method, POST]
- [:scheme, https]
- [:authority, example.data.com]
- [:path, /a/path]
- [Content-Type, text/html]
- [uuid, 1]
- DATA:
content:
encoding: plain
data: client_data_1
- DATA:
content:
encoding: plain
data: client_data_2
```
#### RST_STREAM frame
In some cases, there might be a need to test the behavior of the proxy when the
client or server terminates the stream, either because of an unexpected error or intentionally
cancelling the stream. In HTTP/2, peers terminate a transaction by sending a `RST_STREAM` frame
which includes an error code. In the replay file, this is specified via a `RST_STREAM` frame
node which is ordered within the stream to indicate when it should be sent and includes an
`error-code` node to specify which error code should be used.
The following sample replay file snippet demonstrate how to specify such a test scenario:
```YAML
sessions:
- protocol:
- name: http
version: 2
- name: tls
sni: test_sni
- name: tcp
- name: ip
version: 4
transactions:
- client-request:
frames:
- HEADERS:
headers:
fields:
- [:method, POST]
- [:scheme, https]
- [:authority, example.data.com]
- [:path, /a/path]
- [Content-Type, text/html]
- [Content-Length, '11']
- [uuid, 1]
- RST_STREAM:
error-code: INTERNAL_ERROR
- DATA:
content:
encoding: plain
data: client_test
size: 11
```
Note that this example specifies the following:
* A sequence of frames to be sent under the `client-request` node. The order
of the frames listed is the order of the frames that Proxy Verifier will send during
the traffic replay of the stream.
* Thus, in this case, the client terminates the stream after sending the `HEADERS` frame since
the `RST_STREAM` is specified after the `HEADERS` frame in the sequence.
* The client terminates with the error code `INTERNAL_ERROR`, specified by `error-code` under
the `RST_STREAM` frame.
Be aware that when Proxy Verifier generates HTTP/2 frames, it determines from the frames elements where to
insert the `END_STREAM` flag. If a request or response just has a `HEADERS` frame, then the `END_STREAM`
flag will be added to the end of the `HEADERS` frame. If there are both `HEADERS` and DATA frames, then the
`END_STREAM` will be placed after the `DATA` frame. Proxy Verifier does not, however, use `RST_STREAM`
frames to influence the `END_STREAM` flag. Thus in the above example, having the `RST_STREAM` between the
`HEADERS` and `DATA` frames means that the `HEADERS` will not have an `END_STREAM` flag before the
`RST_STREAM` is sent. For this reason, you must include a `DATA` frame after a `RST_STREAM` frame, even
though the `DATA` frame will not be sent, in order to keep the stream open at the time the `RST_STREAM` is
sent.
The server side stream termination can be set in the same way, but under the `server-response` node.
The available options for `error-code` are:
* NO_ERROR
* PROTOCOL_ERROR
* INTERNAL_ERROR
* FLOW_CONTROL_ERROR
* SETTINGS_TIMEOUT
* STREAM_CLOSED
* FRAME_SIZE_ERROR
* REFUSED_STREAM
* CANCEL
* COMPRESSION_ERROR
* CONNECT_ERROR
* ENHANCE_YOUR_CALM
* INADEQUATE_SECURITY
* HTTP_1_1_REQUIRED
Finer grained frame ordering can also be specified via a `delay` node which specifies the time
delay before sending the `RST_STREAM` frame. For example, to delay sending a `RST_STREAM` frame
for 1 second, specify a `delay: 1s` directive like so:
```YAML
- RST_STREAM:
error-code: INTERNAL_ERROR
delay: 1s
```
A detailed description of the `delay` node can be found [here](#session-and-transaction-delay-specification).
#### GOAWAY frame
The `GOAWAY` frame acts similar to the `RST_STREAM` frame shown above. However, rather than terminating
a stream, `GOAWAY` frame terminates the connection. It supports `error-code` and `delay` as described
for `RST_STREAM` frame above.
HTTP/2 sessions also have a `close-on-goaway` directive. This boolean
configuration informs how the Verifier client should behave when there are more
streams configured in the replay YAML file after a `GOAWAY` frame is received
(see [RFC 7540 section
6.8](https://datatracker.ietf.org/doc/html/rfc7540#section-6.8) for details
about the `GOAWAY` frame). When set to `true`, on receipt of a `GOAWAY` frame,
the client terminates the connection after processing the streams that are
currently being processed. When set to `false`, the client does not terminate
the connection and continues with subsequent streams specified in the replay
file. The default value of this option is `true` since that is in keeping with
the RFC. Here's an example session configured with `close-on-goaway` set to
`false`:
```YAML
sessions:
- protocol:
- name: http
version: 2
- name: tls
sni: test_sni
- name: tcp
- name: ip
version: 4
close-on-goaway: false
transactions:
```
#### Await
By protocol specification, HTTP/1 transactions are serialized. That is, apart
from the rarely used pipelined request protocol feature (which is not supported
by Proxy Verifier), each HTTP/1 transaction in a connection is not exchanged
until the request and response of the previous one is completed. By contrast,
the HTTP/2 protocol specifies that streams (i.e., HTTP/2 transactions) are
multiplexed within their sessions (i.e., HTTP/2 connections). For the Verifier
client, this means that, by default, requests for streams within a session are
each sent immediately in the order that they are specified in the replay file
without waiting for their respective responses. That is, if a replay file
specifies three streams within a session, then the client will send the request
for each stream as quickly as it can serialize it on the socket for the
session before waiting for any responses.
The timing for sending the stream requests can be modified from this default
behavior in several ways. As with replaying HTTP/1 traffic, if the streams
contain timing specification nodes, then the client can be configured to replay
the streams at a rate scaled to that timing information. See the documentation
for the [--rate](#--rate-requestssecond) argument for a description of how
this works. Also, the user can include
[delay](#session-and-transaction-delay-specification) nodes to space out the
replay of each stream.
In addition to these configurations supported in HTTP/1 traffic replay, for
HTTP/2 and HTTP/3 the Verifier client supports the `await` node to control when
it starts streams. This node takes a transaction key or a sequence of
transaction keys, the responses of which are dependencies for sending the request
for the associated stream. That is, a stream with an `await` node will not be
sent until all the responses are received for the listed keys.
As an example, consider the specification of the following session with two
streams:
```YAML
# Specify an HTTP/2 session
- protocol:
- name: http
version: 2
- name: tls
sni: test_sni
- name: tcp
- name: ip
transactions:
# Stream 1
- client-request:
headers:
fields:
- [ :method, GET ]
- [ :scheme, https ]
- [ :authority, www.example.com ]
- [ :path, /pictures/flower.jpeg ]
- [ Content-Type, image/jpeg ]
- [ uuid, first-stream ]
server-response:
headers:
fields:
- [ :status, 200 ]
- [ Content-Type, image/jpeg ]
- [ X-Response, first-response ]
content:
size: 3432
# Stream 2
- client-request:
# This await will cause the Verifier client to hold off on sending this
# request until the response to first-stream is received.
await: first-stream
headers:
fields:
- [ :method, GET ]
- [ :scheme, https ]
- [ :authority, www.example.com ]
- [ :path, /pictures/flower.jpeg ]
- [ Content-Type, image/jpeg ]
- [ uuid, second-stream ]
server-response:
headers:
fields:
- [ :status, 200 ]
- [ Content-Type, image/jpeg ]
- [ X-Response, second-response ]
content:
size: 3432
```
Notice that the second transaction with key `second-stream` contains an
`await` node. Without this node, the Verifier client would send both requests
back to back. That is, the `second-stream` request would be sent immediately
after the `first-stream` request was sent. With the `await` node specifying a
dependency upon the `first-stream` transaction, however, the Verifier client
will instead delay sending the `second-stream` request until the response to
`first-stream` was received from the proxy.
There are a few things related to `await` nodes to be cognizant of:
* `await` nodes can be used in conjunction with `delay` nodes for
`client-request` specifications. When both are used, the Verifier client will
first hold off on sending the request until all dependant responses specified
by the keys in `await` are processed. Once the dependant responses are
received, then the `delay` node is processed and the request is further
delayed for the amount of time specified by the value of the `delay` node.
Then the request is sent after the delay.
* As stated before, the Verifier client is designed to processes streams and
their directives in the order that they are specified in the `transactions`
sequence for their stream. This means that any `await` or `delay` nodes for
one transaction causes a corresponding delay for later nodes before they are
processed. That is to say, if a session is specified with five streams, and
the third stream has an `await` specified for the first two streams, then the
fourth and fifth streams will also be delayed behind the third stream while
it awaits the responses for the first and second streams, even if streams
four and five contain no `await` nor `delay` nodes themselves.
* Transaction keys specified via `await` can only be for earlier transactions
in the same session that the `client-request` node is in. That is, the
Verifier client does not support a stream in one session to `await` the
response to a stream in a different session.
### Protocol Specification
The above discussed the replay YAML nodes that describe how Proxy Verifier will
craft HTTP layer traffic. This section discusses how the user specifies the
lower layer protocols used to transport this HTTP traffic.
As stated above, each HTTP session is described as an item under the `sessions`
node sequence. Each session takes a map. HTTP transactions are described under
the `transactions` key described above. In addition to `transactions`, a
session also takes an optional `protocol` node. This node takes an ordered
sequence of maps, where each item in the sequence describes the characteristics
of a protocol layer. The sequence is expected to be ordered from higher layer
protocols (such as HTTP and TLS) to lower layer protocols (such as IP).
Here is an example protocol node along with `sessions` and `transactions`
nodes provided to give some context:
```YAML
sessions:
- protocol:
- name: http
version: 2
- name: tls
sni: test_sni
- name: tcp
- name: ip
transactions:
# ...
```
Note again how the `protocol` node is under the `sessions` node which takes a
sequence of sessions. This sample shows the start of a single session that, in
this case, provides a protocol description via a `protocol` key. This same
session also has a truncated set of transactions that will be specified under
the `transactions` key. Looking further at the `protocol` node, observe that
this session has four layers described for it: http, tls, tcp, and ip. The
`http` node specifies that the session should use the HTTP/2 protocol. The
`tls` node specifies that the client should use an SNI of "test\_sni" in the
TLS client hello handshake. Further, this should be transported over TCP on IP.
The following nodes are supported for `protocol`:
| Name | Node | Supported Values | Description
| ----- |-------- | ---------------- | -----------
| http | | |
| | version | {1, 2} | Whether to use HTTP/1 or HTTP/2.
| tls | | |
| | sni | string | The SNI to send in the TLS handshake.
| | request-certificate | boolean | Whether the client or server should request a certificate from the proxy.
| | proxy-provided-certificate | boolean | This directs the same behavior as the request-certificate directive. This alias is helpful when the node describes what happened in the past, such as in the context of a replay file specified by [Traffic Dump](https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/traffic_dump.en.html).
| | verify-mode | {0-15} | The value to pass directly to OpenSSL's `SSL_set_verify` to control peer verification in the TLS handshake. This allows fine grained control over TLS verification behavior. `0` corresponds with SSL_VERIFY_NONE, `1` corresponds with SSL_VERIFY_PEER, `2` corresponds with SSL_VERIFY_FAIL_IF_NO_PEER_CERT, `4` corresponds with SSL_VERIFY_CLIENT_ONCE, and `8` corresponds with SSL_VERIFY_POST_HANDSHAKE. Any bitwise OR'd value of these values can be provided. For details about their behavior, see OpenSSL's [SSL_verify_cb](https://www.openssl.org/docs/man1.1.1/man3/SSL_verify_cb.html) documentation.
| | alpn-protocols | sequence of strings | This specifies the server's protocol list used in ALPN selection. See OpenSSL's [SSL_select_next_proto](https://www.openssl.org/docs/man1.0.2/man3/SSL_select_next_proto.html) documentation for details.
| proxy-protocol | | |
| | version | {1, 2} | Whether to use PROXY header v1 or v2
| | src-addr | string | The source address and port in the PROXY header. Specified in the format of `111.111.111.111:11111`.
| | dst-addr | string | The destination address and port in the PROXY header. Specified in the same format of `src-addr`.
| tcp | | |
| ip | | |
The following protocol specification features are not currently implemented:
* HTTP/2 is only supported over TLS. Proxy Verifier uses ALPN in the TLS
handshake to negotiate HTTP/2 with the proxy. HTTP/2 upgrade from HTTP/1
without TLS is not supported.
* The user cannot supply a TLS version to negotiate for the handshake.
Currently, if the TLS node is present, Proxy Verifier will use the highest
TLS version it can negotiate with the peer. This is OpenSSL's default
behavior. An enhancement request to support A TLS version specification
feature request is recorded in issue
[101](https://github.com/yahoo/proxy-verifier/issues/101).
* Similarly, the user cannot specify whether to use IPv4 or IPv6 via an
`ip:version` node. Proxy Verifier can test IPv6, but it does so via the user
passing IPv6 addresses on the commandline. An IP version feature request is
recorded in issue [100](https://github.com/yahoo/proxy-verifier/issues/100).
* Only TCP is supported. There have been recent discussions about adding
HTTP/3 support, which is over UDP, but work for that has not yet started.
* The PROXY protocol support is limited to the `PROXY` command via TCP over IPv4 or IPv6. Therefore,
* No support for `AF_UNIX` socket address as either source and destination address.
* No support for UDP transport type.
* No support for `LOCAL` command(in v2 header).
If there is no `protocol` node specified, then Proxy Verifier will default to
establishing an HTTP/1 connection over TCP (no TLS).
#### PROXY protocol support
Generally speaking, a server sitting downstream from a proxy does not have
visibility into the client's network socket information that lies behind the
proxy. PROXY Protocol is a mechanism to provide visibility for this. PROXY
protocol is a network protocol that communicates a client's source and
destination IP and port information via a set of bytes at the start of a TCP
connection from a proxy. Here is a link to the protocol description:
https://github.com/haproxy/haproxy/blob/master/doc/proxy-protocol.txt
Proxy Verifier supports sending and receiving the PROXY protocol, which can be
helpful to verify the PROXY protocol behavior of the proxy under test.
The feature can be enabled by specifying the `proxy-protocol` protocol node as
outlined above. If enabled, the Verifier client would send out the PROXY
protocol header at the beginning of the connection. Upon receiving a PROXY
protocol header, the Verifier server would display it in the human-readable v1
format. Note that the specification of source and destination addresses are
supported but not required; if not specified, the Proxy Verifier client would
send out PROXY message with the source and destination addresses matching the
underlying socket, similar to how `curl --haproxy-protocol` behaves.
### Session and Transaction Delay Specification
A user can also stipulate per session and/or per transaction delays to be
inserted by the Verifier client and server during the replay of traffic. This
is done via the `delay` node which takes a unit-specified duration for the
associated delay. During traffic replay, the delay is inserted before the
associated session is established or before the client request or server
response is sent. Proxy Verifier recognizes the following time duration units
for the `delay` node:
Unit Suffix | Meaning
----------- | -------
s | seconds
ms | milliseconds
us | microseconds
Here is a sample replay file snippet that demonstrates the specification of a
set of delays:
```YAML
sessions:
- delay: 2s
transactions:
client-request:
delay: 15ms
method: POST
url: /a/path.jpeg
version: '1.1'
headers:
fields:
- [ Content-Length, '399' ]
- [ Content-Type, image/jpeg ]
- [ Host, example.com ]
- [ uuid, 1 ]
server-response:
delay: 17000 us
status: 200
reason: OK
headers:
fields:
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, image/jpeg ]
- [ Transfer-Encoding, chunked ]
- [ Connection, keep-alive ]
content:
size: 3432
```
Note that this example specifies the following delays:
* The client delays 2 seconds before establishing the session.
* The client also delays 15 milliseconds before sending the client
request.
* The server delays 17 milliseconds (17,000 microseconds) before sending the
corresponding response after receiving the request.
Be aware of the following characteristics of the `delay` node:
* The Verifier client interprets and implements `delay` for sessions in the
`sessions` node and for transactions in the `client-request` node. The
Verifier server interprets delay only for transactions in the
`server-response` node and ignores `sessions` delays. Since the server is
passive in receiving connections, it's not obvious what a server-side session
delay would mean in this context.
* Notice that Proxy Verifier supports microsecond level delay granularity, and
does indeed faithfully insert delays at the appropriate times during replay
with that precision of time. Be aware, however, that for the vast majority of
networks anything more precise than a millisecond will not generally be useful.
See also [--rate <requests/second>](#--rate-requestssecond) below for
rate specification of transactions.
### Keep Connection Open
In certain special situations, a user might need to keep the connection open
after the final transaction in a session is done. To specify how long the connection
needs to be kept open, the user can specify the duration as follows (value format is
the same as [Session and Transaction Delay Specification](#session-and-transaction-delay-specification)):
```YAML
sessions:
- protocol:
- name: http
version: 2
- name: tls
sni: test_sni
- name: tcp
- name: ip
version: 4
keep-connection-open: 2s
transactions:
client-request:
delay: 15ms
method: POST
url: /a/path.jpeg
version: '1.1'
headers:
fields:
- [ Content-Length, '399' ]
- [ Content-Type, image/jpeg ]
- [ Host, example.com ]
- [ uuid, 1 ]
server-response:
delay: 17000 us
status: 200
reason: OK
headers:
fields:
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, image/jpeg ]
- [ Transfer-Encoding, chunked ]
- [ Connection, keep-alive ]
content:
size: 3432
```
## Traffic Verification Specification
In addition to replaying HTTP traffic as described above, Proxy Verifier also
implements proxy traffic verification. For any given transaction, Proxy
Verifier can optionally verify characteristics of an HTTP request line (for
HTTP/1 requests - HTTP/2 requests do not have a request line), the response
status, and the content of HTTP fields for requests and responses. Proxy
Verifier also supports field verification of HTTP/2 pseudo header fields.
Whereas `client-request` and `server-response` nodes direct the Verifier client
and server (respectively) on how to send traffic, `proxy-request` and
`proxy-response` nodes direct the server and client (respectively) on how to
verify the traffic it receives from the proxy. Thus:
* `client-request` nodes are used by the Verifier client to direct how it will
generate requests it will send to the proxy.
* `server-response` nodes are used by the Verifier server to direct how it will
generate responses to send to the proxy.
* `proxy-request` nodes are used by the Verifier server to direct how it will
verify requests received from the proxy.
* `proxy-response` nodes are used by the Verifier client to direct how it will
verify responses received from the proxy.
In the event that the proxy does not produce the stipulated traffic according
to a verification directive, the Proxy Verifier client or server that detects
the violation will emit a log indicating the violation and will, upon process
exit, return a non-zero status to the shell.
Traffic verification is an optional feature. Thus if Proxy Verifier is being
used simply to replay traffic and the verification features are not helpful,
then the user can simply omit the `proxy-request` and `proxy-response` nodes
and no verification will be performed.
The following sections describe how to specify traffic verification in the YAML
replay file.
### Field Verification
Recall that in `client-request` and `server-response` nodes, fields are
specified via a sequence of two items: the field name followed by the field
value. For instance, the following incomplete `client-request` node contains
the specification of a single `Content-Length` field (it is incomplete because
it does not specify the method, request target, etc., but this snippet
sufficiently demonstrates a typical description of an HTTP field):
```YAML
client-request:
headers:
fields:
- [ Content-Length, 399 ]
```
For this client request, Proxy Verifier is directed to create a `Content-Length`
HTTP field with a value of `399`.
Field verification is specified in a similar manner, but the second item in the
sequence is a map describing how the field should be verified. The map takes a
`value` item describing the characteristics of the value to verify, if
applicable, and an `as` item providing a directive describing how the field
should be verified. Here is an example of a `proxy-request` node directing the
Verifier server to verify the characteristics of the `Content-Length` field
received from the proxy:
```YAML
proxy-request:
headers:
fields:
- [ Content-Length, { value: 399, as: equal } ]
```
Observe the following from this verification example:
* As described above, notice that the second entry in the field specification
is a map instead of a scalar field value. The Proxy Verifier parser
recognizes this map type as providing a verification specification.
* As with `client-request` and `server-response` field specifications, the
first item in the list describes the field name, in this case
"Content-Length". This indicates that this verification rule applies to the
"Content-Length" HTTP field from the proxy.
* The directive is specified via the `as` key. In this example, `equal` is the
directive for this particular field verification, indicating that the
Verifier server should verify that the proxy's request for this transaction
contains a `Content-Length` field with the exact value of `399`.
Proxy Verifier supports six field verification directives:
Directive | Description
--------- | ------------
absent | The *absence* of a field with the specified field name.
present | The *presence* of a field with the specified field name and having any or no value.
equal | The presence of a field with the specified field name and a value *equal* to the specified value.
contains | The presence of a field with the specified name with a value *containing* the specified value.
prefix | The presence of a field with the specified name with a value *prefixed* with the specified value.
suffix | The presence of a field with the specified name with a value *suffixed* with the specified value.
includes | Helpful for multi-value fields. Specifies that the set of value strings exist in the header field values for a particular header name in any order. Note that each value is matched against the set like ``contains``, so each value is a substring match.
For all of these field verification behaviors, field names are matched case
insensitively while field values are matched case sensitively.
Thus the following field specification requests no field verification because it does not include a directive and the second item in the sequence is a scarlar:
```YAML
- [ X-Forwarded-For, 10.10.10.2 ]
```
Such non-operative fields can also be specified using the map syntax without an
`as` item:
```YAML
- [ X-Forwarded-For, { value: 10.10.10.2 } ]
```
Fields like this in `proxy-request` and `proxy-response` nodes are permissible
by Proxy Verifier's parser even though they have no functional impact (i.e.,
they do not direct Proxy Verifier's traffic behavior because they are not in
`client-request` nor `server-response` nodes, and they do not describe any
verification behavior). Allowing such fields affords the user the ability to
record the proxy's traffic behavior in situations where field verification is
not required or desired. For example, the [Traffic
Dump](https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/traffic_dump.en.html)
Traffic Server plugin records HTTP traffic and uses these proxy HTTP fields to
indicate what fields were sent by the Traffic Server proxy to the client and
the server. This can be helpful for analyzing the proxy's behavior via these
replay files. Thus this proxy traffic recording function can be helpful to the
user even though Proxy Verifier treats the fields as non-operable.
The following demonstrates the `absent` directive which specifies that the HTTP field `X-Forwarded-For` _with any value_ should not have been sent by the proxy:
```YAML
- [ X-Forwarded-For, { as: absent } ]
```
The following demonstrates the `present` directive which specifies that the HTTP field `X-Forwarded-For` _with any value_ should have been sent by the proxy:
```YAML
- [ X-Forwarded-For, { as: present } ]
```
Notice that for both the `absent` and the `present` directives, the `value` map item is not relevant and need not be provided and, in fact, will be ignored by Proxy Verifier if it is provided.
The following demonstrates the `equal` directive which specifies that `X-Forwarded-For` should have been received from the proxy with the exact value "10.10.10.2":
```YAML
- [ X-Forwarded-For, { value: 10.10.10.2, as: equal } ]
```
The following demonstrates the `contains` directive which specifies that `X-Forwarded-For` should have been received from the proxy containing the value "10" at any position in the field value:
```YAML
- [ X-Forwarded-For, { value: 10, as: contains } ]
```
The following demonstrates the `prefix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value starting with "1":
```YAML
- [ X-Forwarded-For, { value: 1, as: prefix } ]
```
The following demonstrates the `suffix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value ending with "2":
```YAML
- [ X-Forwarded-For, { value: 2, as: suffix } ]
```
The following demonstrates the `includes` directive which specifies that `Set-Cookie` should have been received from the proxy including the values "A" and "B" at any position in the field value:
```YAML
- [ Set-Cookie, { value: [A, B] , as: includes } ]
```
Proxy Verifier supports inverting the result of any rule by using `not` instead of `as`. The following demonstrates the `prefix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value not starting with "a":
```YAML
- [ X-Forwarded-For, { value: a, not: prefix } ]
```
Proxy Verifier also supports ignoring the upper/lower case distinction with another directive: `case: ignore`. The following demonstrates the `suffix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value starting with "a" or "A":
```YAML
- [ X-Forwarded-For, { value: a, as: prefix, case: ignore } ]
```
The `not` and `case: ignore` directives can both be applied on the same rule. The following demonstrates the `suffix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value not starting with "a" nor "A":
```YAML
- [ X-Forwarded-For, { value: a, not: prefix, case: ignore } ]
```
In addition to HTTP/2 trailer header replay discussed above, Proxy Verifier also supports trailer header verification, as demonstrated below:
```YAML
proxy-response:
# Other verifications...
trailers:
fields:
# Verify the client receives the response trailers.
- [ x-test-trailer-1, { value: one, as: equal } ]
- [ x-test-trailer-2, { value: two, as: equal } ]
```
To perform multi-value field verification, a specific format must be adhered to.
This format involves specifying each value within a sequence, ensuring that each
value is individually verified according to the defined rules.
Be aware that field verification is order sensitive. That is, the field values
will be verified in the order that they are specified in the verification rule.
See example below, value `B1` and `B2` are specified in the same order in the
header fields and verification rule:
```YAML
server-response:
headers:
fields:
- [:status, 200]
- [Content-Type, text/html]
- [Content-Length, '11']
- [Set-Cookie, "A1=111"]
- [Set-Cookie, "A2=222"]
- [Set-Cookie, "B1=333"]
content:
encoding: plain
data: server_test
size: 11
proxy-response:
headers:
fields:
- [Set-Cookie, { value: [A1=111, A2=222, B1=333] , as: equal }]
```
There is an exception for the `includes` check, where it is not order sensitive.
See exammple below:
```YAML
server-response:
headers:
fields:
- [:status, 200]
- [Content-Type, text/html]
- [Content-Length, '11']
- [Set-Cookie, "A1=111"]
- [Set-Cookie, "A2=222"]
- [Set-Cookie, "B1=333"]
- [Set-Cookie, "B2=444"]
- [Set-Cookie, "C1=555"]
- [Set-Cookie, "C2=666"]
- [Set-Cookie, "D1=777"]
- [Set-Cookie, "D2=888"]
content:
encoding: plain
data: server_test
size: 11
proxy-response:
headers:
fields:
- [Set-Cookie, { value: [B2=, A2=, C2=, D1=, C1=] , as: includes }]
```
### URL Verification
In a manner similar to field verification described above, a mechanism exists
to verify the parts of URLs in the request line being received from the proxy
by the server. This mechanism is useful for verifying the request targets of
HTTP/1 requests. For HTTP/2 requests, the analogous verification is done via
pseudo header field verification of the `:scheme`, `:authority`, and
`:path` fields using the above described field verification.
Request target verification rules are stipulated via a sequence value for the
`url` node rather than the scalar value used in `client-request` nodes. The
verifiable parts of the request target follow the URI specification (see
[RFC 3986 section 3](https://tools.ietf.org/html/rfc3986#section-3) for the formal
definition of these various terms):
* `scheme`
* `host`
* `port`
* `authority` (also known as `net-loc`, the combination of `host` and `port`),
* `path`
* `query`
* `fragment`
For example, consider the following request line:
```
GET http://example.one:8080/path?query=q#Frag HTTP/1.1
```
The request URL in this case is case is
`http://example.one:8080/path?query=q#Frag`. The Verifier server can be
configured to verify the various parts of such URLs using the same map sytax
explained above for field verification using any of those same directives
(`equal`, `contains`, etc.). Continuing with this example URL, the following
uses the `equal` directive for each part of the URL, thus verifying that the
request URL exactly matches the URL given in this example and emitting a
verification error message if any parts of the URL do not match:
```YAML
proxy-request:
url:
- [ scheme, { value: http, as: equal } ]
- [ host, { value: example.one, as: equal } ]
- [ port, { value: 8080, as: equal } ]
- [ path, { value: /path, as: equal } ]
- [ query, { value: query=q, as: equal } ]
- [ fragment, { value: Frag, as: equal } ]
```
Alternatively to `host` and `port`, `authority`, with an alias of `net-loc`, is
supported, which is the combination of the two:
```YAML
- [ authority, { value: example.one:8080, as: equal } ]
```
As another example using other directives, consider a request URL of
`/path/x/y?query=q#Frag`. Verification of this can be specified with the
following:
```YAML
proxy-request:
url:
- [ scheme, { value: http, as: absent } ]
- [ authority, { value: foo, as: absent } ]
- [ path, { value: /path/x/y, as: equal } ]
- [ query, { value: query=q, as: equal } ]
- [ fragment, { value: foo, as: present } ]
```
Note that `scheme` and `authority` parts both use `absent` directives because
this particular URL just has path, query, and fragment components. Thus, with
this verification specification, if the proxy includes scheme or authority in
the request target, it will result in a verification failure. The `path` and
`query` components must match `/path/x/y` and `query=q` exactly because they
use the `equal` directive. The `fragment` of the URL is verified with the
`present` directive in this case, indicating that the received URL from the
proxy only needs to have some fragment of any value to pass this specified
verification.
### Status Verification
Proxy HTTP response status verification is specified in `proxy-response`
nodes. In this case, the response status that the Verifier client should expect
from the proxy is specified in the same way that directs the Verifier server
in what response status should be sent for a given request. Both response code,
such as "403", and HTTP/1 response reason string, such as "Forbidden", are
supported for verification.
For example, the following complete `proxy-response` node directs the Proxy
Verifier client to verify that the proxy replies to the associated HTTP request
with a `404` status:
```YAML
proxy-response:
status: 404
reason: Not Found
```
This verification directive applies to HTTP/1 transactions. For HTTP/2, status
verification is specified via `:status` pseudo header field verification using
the field verification mechanism described above.
### Body Verification
In a manner similar to field verification described above, a mechanism exists
to verify the body content of a request or response. To specify a rule for
verifying body content, a new `verify` node should be added under the `content`
node. The rules follow the same map syntax as described for field verification.
```YAML
proxy-request:
content:
verify: {value: test1, as: equal}
proxy-response:
content:
verify: {value: test2, as: contains}
```
The `value` node in the `verify` node can be ommited if the `data` node is used
to specify the content, since body content can get very long, and/or multi-lined.
However, `value` node has priority over the `data` node, meaning if there is a
`value` node, then the `data` will be ignored.
```YAML
proxy-request:
content:
encoding: plain
data: test1
verify: {as: equal}
proxy-response:
content:
encoding: plain
data: test2
verify: {as: contains}
```
Note that only one body verification node is needed, even if multiple `DATA` frames
are specified as described in [HEADERS and DATA frame](#headers-and-data-frame). The
verification node need to combine all the values specified for the `DATA` frames.
See the example below:
```YAML
client-request:
frames:
- HEADERS:
headers:
fields:
- [:method, POST]
- [:scheme, https]
- [:authority, example.data.com]
- [:path, /a/path]
- [Content-Type, text/html]
- [uuid, 1]
- DATA:
content:
encoding: plain
data: client_data_1
- DATA:
content:
encoding: plain
data: client_data_2
proxy-request:
content:
encoding: plain
data: client_data_1client_data_2
verify: {as: equal}
```
### Example Replay File
The sections leading up to this one have described each of the major components
of a YAML Proxy Verifier replay file. Putting these components together, the
following complete sample replay file demonstrates the description of the
replay of two sessions: one an HTTP/1.1 session, the second an HTTP/2 session.
Each session contains a single transaction with verification of certain parts
of the HTTP messages.
```YAML
meta:
version: '1.0'
sessions:
#
# First session: since there is no "protocol" node for this session,
# HTTP/1.1 over TCP (no TLS) is assumed.
#
- transactions:
#
# Direct the Proxy Verifier client to send a POST request with a body of
# 399 bytes.
#
- client-request:
method: POST
url: /pictures/flower.jpeg
version: '1.1'
headers:
fields:
- [ Host, www.example.com ]
- [ Content-Type, image/jpeg ]
- [ Content-Length, '399' ]
- [ uuid, first-request ]
# A "content" node is not needed if a Content-Length field is specified.
#
# Direct the Proxy Verifier server to verify that the request received from
# the proxy has a path in the request target that contains "flower.jpeg",
# has a path that is not prefixed with "JPEG" (case insensitively),
# and has the Content-Length field of any value.
#
proxy-request:
url:
- [ path, { value: flower.jpeg, as: contains } ]
- [ path, { value: JPEG, not: prefix, case: ignore } ]
headers:
fields:
- [ Content-Length, { value: '399', as: present } ]
#
# Direct the Proxy Verifier server to reply with a 200 OK response with a body
# of 3,432 bytes.
#
server-response:
status: 200
reason: OK
headers:
fields:
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, image/jpeg ]
- [ Transfer-Encoding, chunked ]
- [ Connection, keep-alive ]
# Unlike the request which contains a Content-Length, this response
# will require a "content" node to specify the size of the body.
# Otherwise Proxy Verifier has no way of knowing how large the response
# should be.
content:
size: 3432
#
# Direct the Proxy Verifier client to verify that it receives a 200 OK from
# the proxy with a `Transfer-Encoding: chunked` header field.
#
proxy-response:
status: 200
headers:
fields:
- [ Transfer-Encoding, { value: chunked, as: equal } ]
#
# For the second session, we use a protocol node to configure HTTP/2 using an
# SNI of # test_sni in the TLS handshake.
#
- protocol:
- name: http
version: 2
- name: tls
sni: test_sni
- name: tcp
- name: ip
transactions:
#
# Direct the Proxy Verifier client to send a POST request with a body of
# 399 bytes.
#
- client-request:
headers:
fields:
- [ :method, POST ]
- [ :scheme, https ]
- [ :authority, www.example.com ]
- [ :path, /pictures/flower.jpeg ]
- [ Content-Type, image/jpeg ]
- [ uuid, second-request ]
content:
size: 399
#
# Direct the Proxy Verifier server to verify that the request received from
# the proxy has a path pseudo header field that contains "flower.jpeg"
# and has a field "Content-Type: image/jpeg".
#
proxy-request:
url:
- [ path, { value: flower.jpeg, as: contains } ]
headers:
fields:
- [ :method, POST ]
- [ :scheme, https ]
- [ :authority, www.example.com ]
- [ :path, { value: flower.jpeg, as: contains } ]
- [ Content-Type, { value: image/jpeg, as: equal } ]
#
# Direct the Proxy Verifier server to reply with a 200 OK response with a body
# of 3,432 bytes.
#
server-response:
headers:
fields:
- [ :status, 200 ]
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, image/jpeg ]
content:
size: 3432
#
# Direct the Proxy Verifier client to verify that it receives a 200 OK from
# the proxy.
#
proxy-response:
status: 200
#
# For the third session, we demonstrate how body verification should be specified.
#
- protocol:
- name: http
version: 1.1
- name: tls
sni: test_sni
- name: tcp
- name: ip
version: 4
transactions:
#
# Direct the Proxy Verifier client to send a POST request with a body of
# 11 bytes.
#
- client-request:
method: POST
url: /a/path
version: '1.1'
headers:
fields:
- [ Host, example.data.com ]
- [ Content-Type, text/html ]
- [ Content-Length, '11' ]
- [ uuid, third-request ]
content:
encoding: plain
data: client_test
size: 11
#
# Direct the Proxy Verifier server to verify that the request received from
# the proxy has body content "client_test".
#
proxy-request:
content:
verify: { value: client_test, as: equal }
#
# Direct the Proxy Verifier server to reply with a 200 OK response with a body
# of 11 bytes.
#
server-response:
status: 200
reason: OK
headers:
fields:
- [ Content-Type, text/html ]
- [ Content-Length, '11' ]
content:
encoding: plain
data: |-
server
test
size: 11
#
# Direct the Proxy Verifier client to verify that the response received from
# the proxy has body content "server\ntest".
#
proxy-response:
content:
encoding: plain
data: |-
server
test
verify: { as: equal }
```
## Installing
### Prebuilt Binaries
Starting with the v2.2.0 release, statically linked binaries for Linux and Mac
are provided with the release in the
[Releases](https://github.com/yahoo/proxy-verifier/releases) page. If you do not
need your own customized build of Proxy Verifier, the easiest way to start
using it is to simply download the proxy-verifier `tar.gz` for the desired
release, untar it on the desired box, and copy the `verifier-client` and
`verifier-server` binaries to a convenient location from which to run them.
The Linux binaries should run on Ubuntu, Alma/CentOS/Fedora/RHEL, FreeBSD, and
other Linux flavors.
### Building from Source
These instructions describe how to build a copy of the Proxy Verifier project
on your local machine for development and testing purposes.
#### Prerequisites
Proxy Verifier is built using [SCons](https://scons.org). Scons is a Python
module, so installing it is as straightforward as installing any Python
package. A top-level
[Pipfile](https://github.com/yahoo/proxy-verifier/tree/master/Pipfile) is
provided to install Scons and its use is described and assumed in these
instructions, but it can also be installed using pip if preferred.
Scons will clone and build the dependent libraries using Automake. Thus
building will require the installation of the following system packages:
* git
* pipenv
* autoconf
* libtool
* pkg-config
For system-specific commands to install these packages (Ubuntu, CentOS, etc.),
one can view the
[Dockerfile](https://docs.docker.com/engine/reference/builder/) documents
provided under
[docker](https://github.com/yahoo/proxy-verifier/tree/master/docker). These
demonstrate, for each system, what commands are used to install these package
dependencies. Naturally, performing a `docker build` against these Dockerfiles
can also be used to create Docker images, containers from which builds can be
performed.
In addition to the above system package dependencies, Proxy Verifier utilizes
the following C++ libraries:
* [OpenSSL](https://www.openssl.org) is used to implement TLS encryption.
Proxy Verifier requires the version of OpenSSL that supports QUIC.
* [Nghttp2](https://nghttp2.org) is used for parsing HTTP/2 traffic.
* [ngtcp2](https://github.com/ngtcp2/ngtcp2) is used for parsing QUIC
traffic.
* [nghttp3](https://github.com/ngtcp2/nghttp3) is used for parsing HTTP/3
traffic.
* [yaml-cpp](https://github.com/jbeder/yaml-cpp) is used for parsing the YAML
replay files.
* [libswoc](https://github.com/SolidWallOfCode/libswoc) are a set of C++
library extensions to support string parsing, memory management, logging, and
other features.
*Note*: None of these libraries need to be explicitly installed before you
build. By default, Scons will fetch and build each of these libraries as a
part of building the project.
#### Build
Once the above-listed system packages (git, autoconf, etc.) are installed on
your system, you can build Proxy Verifier using Scons. This involves first
creating the Python virtual environment and then running the `scons` command
to build Proxy Verifier. Here is an example invocation:
```
# Install scons and any of its Python requirements. This only needs to be
# done once before the first invocation of scons.
#
# Note: for older RHEL/CentOS systems, you will have to souce the appropriate
# Python 3 enable script to initialize the correct Python 3 environment. For
# example:
# source /opt/rh/rh-python38/enable
pipenv install
# Now run scons to build proxy-verifier.
pipenv run scons -j4
```
This will build and install `verifier-client` and `verifier-server` in the
`bin/` directory at the root of the repository. `-j4` directs Scons to build
with 4 threads. Adjust according to the capabilities of your build system.
#### Using Prebuilt Libraries
As mentioned above, Scons will by default fetch the various library
dependencies (OpenSSL, Nghttp2, etc.), build, and manage those for you. If you
do not change the fetched source code for these libraries, they will not be
rebuilt after the first `scons` build invocation. This behavior is convenient
as it relieves the burden of fetching and building these libraries from the
developer. However, Scons will rescan the fetched source trees for these
libraries on every call of `scons` to inspect them for any changes. For
long-term development projects, a developer may find it more efficient to build
these libraries externally and relieve Scons from managing them. To
conveniently support this, the
[build_library_dependencies.sh](https://github.com/yahoo/proxy-verifier/blob/master/tools/build_library_dependencies.sh)
script is provided to build these libraries. To build and install the
libraries, run that script, passing as an argument the desired install location
for the various libraries. Then point Scons to those libraries using various
`--with` directives.
Here's an example invocation of `scons` along with the use of the library build
script:
```
# Alter this to your desired library location.
http3_libs_dir=${HOME}/src/http3_libs
bash ./tools/build_http3_dependencies.sh ${http3_libs_dir}
pipenv install
pipenv run scons \
-j4 \
--with-ssl=${http3_libs_dir}/openssl \
--with-nghttp2=${http3_libs_dir}/nghttp2 \
--with-ngtcp2=${http3_libs_dir}/ngtcp2 \
--with-nghttp3=${http3_libs_dir}/nghttp3
```
The [Dockerfile](https://github.com/yahoo/proxy-verifier/tree/master/docker)
documents run this build script, installing the HTTP packages in `/opt`.
Therefore, if you are developing in a container made from images generated from
these Dockerfile documents, you can use the following `scons` command to build
Proxy Verifier:
```
pipenv install
pipenv run scons \
-j4 \
--with-ssl=/opt/openssl \
--with-nghttp2=/opt/nghttp2 \
--with-ngtcp2=/opt/ngtcp2 \
--with-nghttp3=/opt/nghttp3
```
As a further convenience, if these libraries (`openssl`, `nghttp2`, `ngtcp2`,
and `nghttp3`, with those exact names) exist under a single directory, such as
is the case with images built from the provided
[Dockerfile](https://github.com/yahoo/proxy-verifier/tree/master/docker)
documemnts, then you can specify the location of these libraries with a single
`--with-libs` argument. Thus the previous command can be expressed like so:
```
pipenv install
pipenv run scons -j4 --with-libs=/opt
```
#### ASan Instrumentation
The local Sconstruct file is configured to take an optional `--enable-asan`
parameter. If this is passed to the `scons` build line then the Proxy Verifier
objects and binaries will be compiled and linked with the flags that instrument
them for [AddressSanatizer](https://clang.llvm.org/docs/AddressSanitizer.html).
This assumes that the system has the AddressSanatizer library installed on the
system. Thus the above invocation would look like the following to compile it
with AddressSanitizer instrumentation:
```
pipenv install
pipenv run scons \
-j4 \
--with-ssl=/path/to/openssl \
--with-nghttp2=/path/to/nghttp2 \
--with-ngtcp2=/path/to/ngtcp2 \
--with-nghttp3=/path/to/nghttp3 \
--enable-asan
```
#### Debug Build
By default, Scons will build the Proxy Verifier project in `release` mode. This
means that the binaries will compiled with optimization. If an unoptimized
debug build is desired, then pass the `--cfg=debug` option to `scons`:
```
pipenv run scons -j4 --cfg=debug
```
#### Statically Link
The current Scons configuration dynamically links the binaries with the various
OpenSSL and HTTP build libraries. This is fine for local testing and execution,
but can be inconvenient when copying the binaries to other machines. Ideally
the
[Sconstruct](https://github.com/yahoo/proxy-verifier/blob/master/Sconstruct)
file can be updated to support an option to link the binaries statically. This
currently does not exist and is not trivial to create. Future updates to
`scons-parts` may help with this. As a current stopgap measure,
[tools/build_static](https://github.com/yahoo/proxy-verifier/blob/master/tools/build_static)
is provided which automatically links the Verifier binaries statically. It is
run from the root directory of your repository like so:
```
./tools/build_static
```
By default this builds Proxy Verifier with the following invocation:
```
pipenv run scons -j$(nproc)
```
Any arguments passed to `build_static` will be passed through to the scons
command. Thus, if you desire to build Proxy Verifier with `--with-libs=/opt`,
run the script like so:
```
./tools/build_static --with-libs=/opt`
```
### Running the Tests
#### Unit Tests
To build and run the unit tests, use the `run_utest` Scons target (this assumes
you previously ran `pipenv install`, see above):
```
pipenv run scons \
-j4 \
--with-ssl=/path/to/openssl \
--with-nghttp2=/path/to/nghttp2 \
--with-ngtcp2=/path/to/ngtcp2 \
--with-nghttp3=/path/to/nghttp3 \
run_utest::
```
#### Gold Tests
Proxy Verifier ships with a set of automated end to end tests written using the
[AuTest](https://bitbucket.org/autestsuite/reusable-gold-testing-system/src/master/)
framework. To run them, simply run the `autest.sh` script:
```
cd test/autests
./autest.sh
```
When doing development for which a particular AuTest is relevant, the `-f`
option can be used to run just a particular test (or set of tests, specified
in a space-separated list). For instance, the following invocation runs
just the http and https tests:
```
./autest.sh -f http https
```
AuTest supports a variety of other options. Run `./autest.sh --help` to get a
quick description of the various command-line options. See the [AuTest
Documentation](https://autestsuite.bitbucket.io) for further details about the
framework.
**A note for macOS**: The Python virtual environment for these gold tests
requires the [cryptograpy](https://github.com/pyca/cryptography) package as a
dependency of the [pyOpenSSL](https://www.pyopenssl.org/en/stable/) package.
Pipenv will install this automatically, but the installation of the
`cryptography` package will require compiling certain c files against OpenSSL.
macOS has its own SSL libraries which brew's version of OpenSSL does not
replace, for understandable reasons. The building of `cryptography` will
fail against the system's SSL libraries. To point the build to brew's OpenSSL
libraries, the `autest.sh` script exports the following variables before
running `pipenv install`:
```
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"
```
Thus if you stick with using the `autest.sh` script you do not need to worry
about this. But if you install pipenv by hand rather than through the
`autest.sh` script on macOS, then keep this in mind and export those variables
before running `pipenv install`.
## Usage
This section describes how to run the Proxy Verifier client and server at
the command line.
### Required Arguments
At a high level, Proxy Verifier is run in the following manner:
1. Run the verifier-server with the set of HTTP and HTTPS ports to listen on
configured though the command line. The directory containing the replay file
is also configured through a command line argument.
1. Configure and run the proxy to listen on a set of HTTP and HTTPS ports and
to proxy those connections to the listening verifier-server ports.
1. Run the verifier-client with the sets of HTTP and HTTPS ports on which to
connect configured though the command line. The directory containing the
replay file is also configured through a command line argument.
Here's an example invocation of the verifier-server, configuring it to listen on
localhost port 8080 for HTTP connections and localhost port 4443 for HTTPS
connections:
```
verifier-server \
run \
--listen-http 127.0.0.1:8080 \
--listen-https 127.0.0.1:4443 \
--server-cert \
--ca-certs \
```
Here's an example invocation of the verifier-client, configuring it to connect to
the proxy which has been configured to listen on localhost port 8081 for HTTP
connections and localhost port 4444 for HTTPS connections:
```
verifier-client \
run \
--client-cert \
--ca-certs \
--connect-http 127.0.0.1:8081 \
--connect-https 127.0.0.1:4444 \
```
With these two invocations, the verifier-client and verifier-server will replay the
sessions and transactions in `` and perform any field
verification described therein.
On the server either `--listen-http` or `--listen-https` or both must be
provided. That is, for example, if you are only testing HTTPS traffic, you may
only specify `--listen-https`. The same is true on the client: either
`--connect-http` or `--connect-https` or both must be provided. These address
arguments take a comma-separated list of address/port pairs to specify multiple
listening or connecting sockets. The processing of these arguments
automatically detects any IPv6 addresses if provided. The client's processing
of `--connect-http` and `--connect-https` arguments will resolve fully
qualified domain names.
Note that the `--client-cert` and `--server-cert` both take either a
certificate file containing the public and private key or a directory
containing pem and key files. Similarly, the `--ca-certs` takes either a file
containing one or more certificates or a directory with separate certificate
files. For convenience, the
[test/keys](https://github.com/bneradt/proxy-verifier/tree/expand_readme/test/keys)
directory contains key files which can be used for testing. These certificate
arguments are only required if HTTPS traffic will be replayed.
### Optional Arguments
#### --format \
Each transaction has to be uniquely identifiable by the client and server in a
way that is consistent across both replay file parsing and traffic replay
processing. Whatever attributes we use from the messages to uniquely identify
transactions is called the "key" for the dataset. The ability to uniquely
identify these messages is important for at least the following reasons:
* When the Verifier server receives a request, it has to know from which of the
set of parsed transactions it should generate a response. At the time of
processing an incoming message, all it has to go on is the request header
line and the request header fields. From these, it has to be able to identify
which of the potentially thousands of parsed transactions from the replay input
files it should generate a response.
* When the client and server perform field verification, they need to know what
particular verification rules specified in the replay files should be applied
to the given incoming message.
* If the client and server are processing many transactions, generic log
messages could be near useless if there was not a way for the logs to
identify individual transactions to the user somehow.
By default the Verifier client and server both expect a `uuid` header field
value to function as the key.
If the user would like to use other attributes as a key, they can specify
something else via the `--format` argument. The format argument currently
supports generating a key on arbitrary field values and the `URL` of the
request. Some example `--format` expressions include:
* `--format "{field.uuid}"`: This is the default key format. It treats the UUID
header field value as the transaction key.
* `--format "{url}"`: Treat the request `URL` as the key.
* `--format "{field.host}"`: Treat the `Host` header field value as the key.
* `--format "{field.host}/{url}"`: Treat the combination of the `Host` header
field and the request `URL` as the key.
#### --keys \
`--keys` can be passed to the verifier-client to specify a subset of keys from
the replay file to run. Only the transactions from the space-separated list of
keys will be replayed. For example, the following invocation will only run the
transactions with keys whose values are 3 and 5:
```
verifier-client \
run \
\
127.0.0.1:8082 \
127.0.0.1:4443 \
--keys 3 5
```
This is a client-side only option.
#### --verbose
Proxy Verifier has four levels of verbosity that it can run with:
| Verbosity | Description |
| --------- | ----------- |
| error | Transactions either failed to run or failed verification. |
| warning | A non-failing problem occurred but something is likely to go wrong in the future. |
| info | High level test execution information. |
| diag | Low level debug information. |
Each level implies the ones above it. Thus, if a user specifies a verbosity
level of `warning`, then both warning and error messages are reported.
By default, Proxy Verifier runs at `info` verbosity, only producing summary
output by both the client and the server along with any warnings and errors it
found. This can be tweaked via the `--verbose` flag. Here's an example of requesting
the most verbose level of logging (`diag`):
```
verifier-client \
run \
\
127.0.0.1:8082 \
127.0.0.1:4443 \
--verbose diag
```
#### --interface \
Initiate connections from the specified interface, such as eth0:1.
This is a client-side only option.
#### --no-proxy
As explained above, replay files contain traffic information for both client to
proxy traffic and proxy to server traffic. Under certain circumstances it may
be helpful to run the Verifier client directly against the Verifier server.
This can be useful while developing Proxy Verifier itself, for example,
allowing the developer to do some limited testing without requiring the setup
of a test proxy.
To support this, the Verifier client has the `--no-proxy` option. If this
option is used, then the client has its expectations configured such that it
assumes it is run against the Verifier server rather than a proxy. Effectively
this means that instead of trying to run the client to proxy traffic, it will
instead act as the proxy host for the Verifier server and will run the proxy to
server traffic. Concretely, this means that the Verifier client will replay the
`proxy-request` and `proxy-response` nodes rather than the `client-request` and
`client-response` nodes.
This is a client-side only option.
#### --strict
Generally, very little about the replayed traffic is verified except what is
explicitly specified via field verification (see above). This is by design,
allowing the user to replay traffic with only the requested content being
verified. In high-volume cases, such as situations where Proxy Verifier is
being used to scale test the proxy, traffic verification may be considered
unimportant or even unnecessarily noisy. If, however, the user wants every
field to be verified regardless of specification, then the `--strict` option
can be passed to either or both the Proxy Verifier client and server to report
any verification issues against every field specified in the replay file.
#### --rate \
By default, the client will replay the session and transactions in the replay
files as fast as possible. If the user desires to configure the client to
replay the transactions at a particular rate, they can provide the `--rate`
argument. This argument takes the number of requests per second the client will
attempt to send requests at.
Note session and transaction timing data can be specified in the replay files.
These are provided via `start-time` nodes for each `session` and `transaction`.
`start-time` takes as a value the number of nanoseconds since Unix epoch (or
whatever other time of reference observed by all `start-time` nodes in the set
of replay files being run) associated for that session or transaction. With
this timing information, if `--rate` is provided, Proxy Verifier simply scales
the relative time deltas between sessions and transactions that appropriately
achieves the desired transaction rate. [Traffic
Dump](https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/traffic_dump.en.html)
records such timing information when it writes replay files. In the absence of
`start-time` nodes, Proxy Verifier will attempt to apply an appropriate uniform
delay across the sessions and transactions to achieve the specified `--rate`
value.
This is a client-side only option.
#### --repeat \
By default, the client will replay all the transactions once in the set of
input replay files. If the user would like the client to automatically repeat
this set a number of times, they can provide the `--repeat` option. The
argument takes the number of times the client should replay the entire dataset.
This is a client-side only option.
#### --run-continuously
Run the set of sessions in the input replay files in an infinite loop. Passing
this option can be thought of as `--repeat` with a value of infinity.
This is a client-side only option.
#### --thread-limit \
Each connection, corresponding to a `session` in a replay file, is dispatched
on the client in parallel. Likewise, each accepted connection on the server is
handled in parallel. Each of these sessions is handled via a single thread of
execution. By default, Proxy Verifier limits the number of threads for handling
these connections to 2,000. This limit can be changed via the `--thread-limit`
option. Setting a value of 1 on the client will effectively cause sessions
to be replayed in serial.
#### --qlog-dir \
Proxy Verifier supports logging of replayed QUIC traffic information conformant
to the qlog format. If the `--qlog-dir` option is provided, then qlog files for all
replayed QUIC traffic will be written into the specified directory. qlog
diagnostic logging is disabled by default.
#### --tls-secrets-log-file \
To facilitate debugging, Proxy Verifier supports logging TLS keys for encrypted
replayed traffic. If this option is used, TLS key logging will be appended to
the specified filename. This file can then be provided to protocol analyzers
such as Wireshark to decrypt the traffic. TLS key logging is disabled by
default.
#### --send-buffer-size \
Configure a `SO_SNDBUF` value to set on the server socket. This can be a
helpful parameter to tune when dealing with the replay of large response
bodies. If this optional value is not set, then the `SO_SNDBUF` option is not
set on the socket and, thus, the system default socket buffer size will be
used.
This is a server-side only option.
#### --poll-timeout \
When Proxy Verifier performs read and write operations, it does so using
non-blocking sockets with a timeout. By default, this timeout is set to 5
seconds (5,000 milliseconds). This optional argument provides a way to specify a
different timeout in milliseconds for these operations.
## Tools
This section describes how to use some of the scripts under the [tools](tools) directory.
### Replay Gen [replay_gen.py](tools/replay_gen.py)
This tool is used to generate mock replay files for easy testing.
Listed below are the available arguments for this script.
#### -n,--number \
Number of total transactions.
#### -tl,--trans-lower \
The lower limit of transactions per session.
#### -tu,--trans-upper \
The upper limit of transactions per session.
#### -sl,--sess-lower \
The lower limit of sessions per file.
#### -su,--sess-upper \
The upper limit of sessions per file.
#### -tp,--trans-protocols \
A comma separated list of protocols that are allowed to be generated.
Available options are: http, tls, h2, all.
#### -u,--url-file \
Path to a file with the list of URLs that can be used.
The URL list file can be acquired by running the [Remap Config to URL List]() script described below.
#### -o,--output \
Path to a directory where the replay files are generated.
#### -p,--prefix \
Prefix for the replay file names.
#### -j,--out-json
Dump replay files in JSON format. By default replay files will be formatted as YAML.
### Remap Config to URL List [remap_config_to_url_list.py](tools/remap_config_to_url_list.py)
This tool converts a `remap.config` file to a URL list file that is used by [Replay Gen](#replay-gen-replay_genpytoolsreplay_genpy)
Listed below are the available arguments for this script.
#### -o,--output \
A filename to which to write the list of URLs. Defaults to `stdout`.
#### --no-ip
Ignore ip address (in the "replacement" section) in the `remap.config` file.
## Contribute
Please refer to [CONTRIBUTING](CONTRIBUTING.md) for information about how to get involved. We welcome issues, questions, and pull requests.
## License
This project is licensed under the terms of the Apache 2.0 open source license.
Please refer to [LICENSE](LICENSE) for the full terms.