Contributing to cosbench

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Contributing to cosbench

balaji

I work for Cloudian. We make a S3 compatible object store.  I would like to propose/make 2 changes to cosbench-s3


1) Change createObject in S3Storage to use TransferManager, instead of straight putObject. TransferManager will use MultiPart uploads for large objects (>16M by default).

2) We currently have to use haproxy to run cosbench against Cloudian. I also modified S3Storage to read a set of endpoint host:port from a properties file and round-robin between hosts. This is mostly nice to have. Changes are at https://github.com/balaji-g/cosbench/commit/b61c138f5020185a57e073e6d8c9f916197d6cb1

What do you think? I am also open to creating cosbench-cloudian driver if you don't want to modify cosbench-s3.
Reply | Threaded
Open this post in threaded view
|

Re: Contributing to cosbench

ywang19
Administrator
hi,

It is ok for me to use transfer manager in s3storage. But I don't understand why you need haproxy and round-robin properties. Actually, you could define multiple main stages, each has its own auth/storage section, where you could use different host:port.

-yaguang

在 2016年11月2日,下午8:44,balaji [via COSBench] <[hidden email]> 写道:


I work for Cloudian. We make a S3 compatible object store.  I would like to propose/make 2 changes to cosbench-s3


1) Change createObject in S3Storage to use TransferManager, instead of straight putObject. TransferManager will use MultiPart uploads for large objects (>16M by default).

2) We currently have to use haproxy to run cosbench against Cloudian. I also modified S3Storage to read a set of endpoint host:port from a properties file and round-robin between hosts. This is mostly nice to have. Changes are at https://github.com/balaji-g/cosbench/commit/b61c138f5020185a57e073e6d8c9f916197d6cb1

What do you think? I am also open to creating cosbench-cloudian driver if you don't want to modify cosbench-s3.


If you reply to this email, your message will be added to the discussion below:
http://cosbench.1094679.n5.nabble.com/Contributing-to-cosbench-tp530.html
To start a new topic under cosbench-developer, email [hidden email]
To unsubscribe from COSBench, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: Contributing to cosbench

balaji
We have a N node cluster and to balance load across all nodes, haproxy is used. We are trying to do some benchmarking with large objects, 16M, 32M, 64M, and I am trying to eliminate all the intermediate hops between cosbench and my cluster. The change I put in, the Host header is set to whatever was configured in the xml file, but when I make the actual connection, I use the ip addr. Does that make sense? Pls let me know tf there is any other way to bypass haproxy/LB and send requests directly to nodes.
Reply | Threaded
Open this post in threaded view
|

Re: Contributing to cosbench

balaji
In reply to this post by ywang19
figured out how to use cosbench with multiple main stages and different host:port.

thanks