Understanding endpoint syntax for s3 workloads

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Understanding endpoint syntax for s3 workloads

Bill Owen
We are running cosbench to compare performance of swift object storage with native swift interface vs swift3 s3 emulation.

When endpoint is defined as an IP address, I see the expected behavior.  

For example:
  <storage type="s3" config="accesskey=s3accesskey;
                             secretkey=s3secretkey;
                             proxyhost=;
                             proxyport=;
                             endpoint=<a href="http://1.2.3.4:8080&quot;/&gt;">http://1.2.3.4:8080"/>

Requests go to:
http://1.2.3.4:8080/bucketname

When the endpoint is defined as a host name, fqn or just host name, it seems that virtual host feature is automatically applied and requests are sent to "bucketname.endpoint".  

For example,
  <storage type="s3" config="accesskey=s3accesskey;
                             secretkey=s3secretkey;
                             proxyhost=;
                             proxyport=;
                             endpoint=<a href="http://hostname:8080&quot;/&gt;">http://hostname:8080"/>

Requests go to:
http://bucketname.hostname:8080/

Amazon docs describe "Amazon S3 supports both virtual-hosted–style and path-style URLs to access a bucket.", but I don't see how to for the use of path-style url.
See:  http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html

Is there a way to force "path-style url" to be used so that hostname is same as what is specified in endpoint, and bucket is appended to url in normal REST approach?

The real problem we are trying to solve is to allow a cosbench configuration with N drivers to send load to swift configuration with N proxy servers without using a load balancer.  The way we do this with swift interface is to define the swift endpoints as "virtualhost" and then on each driver add virtualhost to /etc/hosts with each driver resolving to one proxy server IP, one to one.  

We are not able to do this with S3 because of this problem.
Reply | Threaded
Open this post in threaded view
|

Re: Understanding endpoint syntax for s3 workloads

ywang19
Administrator
Hi Bill,

there is one configuration item for called "path_style_access" for "config", which is exactly for the case, by default it's "false", you could set it to "true" to enable path style access in S3 like

<storage type="s3" config="path_style_access=true; accesskey=s3accesskey;
                             secretkey=s3secretkey;
                             proxyhost=;
                             proxyport=;
                             endpoint=<a href="<a href="http://hostname:8080&amp;quot;/&amp;gt;&quot;">http://hostname:8080&quot;/&gt;">http://hostname:8080"/>


-yaugang
Reply | Threaded
Open this post in threaded view
|

Re: Understanding endpoint syntax for s3 workloads

Bill Owen
Hi Yaugang,
That solved my problem.  Thanks!

Is path_style_access configuration option going to be added to the user doc?
Reply | Threaded
Open this post in threaded view
|

Re: Understanding endpoint syntax for s3 workloads

ywang19
Administrator
Hi Bill,

Glad to help your problem is resolved:-). Yes, the parameter should be on user doc, sorry for my laziness.


-yaguang
Reply | Threaded
Open this post in threaded view
|

RE: Understanding endpoint syntax for s3 workloads

ywang19
Administrator

The user guide was updated to cover a few already existed parameters on branch 0.4.2.0:

-          Add “transfer_rate” parameter for swift storage, then “max_connections” and “path_style_access” parameters for S3 storage in section 4.5.3

 

-yaguang

From: ywang19 [via COSBench] [mailto:ml-node+[hidden email]]
Sent: Wednesday, September 02, 2015 4:15 PM
To: Wang, Yaguang
Subject: Re: Understanding endpoint syntax for s3 workloads

 

Hi Bill,

Glad to help your problem is resolved:-). Yes, the parameter should be on user doc, sorry for my laziness.


-yaguang


If you reply to this email, your message will be added to the discussion below:

http://cosbench.1094679.n5.nabble.com/Understanding-endpoint-syntax-for-s3-workloads-tp417p420.html

To start a new topic under cosbench-user, email [hidden email]
To unsubscribe from COSBench, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

RE: Understanding endpoint syntax for s3 workloads

Bill Owen

Thanks Yaguang!


Bill Owen
[hidden email]
GPFS and OpenStack
520-799-4829


Inactive hide details for "ywang19 [via COSBench]" ---09/08/2015 07:00:33 PM---The user guide was updated to cover a few alread"ywang19 [via COSBench]" ---09/08/2015 07:00:33 PM---The user guide was updated to cover a few already existed parameters on branch 0.4.2.0: - A

From: "ywang19 [via COSBench]" <[hidden email]>
To: Bill Owen/Tucson/IBM@IBMUS
Date: 09/08/2015 07:00 PM
Subject: RE: Understanding endpoint syntax for s3 workloads





The user guide was updated to cover a few already existed parameters on branch 0.4.2.0:

- Add “transfer_rate” parameter for swift storage, then “max_connections” and “path_style_access” parameters for S3 storage in section 4.5.3

-yaguang

From: ywang19 [via COSBench] [[hidden email][hidden email]]
Sent:
Wednesday, September 02, 2015 4:15 PM
To:
Wang, Yaguang
Subject:
Re: Understanding endpoint syntax for s3 workloads

Hi Bill,

Glad to help your problem is resolved:-). Yes, the parameter should be on user doc, sorry for my laziness.


-yaguang




If you reply to this email, your message will be added to the discussion below:

http://cosbench.1094679.n5.nabble.com/Understanding-endpoint-syntax-for-s3-workloads-tp417p420.html
To start a new topic under cosbench-user, email [hidden email]
To unsubscribe from COSBench,
click here.
NAML





If you reply to this email, your message will be added to the discussion below:
http://cosbench.1094679.n5.nabble.com/Understanding-endpoint-syntax-for-s3-workloads-tp417p421.html
To unsubscribe from Understanding endpoint syntax for s3 workloads, click here.
NAML