Unable to execute HTTP request: Read timed out

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Unable to execute HTTP request: Read timed out

karan singh
Hello Guys

Your pointers on this problem will be much appreciated.

Problem while GET objects using S3 , backend storage Ceph Jewel
2 RGWs , 8 Clients , 128 workers/client , object sizes 1,32,64,128M

GET operation on 1M object work , however when it comes to 32M object sizes it fails after sometime. i.e COSBench reads some 32M object and then fails with the below error

- You can see below , the objects / containers which COSBench throws read time out errors are present on storage and can be viewed using CLI commands.

- If you see my workload file i have timeout=100000 set which is a large timeout
- Is there any other placed to define timeout parameter ?


2016-08-09 00:41:49,550 [INFO] [NoneStorage] - performing GET at /mycontainers93/myobjects234
2016-08-09 00:41:49,728 [INFO] [NoneStorage] - performing GET at /mycontainers98/myobjects134
2016-08-09 00:41:49,895 [INFO] [NoneStorage] - performing GET at /mycontainers88/myobjects455
2016-08-09 00:41:49,919 [WARN] [S3Storage] - below exception encountered when retrieving object myobjects289 at bucket mycontainers64: Unable to execute HTTP request: Read timed out
2016-08-09 00:41:49,920 [DEBUG] [S3Storage] - initialize S3 client with storage config: {path_style_access=false, endpoint=http://10.5.13.140:80, max_connections=200, proxyport=, accesskey=S3user1, secretkey=S3user1key, logging=true, timeout=1000000, proxyhost=}
2016-08-09 00:41:49,921 [DEBUG] [S3Storage] - S3 client has been initialized
2016-08-09 00:41:50,058 [INFO] [NoneStorage] - performing GET at /mycontainers57/myobjects357
2016-08-09 00:41:50,355 [INFO] [NoneStorage] - performing GET at /mycontainers75/myobjects383
2016-08-09 00:41:50,504 [INFO] [NoneStorage] - performing GET at /mycontainers92/myobjects301
2016-08-09 00:41:50,515 [INFO] [NoneStorage] - performing GET at /mycontainers67/myobjects116
2016-08-09 00:41:50,652 [INFO] [NoneStorage] - performing GET at /mycontainers56/myobjects369
2016-08-09 00:41:50,858 [INFO] [NoneStorage] - performing GET at /mycontainers58/myobjects175
2016-08-09 00:41:51,111 [INFO] [NoneStorage] - performing GET at /mycontainers100/myobjects344
2016-08-09 00:41:51,249 [INFO] [NoneStorage] - performing GET at /mycontainers59/myobjects85
2016-08-09 00:41:51,309 [WARN] [S3Storage] - below exception encountered when retrieving object myobjects497 at bucket mycontainers86: Unable to execute HTTP request: Read timed out
2016-08-09 00:41:51,310 [DEBUG] [S3Storage] - initialize S3 client with storage config: {path_style_access=false, endpoint=http://10.5.13.140:80, max_connections=200, proxyport=, accesskey=S3user1, secretkey=S3user1key, logging=true, timeout=1000000, proxyhost=}
2016-08-09 00:41:51,311 [DEBUG] [S3Storage] - S3 client has been initialized
2016-08-09 00:41:51,315 [INFO] [NoneStorage] - performing GET at /mycontainers51/myobjects294
2016-08-09 00:41:51,847 [INFO] [NoneStorage] - performing GET at /mycontainers79/myobjects215
2016-08-09 00:41:51,875 [INFO] [NoneStorage] - performing GET at /mycontainers83/myobjects237
2016-08-09 00:41:51,996 [INFO] [NoneStorage] - performing GET at /mycontainers95/myobjects111

Full logs here :
----------------
https://gist.githubusercontent.com/ksingh7/9c33f3b9922885736452d49e242f902f/raw/b940908f0d5a48486008ea4a0af18a8006e1c890/workload-logs-cosbench

Containers / objects are available from CLI:
--------------------------------------------------

[root@ceph-rgw1 ~]# swift  -A http://ceph-rgw1:80/auth/1.0 -U user1:swift -K 'Swiftuser1key' list mycontainers64 | grep -i myobjects289
myobjects289
[root@ceph-rgw1 ~]# swift  -A http://ceph-rgw1:80/auth/1.0 -U user1:swift -K 'Swiftuser1key' list mycontainers86 | grep -i myobjects497
myobjects497
[root@ceph-rgw1 ~]#


Workload file:
---------------
https://paste.fedoraproject.org/404568/6940101/raw/

Reply | Threaded
Open this post in threaded view
|

Re: Unable to execute HTTP request: Read timed out

karan singh
If you see FULL logs link i have added above , i am getting some AmazonClient errors as well

2016-08-09 00:40:44,806 [INFO] [NoneStorage] - performing GET at /mycontainers83/myobjects494
2016-08-09 00:40:44,934 [ERROR] [AbstractOperator] - worker 4 fail to perform operation mycontainers77/myobjects14
com.intel.cosbench.api.storage.StorageException: AmazonS3Exception: Status Code: 503, AWS Service: Amazon S3, AWS Request ID: null, AWS Error Code: null, AWS Error Message: null, S3 Extended Request ID: null
        at com.intel.cosbench.api.S3Stor.S3Storage.getObject(S3Storage.java:125)
        at com.intel.cosbench.driver.operator.Reader.doRead(Reader.java:89)
        at com.intel.cosbench.driver.operator.Reader.operate(Reader.java:69)
        at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:76)
        at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:197)
        at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:177)
        at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:134)
        at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:44)
        at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: AmazonS3Exception: Status Code: 503, AWS Service: Amazon S3, AWS Request ID: null, AWS Error Code: null, AWS Error Message: null, S3 Extended Request ID: null
        at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2974)
        at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:911)
        at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:801)
        at com.intel.cosbench.api.S3Stor.S3Storage.getObject(S3Storage.java:120)
        ... 12 more
2016-08-09 00:40:44,934 [INFO] [NoneStorage] - performing GET at /mycontainers96/myobjects14
2016-08-09 00:40:45,090 [INFO] [NoneStorage] - performing GET at /mycontainers97/myobjects134