HTTP 409: Cannot delete an non-empty container

[Brian]

Has anyone ever encountered this error during the dispose phase of COSBench?
I see this running either 2.1.0.GA and/or 0.3.3.0 b2.

2013-10-21 22:13:37,716 [ERROR] com.intel.cosbench.driver.operator.AbstractOperator - fail to perform clean operation
com.intel.cosbench.api.storage.StorageException: HTTP/1.1 409 Conflict
        at com.intel.cosbench.api.swift.SwiftStorage.DELETE(SwiftStorage.java:129)
        at com.intel.cosbench.driver.operator.Cleaner.doDispose(Cleaner.java:76)
        at com.intel.cosbench.driver.operator.Cleaner.operate(Cleaner.java:53)
        at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:30)
        at com.intel.cosbench.driver.operator.Disposer.operate(Disposer.java:1)
        at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:133)
        at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:125)
        at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:86)
        at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:26)
        at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
Caused by: com.intel.cosbench.client.swift.SwiftConflictException: cannot delete an non-empty container
        at com.intel.cosbench.client.swift.SwiftClient.deleteContainer(SwiftClient.java:147)
        at com.intel.cosbench.api.swift.SwiftStorage.DELETE(SwiftStorage.java:122)
        ... 14 more


OS: CentOS 6.4
Config:
<workload name="demo" description="demo benchmark">
<auth type="keystone" config="username=....;password=....;tenname=....;url=http://....:5000/v2.0;service=swift" />
<storage type="swift" />
 
  <workflow>
   
    <workstage name="init">
      <work type="init" workers="1" config="containers=r(1,32)" />
    </workstage>
   
    <workstage name="prepare">
      <work type="prepare" workers="1" config="containers=r(1,32);objects=r(1,50);sizes=c(64)KB" />
    </workstage>
   
    <workstage name="main">
      <work name="main" workers="8" rampup="60" runtime="300">
        <operation type="read" ratio="80" config="containers=u(1,32);objects=u(1,50)" />
        <operation type="write" ratio="20" config="containers=u(1,32);objects=u(51,100);sizes=c(64)KB" />
      </work>
    </workstage>
   
    <workstage name="cleanup">
      <work type="cleanup" workers="1" config="containers=r(1,32);objects=r(1,100)" />
    </workstage>
   
    <workstage name="dispose">
      <work type="dispose" workers="1" config="containers=r(1,32)" />
    </workstage>
  </workflow>
</workload>

Thanks,
Brian

[vills]

In openstack Swift, you should delete all objects in container before delete container itself.

Looks like cosbench didn't delete some object(s) in that container before.

[ywang19]

As swift is eventually consistent system, there is time window from object deletion to object invisible, which potentially causes container deletion failure. one issue is already filled to track it: https://github.com/intel-cloud/cosbench/issues/48.

[Brian]

Just to follow-up with this issue, when my file sizes are 'small', ie. 4K - 10M, and I am doing Writes, I always see this issue.  Do not see this with pure Reads. When I changed the filesize to 50MB, I did not see this issue. I'm guessing since the files are larger that Swift takes longer in processing them. It is then able to delete the objects before attempting to delete the containers.

Also, we made sysctl changes and limits.conf changes and did not see this issue with our 4K - 10MB files.

[ywang19]

I can't give the exact reason of the difference at small and large objects, the source may be from background processes like object replicator or container updater. But the error 409 at dispose stage is an known issue, and one workaround is to create more than one "cleanup" stage to ensure all objects are deleted before container deletion in dispose stage as following:

    <workstage name="cleanup-1">
      <work type="cleanup" workers="1" config="containers=r(1,32);objects=r(1,100)" />
    </workstage>
    <workstage name="cleanup-2">
      <work type="cleanup" workers="1" config="containers=r(1,32);objects=r(1,100)" />
    </workstage>

-yaguang