Benchmark Factory Forum

jhieb
Follow / 17 Nov 2017 at 10:13pm / Benchmark Factory
Latest post by on 28 Nov 2017 at 4:33pm

How to link a job/jobname or GUID to the corresponding test run, userload...

Expand content
I've read Kevin Dalton's blog posts which were good (Hoping that he explains this issue in part 3 ): http://www.toadworld.com/products/benchmark-factory/b/weblog/archive/2017/05/15/benchmark-factory...

I've read Kevin Dalton's blog posts which were good (Hoping that he explains this issue in part 3 Smile ):
http://www.toadworld.com/products/benchmark-factory/b/weblog/archive/2017/05/15/benchmark-factory-rest-api-part-1
http://www.toadworld.com/products/benchmark-factory/b/weblog/archive/2017/05/15/benchmark-factory-rest-api-part-2

The issue i'm running into now is how do I link the given testrun route to a job name?  Currently in the testrun information given back it has like the subtest name, SQL server connection profile, some IDs but unless i'm missing something there isn't a connection to a job without delving into the subkeys of the job JSON object?


Test run info that i need to capture (I use the run id from this to pull the following results information) Also note that the testid relates to the test object for the TPC-C transaction mix for that job:

C:\Users\Administrator\git\vmwareSysbenchBMFAutomation [master]> (Invoke-RestMethod -Uri localhost:30110/.../testruns)[0]


classname : CTestRunHistory
testid : 2
testversion : 9
testname : TPC-C Transaction Mix
runid : 12
profileid : 11
profilename : MSSQL_10.108.4.77_ODBC
comment :
statusid : 1
starttime : Wed Nov 15 14:14:24 2017
endtime : Wed Nov 15 16:44:39 2017
elapsedtime : 02 hour(s), 30 minute(s), 15 second(s)

The Job that this result came from:

C:\Users\Administrator\git\vmwareSysbenchBMFAutomation [master]> invoke-restmethod -uri localhost:30110/.../TPC-C_BMF_JOB_1


classname : CJob
name : TPC-C_BMF_JOB_1
comment :
connectionname : MSSQL_10.108.4.77_ODBC
emailoperator : 0
operatoraddress :
emailpager : 0
pageraddress :
sendnet : 0
netsendname :
realtimereportinterval : 5
realtimereportcounters :
expanded : 1
stoponerror : 1
globalvariables : {}
tests : {@{classname=CBenchmarkObjectsTest; benchmarkid=10; create_objects=1; always_delete_before_create=0; benchmark_properties=; run_on_agents=1; virtualusers_for_table=200;
parallel_index_create=1; delete_after_run=0; reload_from_backup=0; refresh_using_inserts=0; name=Create Objects for TPC-C; testid=-1; guid={728F120F-4F6D-4E94-879E-0E1C64E4FFEB};
options=4096; enable_latency_scaling=0; latency_scaling_factor=1; commitnumber=0; expanded=1; instructions=System.Object[]; performcheckpoints=0; performcheckpointsatstart=0;
numberofcheckpoints=2; baselinerunid=-1; workloadid=-104; workloadinfor=; statisreportnames=; resultlogname=}, @{classname=CMixTest; name=TPC-C Transaction Mix; testid=2;
guid={D48CB95A-8003-4339-8C91-4174ED5631FA}; options=5120; enable_latency_scaling=0; latency_scaling_factor=1; commitnumber=0; expanded=1; instructions=System.Object[];
performcheckpoints=0; performcheckpointsatstart=0; numberofcheckpoints=2; baselinerunid=-1; workloadid=-104; workloadinfor=; statisreportnames=;
resultlogname=C:\Users\Administrator\Documents\My Benchmark Factory\8.0.0\x64\Result Logs\TPC-C_BMF_JOB_1.log; userloads=15000; duration=; enablepreiterationshelltimeout=1;
preiterationshelltimeout=30; preiterationshellfilename=; enablepostiterationshelltimeout=1; postiterationshelltimeout=30; postiterationshellfilename=}}
user_agent : {@{classname=CUserAgent; host=10.108.2.38; username=administrator; session_cred=0; version=; os=Microsoft Windows 10 Pro; processor=Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz; no_core=0;
no_cpus=1; physical_memory=12884430848.0; virtual_memory=0.0; disk_size=33494659072.0; disk_free_size=5140746240.0; number_of_agent=5; max_virtual_user=0; checked=1; flags=0;
address_width=0}, @{classname=CUserAgent; host=10.108.2.204; username=administrator; session_cred=0; version=; os=Microsoft Windows 10 Pro; processor=Intel(R) Xeon(R) CPU E5-2630 v3 @
2.40GHz; no_core=0; no_cpus=1; physical_memory=12884430848.0; virtual_memory=0.0; disk_size=33494659072.0; disk_free_size=5140746240.0; number_of_agent=5; max_virtual_user=0; checked=1;
flags=0; address_width=0}}
schedule : @{classname=CJobSchedule; enable=0; occursonce=1; executenexttime=1508978435; occursdwm=0; occurseverydwm=1; everymonday=0; everytuesday=0; everywednesday=0; everythursday=0;
everyfriday=0; everysaturday=0; everysunday=0; monthlyparticularday=1; dayxofevery=1; monthly1stlast=0; monthlydayofweek=0; monthlyeverynmonths=1; occursdailyonce=1; occursdailyhm=0;
occursdailyeveryhm=1; dailystartat=335772000; dailyendat=335858399; executeonceperdayat=1508978435; scheduleends=0; scheduleend=1508997599; deleteafterrun=0}
connection : @{connectiontype=SQLServerODBC; name=MSSQL_10.108.4.77_ODBC; username=sa; datasourcename=10.108.4.77_ODBC; infinitetimeout=False; timeout=60; reconnectoption=;
customproperties=System.Object[]; serverproperties=System.Object[]}
resultlog_path :
agentlogfilenames : BMFConsole_Error.log,error_BMFAgent_1_12.log,error_BMFAgent_1_11.log,error_BMFAgent_1_10.log,error_BMFAgent_1_14.log,error_BMFAgent_1_13.log,error_BMFAgent_1_17.log,error_BMFAgent_1_16.lo
g,error_BMFAgent_1_15.log,error_BMFAgent_1_18.log,error_BMFAgent_1_19.log,GeneralInformations_BMFAgent_1.log,GeneralInformations_BMFAgent_1.log,


I use the previous testrun ID for pulling the test result summary (testrun id 12):

C:\Users\Administrator\git\vmwareSysbenchBMFAutomation [master]> Invoke-RestMethod -Method Get -Uri localhost:30110/.../UserloadResults


classname : CStationResults
stationid : 0
computername :
stationcfgid : 0
userid : 0
testid : 0
testversion : 0
userload : 15000
runid : 12
profileid : -1000
tps : 1701.6669
bps : 774610.0
rps : 9275.1689
avgtime : 4072
mintime : 1
maxtime : 296038
totaltime : 2147483647
testedtime : 0
totalrows : 16680634
totalbytes : 1393071565
executions : 3060309
totalxactexecutions : 0
totalerrors : 0
deadlocks : 17
phaseid : 1
totalrollback : 0
avgresponsetime : 4072
minresponsetime : 1
maxresponsetime : 296038
totalresponsetime : 2147483647
avgthinktime : 2333
minthinktime : 0
maxthinktime : 6012
totalthinktime : 0
avgarrivaltime : 0
minarrivaltime : 0
maxarrivaltime : 0
totalarrivaltime : 0
avgkeyingtime : 2407
minkeyingtime : 500
maxkeyingtime : 4500
totalkeyingtime : 0
ninetythresponsetime : 0
ninetythtransactiontime : 0
ninetyththinktime : 0
ninetytharrivaltime : 0
ninetythkeyingtime : 0
testguid : {900ECBE0-4319-456C-B31E-A8F43AB9AB38}
v_stationid : 0
node_number : -1

Also last question Smile.  I need the time series samples of the test as well from the sampling period.  Is that part of the transactionResult route?  If so what is the userloadresultindex?  I've tried the following guesses on what that is and they don't work?

C:\Users\Administrator\git\vmwareSysbenchBMFAutomation [master]> Invoke-RestMethod -Method Get -Uri localhost:30110/.../TransactionResults
Invoke-RestMethod : 404 Not FoundNot FoundThe requested URL was not found.
At line:1 char:1
+ Invoke-RestMethod -Method Get -Uri localhost:30110/.../testrun ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

C:\Users\Administrator\git\vmwareSysbenchBMFAutomation [master]> Invoke-RestMethod -Method Get -Uri localhost:30110/.../TransactionResults
Invoke-RestMethod : 404 Not FoundNot FoundThe requested URL was not found.
At line:1 char:1
+ Invoke-RestMethod -Method Get -Uri localhost:30110/.../testrun ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Answered 277 6 / Create an account to join the discussion.
jhieb
Follow / 15 Nov 2017 at 6:09pm / Benchmark Factory
Latest post by on 19 Nov 2017 at 11:50pm

BMF Job Post route doesn't "check" agents??

Expand content
When running the REST route to add a job it doesn't "check" the agents for usage with the job if I add them to the user_agent key. Using the post route for the job to add agents it will correctly...

When running the REST route to add a job it doesn't "check" the agents for usage with the job if I add them to the user_agent key.  Using the post route for the job to add agents it will correctly check the keys.

This works:


$agentHash = @{
classname= "CUserAgent"
host= "10.108.4.150"
username= "administrator"
password= "ssdrive01"
session_cred= 0
version= ""
os= "Microsoft Windows 10 Pro"
processor= "Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz"
no_core= 0
no_cpus= 1
physical_memory= 12884430848.0
virtual_memory= 0.0
disk_size= 33494659072.0
disk_free_size= 5140746240.0
number_of_agent= 5
max_virtual_user= 0
checked= 1
flags= 0
address_width= 0
}


Invoke-RestMethod -Method Post -Uri "localhost:30110/.../agents" -Body ($agentHash | ConvertTo-Json -Depth 12 -Compress)

But adding the same agent information to the user_agent key within the job post method does not.

Invoke-RestMethod -Method Post -Uri ""localhost:30110/.../jobs" -ContentType 'application/json' -Body ($baseJobJSON | ConvertTo-Json -Depth 12 -Compress)

Not sure what i'm doing wrong as the other route works fine.  Thanks for any assistance on this.

Answered 326 6 / Create an account to join the discussion.
erdiaz
Follow / 7 Mar 2017 at 3:05pm / Benchmark Factory
Latest post by on 7 Apr 2017 at 5:17pm

Run Benchmark Factory TPC-C job against Postgresql or EDB Postgres...

Expand content
I've been trying to run Benchmark Factory against both the Open Source PostgreSQL and the EDB Postgres Advanced Server via ODBC. On both I get the following error when creating objects for TPC-C: ...

I've been trying to run Benchmark Factory against both the Open Source PostgreSQL and the EDB Postgres Advanced Server via ODBC.  On both I get the following error when creating objects for TPC-C:  

Started test Create Objects for TPC-C

Delete Tables

Create Tables

Load Tables

ODBC Error (42P07,7) - ERROR: relation "tpc_c_properties" already exists;

Error while executing the query

Any ideas on how to get past this?  The tables appear to have all been created.

Ed

Unanswered 1008 3 / Create an account to join the discussion.
GrantGo
Follow / 13 Jun 2016 at 10:47am / Benchmark Factory
Latest post by on 15 Jun 2016 at 6:07pm

Configuring test for running multiple usp's at the same time

Expand content
Hi I'm new to the product (i'm actually trialing it to see if we use it across our organisation) and I was hoping for some assistance in setting up a test to cover our key needs. I'd like...

Hi

I'm new to the product (i'm actually trialing it to see if we use it across our organisation)  and I was hoping for some assistance in setting up a test to cover our key needs.

I'd like to be able to run executions of our most used stored procs at the same time (i.e. EXEC usp1; EXEC usp2 & EXEC usp3) and then scale that up to running those three usp's (at the same time) for up to 100 simultaneous users.

How do i configure the test to get the three usp's to run at the same time on one test ? If i configure multiple statements in a Scalability test they run one after each other.

Thanks

Grant

Unanswered 1759 4 / Create an account to join the discussion.
sglixman
Follow / 30 May 2016 at 9:27am / Benchmark Factory
Latest post by on 1 Jun 2016 at 7:05am

Test completes without errors but results are missing

Expand content
Hello, I'm running tpc-e test, test completes without errors, but at the end of the run says: "No results found in the repository for this test run". Checked error logs at: C:\Users\Administrator...

Hello,

I'm running tpc-e test, test completes without errors, but at the end of the run says: "No results found in the repository for this test run".

Checked error logs at: C:\Users\Administrator\Documents\My Benchmark Factory\Error Logs

But could not see any related error.

1452 2 / Create an account to join the discussion.
BarryLynch
Follow / 17 May 2016 at 12:40pm / Benchmark Factory
Latest post by on 19 May 2016 at 4:00pm

Random Integers and Avoiding Key Collisions

Expand content
HI there Just getting started with Benchmark Factory sp apologies in advance for the stupid question - maybe I'm coming at this completely the wrong way I want to test a stored procedure that executes...

HI there

Just getting started with Benchmark Factory sp apologies in advance for the stupid question - maybe I'm coming at this completely the wrong way

I want to test a stored procedure that executes a lot of comoplicated stored procedures. This Main stored procedure takes a single integer parameter value ("key Value")

To provide a list of viable Key values to be passed to the the SPROC I have created a lookup table of 1 million rows (1 million unique key vals) each with a sequential rownum  for use as a means of generating a good keynnum value for the parameter to the sproc. Table looks like this (the keyvals are non sequential and basically random trying to to randomly pick a keyval would have a high miss rate)

RowNo  KeyVal
1              3625525

2              7366366

3              7356535

etc

The plan is to pass a random number to the BF scalability job that calls the stored procedure (I want to test something like $BFURandRange(1,1000000) )  and then the agent uses that random number to go to the lookup table table and get the  key value for the matching RowNo value and pass it to the sproc via the bound sproc parameter value

However the problem is that it just picks the same number every time and so I end up erroring out after 5 rows. How can I make it pick a different random number for Rowno each time so that I can get a different keyVal each time?


And to really put the bonus icing on the cake how can I guarantee that it never picks the same random number more than once?
Thanks a million for any insight - as I say, maybe I'm coming at this completely the wrong way

Cheers

Barry

2237 8 / Create an account to join the discussion.
yaping.li
Follow / 16 May 2016 at 10:39pm / Benchmark Factory
Latest post by on 17 May 2016 at 3:27pm

when trying to create a new connection to SQL server 2008 from, only...

Expand content
I have Benchmark Factory 7.3 installed. I am new to this. when I was trying to create a new connection for a SQL 2008 database, I only see oracle from the drop down list, there is no SQL server I can select...

I have Benchmark Factory 7.3 installed. I am new to this. when I was trying to create a new connection for a SQL 2008 database, I only see oracle from the drop down list,  there is no SQL server I can select, why?

1360 2 / Create an account to join the discussion.
JimR.reilly
Follow / 7 Mar 2016 at 11:43am / Benchmark Factory
Latest post by on 20 Mar 2016 at 10:15pm

Capture and Replay Problem

Expand content
I have been working on capturing and playing back workloads. I tried a simple test where I captured a work load consisting of several select statements and was able to play it back successfully. Next...

I have been working on capturing and playing back workloads. I tried a simple test where I captured a work load consisting of several select statements and was able to play it back successfully.

Next I captured 20 minutes of users testing their application.  The capture went ok but when playing it back it fails with the following error :-

ORA-06550: line 1, column 7:
PLS-00306: wrong number or types of arguments in call to 'DESCRIBE_PROCEDURE'

Any ideas as to how I would get past this problem ?


Thanks,

Jim.

2157 7 / Create an account to join the discussion.
EE
Follow / 9 Mar 2016 at 8:25am / Benchmark Factory
Latest post by on 17 Mar 2016 at 6:42pm

Replay test

Expand content
Hello , After capture workload finished, i tried to reply this on oracle database server File does not contain any SQL to import that was not a dependent statement" we should enable something...

Hello ,

After capture workload finished, i tried to reply this on oracle database server 

File does not contain any SQL to import that was not a dependent statement"

we should enable something on database server to get on all sql statements which run on database???

from where BMF will get on all sql statement which run before on oracle database server

Thanks

1334 1 / Create an account to join the discussion.
EE
Follow / 27 Feb 2016 at 9:03pm / Benchmark Factory
Latest post by on 1 Mar 2016 at 5:09pm

BMF

Expand content
Hello , I want to test database server hardware resources during workload such as CPU , Memory, I/O percentage usage (%) How much database consume CPU, memory during workload from benchmark factory...

Hello ,

I want to test database server hardware resources  during workload such as CPU , Memory, I/O percentage usage (%) 

 How much database consume CPU, memory during workload from benchmark factory for databases

Thanks

1378 1 / Create an account to join the discussion.
EE
Follow / 27 Feb 2016 at 10:06pm / Benchmark Factory
Latest post by on 1 Mar 2016 at 5:02pm

shared directory

Expand content
Hello , when i try to using benchmark factory for databases to capture workload from oracle database. I shared folder in benchmark client machine to use this folder share in both directory server &...

Hello ,

when i try to using benchmark factory for databases to capture workload from oracle database.

I shared folder in benchmark client machine to use this folder share in both directory server & client directory

but i don't know how write network path

the issue in capture directory

Note:

Shared folder "everyone can access this folder full control"

Thanks

1341 1 / Create an account to join the discussion.
Jonathan Serralheiro
Follow / 21 Oct 2015 at 9:52pm / Benchmark Factory
Latest post by on 26 Feb 2016 at 5:28pm

BenchMark Factory - Understand Results

Expand content
Hey Guys! I would appreciate your help, I am doing a case study database comparison. I utlizando Factory realizanção benchmark for TPC-C test. My biggest difficulty is to understand the...

Hey Guys!

I would appreciate your help, I am doing a case study database comparison. I utlizando Factory realizanção benchmark for TPC-C test. My biggest difficulty is to understand the results generated by the test could give me a brief explanation of what each one of them? And which of them consider to put in my documentation?

Userload Test Phase TPS BPS Rows Bytes Deadlocks Rollbacks Errors Avg Time Min Time Max Time 90th Time Avg Response Time Min Response Time
10 1     0,56   223,26 517 40230 0 0 0    0,142    0,006 5.991    0,239    0,142    0,006
50 1     2,65  1188,06 2552 216676 0 0 0    0,054    0,003    0,933    0,154    0,054    0,002
100 1     5,35  2465,85 5272 440095 0 0 0    0,036    0,002    0,475    0,108    0,036    0,002

Max Response Time 90th Response Time Avg Think Time Min Think Time Max Think Time 90th Think Time Avg Keying Time
5.991    0,239 8.601    0,000 23.995 16.386 10.606
   0,933    0,154 9.478    0,000 24.022 18.104 9.571
   0,475    0,108 9.211    0,000 23.970 18.479 9.618

sorry my english!!!


Unanswered 2595 14 / Create an account to join the discussion.
EE
Follow / 29 Dec 2015 at 8:28pm / Benchmark Factory
Latest post by on 18 Jan 2016 at 12:20am

Question

Expand content
Hello , I don't understand result summary when run TPC test there is 3 axes, one of them indicate to user load & the other is response time but what about the last one in the right indicate...

Hello ,

I don't understand result summary when run TPC test

there is 3 axes, one of them indicate to user load & the other is response time but what about the last one in the right indicate for what ?

and also response time is very high for 1 user load ..how this happen

for 1 user load...response time was 9.25 ms

for 10 user load..response time was 3.25 ms

Thanks

1386 1 / Create an account to join the discussion.
ArgenBarbie
Follow / 16 Dec 2015 at 7:54am / Benchmark Factory
Latest post by on 16 Dec 2015 at 6:04pm

What should the curve of tps look like in realtime statistics?

Expand content
Hi, The curve of tps I got is always as follow, bouncing from zero to the peak value. Is this normal? Why zero appears? Thank you.

Hi,

The curve of tps I got is always as follow, bouncing from zero to the peak value.

Is this normal? Why zero appears?

Thank you.

Unanswered 1375 1 / Create an account to join the discussion.
ArgenBarbie
Follow / 16 Jul 2015 at 2:41am / Benchmark Factory
Latest post by on 4 Nov 2015 at 2:25pm

Can Benchmark Factory present tpsE?

Expand content
The Performance Metric for TPCE is expressed in transactions-per-second-E (tpsE). But I cannot find the value of tpsE even in the detail report. Moreover, the report of TPCC shows the response time of...

The Performance Metric for TPCE is expressed in transactions-per-second-E (tpsE). But I cannot find the value of tpsE even in the detail report.

Moreover, the report of TPCC shows the response time of each transaction and 90th percentile, but I cannot find that in the report of TPCE as well.

So, should I setup something to make these displayed, or BF cannot show these now?

Unanswered 1665 2 / Create an account to join the discussion.
Jonathan Serralheiro
Follow / 23 Oct 2015 at 3:32pm / Benchmark Factory
Latest post by on 3 Nov 2015 at 4:11pm

Erro BenchMark Agent(PRED40-152) Error: New Order:.....

Expand content
Hey Guys! I I am encountering an error when running the benchmark in MySQL Enterprise database, well I can run TPC-C with 1,4,6 .. When I use up 10+ user loads, the following message appears Agent (PRED...

Hey Guys! I
I am encountering an error when running the benchmark in MySQL Enterprise database, well I can run TPC-C with 1,4,6 .. When I use up 10+ user loads, the following message appears


Agent (PRED 40-152) Error: New Order: Getting Timed Out Order Number Transaction "New Order Transaction" run failed by one virtual user agent in one of PRED40-152


I'm using 10 agents, my database is 10GB (Windows)

Note: I have tested it with more agents and the message is the same

Sorry my English!

Unanswered 1432 1 / Create an account to join the discussion.
Jonathan Serralheiro
Follow / 27 Oct 2015 at 10:20am / Benchmark Factory
Latest post by on 3 Nov 2015 at 4:02pm

Question - Results BenchMark Factory

Expand content
Hi guys! I have a question about 2 results of my test. Could give a brief introduction of the meaning of terms Avg "Think Time": Avg "Keying Time": Sorry my English.

Hi guys!
I have a question about 2 results of my test. Could give a brief introduction of the meaning of terms
Avg "Think Time":
Avg "Keying Time":


Sorry my English.

Unanswered 1359 1 / Create an account to join the discussion.
EE
Follow / 5 Oct 2015 at 1:47pm / Benchmark Factory
Latest post by on 5 Oct 2015 at 5:01pm

SQL stability

Expand content
Hello , I didn't understand we use SQL stability test to test specific SQL statement when more than users working on database or what? Thanks Shaimaa

Hello ,

I didn't understand we use SQL stability test to test specific SQL statement  when more than users working on database or what?

Thanks

Shaimaa

1407 1 / Create an account to join the discussion.
EE
Follow / 13 Sep 2015 at 1:42pm / Benchmark Factory
Latest post by on 28 Sep 2015 at 9:15pm

capture & replay

Expand content
When i used benchmark factory to capture workload from SQL server production, i don't know how replay this capture file against test environment please help me in details how do this? Thanks

When i used benchmark factory to capture workload from SQL server production, i don't know how replay this capture file against test environment

please help me in details how do this?

Thanks 

2269 12 / Create an account to join the discussion.
EE
Follow / 23 Aug 2015 at 8:24pm / Benchmark Factory
Latest post by on 22 Sep 2015 at 12:04pm

transaction

Expand content
Hi which transaction is used during benchmark standard test, if the test run query or what? and if the test run query which SQL statements is used? Thanks

Hi

which transaction is used during benchmark standard test, if the test run query or what?

and if the test run query which SQL statements is used?

Thanks

1958 6 / Create an account to join the discussion.