jhieb
Follow / 17 Nov 2017 at 10:13pm / Benchmark Factory
Latest post by on 28 Nov 2017 at 4:33pm

How to link a job/jobname or GUID to the corresponding test run, userload...

Expand content
I've read Kevin Dalton's blog posts which were good (Hoping that he explains this issue in part 3 ): http://www.toadworld.com/products/benchmark-factory/b/weblog/archive/2017/05/15/benchmark-factory...

I've read Kevin Dalton's blog posts which were good (Hoping that he explains this issue in part 3 Smile ):
http://www.toadworld.com/products/benchmark-factory/b/weblog/archive/2017/05/15/benchmark-factory-rest-api-part-1
http://www.toadworld.com/products/benchmark-factory/b/weblog/archive/2017/05/15/benchmark-factory-rest-api-part-2

The issue i'm running into now is how do I link the given testrun route to a job name?  Currently in the testrun information given back it has like the subtest name, SQL server connection profile, some IDs but unless i'm missing something there isn't a connection to a job without delving into the subkeys of the job JSON object?


Test run info that i need to capture (I use the run id from this to pull the following results information) Also note that the testid relates to the test object for the TPC-C transaction mix for that job:

C:\Users\Administrator\git\vmwareSysbenchBMFAutomation [master]> (Invoke-RestMethod -Uri localhost:30110/.../testruns)[0]


classname : CTestRunHistory
testid : 2
testversion : 9
testname : TPC-C Transaction Mix
runid : 12
profileid : 11
profilename : MSSQL_10.108.4.77_ODBC
comment :
statusid : 1
starttime : Wed Nov 15 14:14:24 2017
endtime : Wed Nov 15 16:44:39 2017
elapsedtime : 02 hour(s), 30 minute(s), 15 second(s)

The Job that this result came from:

C:\Users\Administrator\git\vmwareSysbenchBMFAutomation [master]> invoke-restmethod -uri localhost:30110/.../TPC-C_BMF_JOB_1


classname : CJob
name : TPC-C_BMF_JOB_1
comment :
connectionname : MSSQL_10.108.4.77_ODBC
emailoperator : 0
operatoraddress :
emailpager : 0
pageraddress :
sendnet : 0
netsendname :
realtimereportinterval : 5
realtimereportcounters :
expanded : 1
stoponerror : 1
globalvariables : {}
tests : {@{classname=CBenchmarkObjectsTest; benchmarkid=10; create_objects=1; always_delete_before_create=0; benchmark_properties=; run_on_agents=1; virtualusers_for_table=200;
parallel_index_create=1; delete_after_run=0; reload_from_backup=0; refresh_using_inserts=0; name=Create Objects for TPC-C; testid=-1; guid={728F120F-4F6D-4E94-879E-0E1C64E4FFEB};
options=4096; enable_latency_scaling=0; latency_scaling_factor=1; commitnumber=0; expanded=1; instructions=System.Object[]; performcheckpoints=0; performcheckpointsatstart=0;
numberofcheckpoints=2; baselinerunid=-1; workloadid=-104; workloadinfor=; statisreportnames=; resultlogname=}, @{classname=CMixTest; name=TPC-C Transaction Mix; testid=2;
guid={D48CB95A-8003-4339-8C91-4174ED5631FA}; options=5120; enable_latency_scaling=0; latency_scaling_factor=1; commitnumber=0; expanded=1; instructions=System.Object[];
performcheckpoints=0; performcheckpointsatstart=0; numberofcheckpoints=2; baselinerunid=-1; workloadid=-104; workloadinfor=; statisreportnames=;
resultlogname=C:\Users\Administrator\Documents\My Benchmark Factory\8.0.0\x64\Result Logs\TPC-C_BMF_JOB_1.log; userloads=15000; duration=; enablepreiterationshelltimeout=1;
preiterationshelltimeout=30; preiterationshellfilename=; enablepostiterationshelltimeout=1; postiterationshelltimeout=30; postiterationshellfilename=}}
user_agent : {@{classname=CUserAgent; host=10.108.2.38; username=administrator; session_cred=0; version=; os=Microsoft Windows 10 Pro; processor=Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz; no_core=0;
no_cpus=1; physical_memory=12884430848.0; virtual_memory=0.0; disk_size=33494659072.0; disk_free_size=5140746240.0; number_of_agent=5; max_virtual_user=0; checked=1; flags=0;
address_width=0}, @{classname=CUserAgent; host=10.108.2.204; username=administrator; session_cred=0; version=; os=Microsoft Windows 10 Pro; processor=Intel(R) Xeon(R) CPU E5-2630 v3 @
2.40GHz; no_core=0; no_cpus=1; physical_memory=12884430848.0; virtual_memory=0.0; disk_size=33494659072.0; disk_free_size=5140746240.0; number_of_agent=5; max_virtual_user=0; checked=1;
flags=0; address_width=0}}
schedule : @{classname=CJobSchedule; enable=0; occursonce=1; executenexttime=1508978435; occursdwm=0; occurseverydwm=1; everymonday=0; everytuesday=0; everywednesday=0; everythursday=0;
everyfriday=0; everysaturday=0; everysunday=0; monthlyparticularday=1; dayxofevery=1; monthly1stlast=0; monthlydayofweek=0; monthlyeverynmonths=1; occursdailyonce=1; occursdailyhm=0;
occursdailyeveryhm=1; dailystartat=335772000; dailyendat=335858399; executeonceperdayat=1508978435; scheduleends=0; scheduleend=1508997599; deleteafterrun=0}
connection : @{connectiontype=SQLServerODBC; name=MSSQL_10.108.4.77_ODBC; username=sa; datasourcename=10.108.4.77_ODBC; infinitetimeout=False; timeout=60; reconnectoption=;
customproperties=System.Object[]; serverproperties=System.Object[]}
resultlog_path :
agentlogfilenames : BMFConsole_Error.log,error_BMFAgent_1_12.log,error_BMFAgent_1_11.log,error_BMFAgent_1_10.log,error_BMFAgent_1_14.log,error_BMFAgent_1_13.log,error_BMFAgent_1_17.log,error_BMFAgent_1_16.lo
g,error_BMFAgent_1_15.log,error_BMFAgent_1_18.log,error_BMFAgent_1_19.log,GeneralInformations_BMFAgent_1.log,GeneralInformations_BMFAgent_1.log,


I use the previous testrun ID for pulling the test result summary (testrun id 12):

C:\Users\Administrator\git\vmwareSysbenchBMFAutomation [master]> Invoke-RestMethod -Method Get -Uri localhost:30110/.../UserloadResults


classname : CStationResults
stationid : 0
computername :
stationcfgid : 0
userid : 0
testid : 0
testversion : 0
userload : 15000
runid : 12
profileid : -1000
tps : 1701.6669
bps : 774610.0
rps : 9275.1689
avgtime : 4072
mintime : 1
maxtime : 296038
totaltime : 2147483647
testedtime : 0
totalrows : 16680634
totalbytes : 1393071565
executions : 3060309
totalxactexecutions : 0
totalerrors : 0
deadlocks : 17
phaseid : 1
totalrollback : 0
avgresponsetime : 4072
minresponsetime : 1
maxresponsetime : 296038
totalresponsetime : 2147483647
avgthinktime : 2333
minthinktime : 0
maxthinktime : 6012
totalthinktime : 0
avgarrivaltime : 0
minarrivaltime : 0
maxarrivaltime : 0
totalarrivaltime : 0
avgkeyingtime : 2407
minkeyingtime : 500
maxkeyingtime : 4500
totalkeyingtime : 0
ninetythresponsetime : 0
ninetythtransactiontime : 0
ninetyththinktime : 0
ninetytharrivaltime : 0
ninetythkeyingtime : 0
testguid : {900ECBE0-4319-456C-B31E-A8F43AB9AB38}
v_stationid : 0
node_number : -1

Also last question Smile.  I need the time series samples of the test as well from the sampling period.  Is that part of the transactionResult route?  If so what is the userloadresultindex?  I've tried the following guesses on what that is and they don't work?

C:\Users\Administrator\git\vmwareSysbenchBMFAutomation [master]> Invoke-RestMethod -Method Get -Uri localhost:30110/.../TransactionResults
Invoke-RestMethod : 404 Not FoundNot FoundThe requested URL was not found.
At line:1 char:1
+ Invoke-RestMethod -Method Get -Uri localhost:30110/.../testrun ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

C:\Users\Administrator\git\vmwareSysbenchBMFAutomation [master]> Invoke-RestMethod -Method Get -Uri localhost:30110/.../TransactionResults
Invoke-RestMethod : 404 Not FoundNot FoundThe requested URL was not found.
At line:1 char:1
+ Invoke-RestMethod -Method Get -Uri localhost:30110/.../testrun ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Answered 299 6 / Create an account to join the discussion.
jhieb
Follow / 15 Nov 2017 at 6:09pm / Benchmark Factory
Latest post by on 19 Nov 2017 at 11:50pm

BMF Job Post route doesn't "check" agents??

Expand content
When running the REST route to add a job it doesn't "check" the agents for usage with the job if I add them to the user_agent key. Using the post route for the job to add agents it will correctly...

When running the REST route to add a job it doesn't "check" the agents for usage with the job if I add them to the user_agent key.  Using the post route for the job to add agents it will correctly check the keys.

This works:


$agentHash = @{
classname= "CUserAgent"
host= "10.108.4.150"
username= "administrator"
password= "ssdrive01"
session_cred= 0
version= ""
os= "Microsoft Windows 10 Pro"
processor= "Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz"
no_core= 0
no_cpus= 1
physical_memory= 12884430848.0
virtual_memory= 0.0
disk_size= 33494659072.0
disk_free_size= 5140746240.0
number_of_agent= 5
max_virtual_user= 0
checked= 1
flags= 0
address_width= 0
}


Invoke-RestMethod -Method Post -Uri "localhost:30110/.../agents" -Body ($agentHash | ConvertTo-Json -Depth 12 -Compress)

But adding the same agent information to the user_agent key within the job post method does not.

Invoke-RestMethod -Method Post -Uri ""localhost:30110/.../jobs" -ContentType 'application/json' -Body ($baseJobJSON | ConvertTo-Json -Depth 12 -Compress)

Not sure what i'm doing wrong as the other route works fine.  Thanks for any assistance on this.

Answered 346 6 / Create an account to join the discussion.
Kevin Dalton

The Beta for Benchmark Factory 8.0 has just been released!

Expand content
The beta version of Benchmark Factory 8.0 is now available to download! This is your opportunity to try out the new features and give product feedback before the official release!! New in this release...

The beta version of Benchmark Factory 8.0 is now available to download!  

This is your opportunity to try out the new features and give product feedback before the official release!!  New in this release, so far;

- Enhanced REST API.  The new REST API gives users even more control of jobs, tests, transactions, and test runs and results!  
- The REST API now accepts and returns not only XML, like it did in the last release, but now JSON!

Please try out this new beta and post comments/problems/suggestions to this forum.

279 0 / Create an account to join the discussion.
erdiaz
Follow / 7 Mar 2017 at 3:05pm / Benchmark Factory
Latest post by on 7 Apr 2017 at 5:17pm

Run Benchmark Factory TPC-C job against Postgresql or EDB Postgres...

Expand content
I've been trying to run Benchmark Factory against both the Open Source PostgreSQL and the EDB Postgres Advanced Server via ODBC. On both I get the following error when creating objects for TPC-C: ...

I've been trying to run Benchmark Factory against both the Open Source PostgreSQL and the EDB Postgres Advanced Server via ODBC.  On both I get the following error when creating objects for TPC-C:  

Started test Create Objects for TPC-C

Delete Tables

Create Tables

Load Tables

ODBC Error (42P07,7) - ERROR: relation "tpc_c_properties" already exists;

Error while executing the query

Any ideas on how to get past this?  The tables appear to have all been created.

Ed

Unanswered 1026 3 / Create an account to join the discussion.
erdiaz
Follow / 10 Mar 2017 at 6:25pm / Benchmark Factory Beta Program
Latest post by on 13 Mar 2017 at 6:05pm

Installed BMF 7.6 Beta: Issues running TPC-C job

Expand content
I cannot get a TPC-C test to run against EDB 9.5 Postgres Plus Advanced Server using ODBC driver. It always gets stuck on the "Create Objects for TPC-C" test. In the database I created a user...

I cannot get a TPC-C test to run against  EDB 9.5 Postgres Plus Advanced Server using ODBC driver.  It always gets stuck on the "Create Objects for TPC-C" test.

In the database I created a user called pgtest that is a superuser and a db called pgtest that I drop & recreate between each attemped run.

Run 1

I create a TPC-C job with a Benchmark Scale of 1 (size 85.90 MB).  Under Test Options, neither Distribute index creation nor Distribute load using Benchmark Factory Agents are enabled.  I use the "Recreate objects and reload data every execution".   

Result:  The database objects are all created but the data is never loaded into the tables.  The job stays on the "Create Objects for TPC-C" test with a Status of "Loading Tables" and never finishes.

Messages:

TESTJOB2
Job Started
Starting Agent(s) on *******
Started test Create Objects for TPC-C
Delete Tables
Create Tables
Load Tables
ODBC Error (42P07,7, 1) - ERROR: relation "tpc_c_properties" already exists;
Error while executing the query
...
ODBC Error (42P07,7, 1) - ERROR: relation "tpc_c_load_progress" already exists;
Error while executing the query
Failed to create load progress table!
ODBC Error (42P07,7, 1) - ERROR: relation "tpc_c_load_progress" already exists;
Error while executing the query

...

Stopping test - Create Objects for TPC-C
Exceeded virtual user communication timeout. Continuing to stop test.
Test cancelled due to errors.
Shutting down Agent(s) on *********
** Job finished.
Total Time to perform the job:
Start Time: 3/10/2017 12:36:55 PM
End Time : 3/10/2017 12:48:30 PM
Elapsed : 0 Days 00:11:35.339

Run 2

Same as previous run, except under Test Options, I enable Distribute load using Benchmark Factory Agents with 9 virtual users.

Result:  The database objects are all created and load begins and never ends.  The Status gets to Load Data 100% on each of the tables but the loading continues.  The job stays on the "Create Objects for TPC-C" step with a Status of "Loading Tables" and never finishes.  It does say 100% complete, but if I look at the row counts they just keep increasing and the 2nd step TPC-C Transaction Mix never becomes active.  

I then had to stop the job.

Messages:

TESTJOB2
Job Started
Starting Agent(s) on *****
Started test Create Objects for TPC-C
Delete Tables
Create Tables
Load Tables
Agent(*****) Error: ODBC Error (42P07,7, 1) - ERROR: relation "tpc_c_properties" already exists;
Error while executing the query
Stopping test - Create Objects for TPC-C
Test cancelled due to errors.
Shutting down Agent(s) on *****
** Job finished.
Total Time to perform the job:
Start Time: 3/10/2017 12:49:59 PM
End Time : 3/10/2017 1:02:01 PM
Elapsed : 0 Days 00:12:01.735

----------------------------------


If I just change the the database connection to be against an Oracle database it finishes successfully in about 7 minutes.

Thanks.

(By the way, I notice that no procedures are created in the EDB database - I guess this could be that the test just does straight ANSI SQL calls).

Answered 641 7 / Create an account to join the discussion.