I'm trying to convert an object detection lst file to a rec file and train with it in SageMaker. My list looks something like this:
10 2 5 9.0000 1008.0000 1774.0000 1324.0000 1953.0000 3.0000 2697.0000 3340.0000 948.0000 1559.0000 0.0000 0.0000 0.0000 0.0000 0.0000 IMG_1091.JPG
58 2 5 11.0000 1735.0000 2065.0000 1047.0000 1300.0000 6.0000 2444.0000 2806.0000 1194.0000 1482.0000 1.0000 2975.0000 3417.0000 1739.0000 2139.0000 IMG_7000.JPG
60 2 5 12.0000 1243.0000 1861.0000 1222.0000 1710.0000 6.0000 2423.0000 2971.0000 1205.0000 1693.0000 0.0000 0.0000 0.0000 0.0000 0.0000 IMG_7061.JPG
80 2 5 1.0000 1865.0000 2146.0000 818.0000 969.0000 14.0000 1559.0000 1918.0000 1658.0000 1914.0000 6.0000 2638.0000 3042.0000 2125.0000 2490.0000 IMG_9479.JPG
79 2 5 13.0000 1556.0000 1812.0000 1440.0000 1637.0000 7.0000 2216.0000 2452.0000 1595.0000 1816.0000 0.0000 0.0000 0.0000 0.0000 0.0000 IMG_9443.JPG
Where the columns are
index, header length, object length, class id, xmin, ymin, xmax, ymax, (repeat any other ids...), image path
I then run the list through im2rec with
$ /incubator-mxnet/tools/im2rec.py my_lst.lst my_image_folder
I then upload the resultant .rec file to s3.
I then pull the necessary parts from this AWS sample notebook.
I think the only key piece is probably this:
def set_hyperparameters(num_epochs, lr_steps):
num_classes = 16
num_training_samples = 227
print('num classes: {}, num training images: {}'.format(num_classes, num_training_samples))
od_model.set_hyperparameters(base_network='resnet-50',
use_pretrained_model=1,
num_classes=num_classes,
mini_batch_size=16,
epochs=num_epochs,
learning_rate=0.001,
lr_scheduler_step=lr_steps,
lr_scheduler_factor=0.1,
optimizer='sgd',
momentum=0.9,
weight_decay=0.0005,
overlap_threshold=0.5,
nms_threshold=0.45,
image_shape=512,
label_width=350,
num_training_samples=num_training_samples)
set_hyperparameters(100, '33,67')
Ultimately I get the error: Not enough label packed in img_list or rec file.
Can someone help me identify what parts I'm missing in order to properly train with SageMaker and RecordIO files?
Thanks for your help!
Also, if I instead use
$ /incubator-mxnet/tools/im2rec.py my_lst.lst my_image_folder --pass-through --pack-label
I get the error:
Expected number of batches: 14, did not match the number of batches processed: 5. This may happen when some images or annotations are invalid and cannot be parsed. Please check the dataset and ensure it follows the format in the documentation.
This may come late but did you label your classes starting from 0 in the .lst file?
In the link you posted:
The classes should be labeled with successive numbers and start with 0.
Related
I have a dataset of some small fibrous objects on which I tried with YOLOv5OBB method but the results are not satisfying.
The detection result is poor (P=0.3, R=0.28 for total boxes = 1200 in 120 images).
The boxes are not fit well to the objects and often there are overlapped boxes in different orientation.
The considered fibrous objects include fibrous with less than 3 microns in diameters and longer than 5 microns, therefore I can not use scaling and shearing which change the size of the objects. traslate and mosaic will also damage the results as objects are cut.
Do you have any suggestion how to improve it? hyp parameters or even a different method/model?
Some output examples:
1
2
Hyp parameters:
lr0: 0.00258
lrf: 0.017
momentum: 0.779
weight_decay: 0.00058
warmup_epochs: 1.93
warmup_momentum: 0.443
warmup_bias_lr: 0.059
box: 0.0339
cls: 0.33
cls_pw: 0.825
theta: 0.299
theta_pw: 0.825
obj: 0.632
obj_pw: 1.0
iou_t: 0.2
anchor_t: 3.44
anchors: 2.5
fl_gamma: 0.0
hsv_h: 0.0188
hsv_s: 0.354
hsv_v: 0.35
degrees: 0.03
translate: 0.05
scale: 0.02
shear: 0.0
perspective: 0.0
flipud: 0.7
fliplr: 0.7
mosaic: 0.0
mixup: 0.5
copy_paste: 0.4
cls_theta: 180
csl_radius: 2.0
Could anyone help me to find the error in this AMPL's code for a simple least-square error base on the function:
F(X)=1/1+e^-x
param N>=1;# N Number of simulations
param M>=1;# Number of inputs
param simulations {1..N};
param training{1..N,1..M};
var W{1..10};
minimize obj: sum{i in simulations , j in 1..4} (1/(1+exp-(W[9]/(1+exp(-
W[j]/(1+exp(-training[i][j]))))+ W[10]/(1+exp(-W[2*j]/(1+exp(-training[i][j]))))))-training[i][5])^2;
'###### DATA
param N:=6;
param M:=4;
param training:
1 2 3 4 5 :=
1 0.209 0.555 0.644 0.355 0.0
2 0.707 0.450 0.587 0.305 1.0
3 0.579 0.521 0.745 0.394 1.0
4 0.574 0.883 0.211 0.550 1.0
5 0.797 0.055 0.430 0.937 1.0
6 0.782 0.865 0.114 0.317 1.0 ;
Thank you!
A couple of things:
is that quote mark before ###### DATA meant to be there?
You have specified that training has dimension N x M, and your data specifies that N=6, M=4, but you then define training as 6 x 5 and your objective function also refers to column 5.
If that doesn't answer your question, you might want to give more information about what error messages you're getting.
I'm building a DNN predicted (0 or 1) model based on skflow with TF v0.9.
My code with TensorFlowDNNClassifier is like this. I train about 26,000 records and test 6,500 one.
classifier = learn.TensorFlowDNNClassifier(hidden_units=[64, 128, 64], n_classes=2)
classifier.fit(features, labels, steps=50000)
test_pred = classifier.predict(test_features)
print(classification_report(test_labels, test_pred))
It takes about 1 minute and gets a result.
precision recall f1-score support
0 0.77 0.92 0.84 4265
1 0.75 0.47 0.58 2231
avg / total 0.76 0.76 0.75 6496
But I got
WARNING:tensorflow:TensorFlowDNNClassifier class is deprecated.
Please consider using DNNClassifier as an alternative.
So I updated my code with DNNClassifier simply.
classifier = learn.DNNClassifier(hidden_units=[64, 128, 64], n_classes=2)
classifier.fit(features, labels, steps=50000)
It also works well. But result was not the same.
precision recall f1-score support
0 0.77 0.96 0.86 4265
1 0.86 0.45 0.59 2231
avg / total 0.80 0.79 0.76 6496
1 's precision is improved.
Of course this is a good for me, but why it is improved?
And It takes about 2 hours.
This is about 120 times slower than previous example.
Do I have something wrong? or miss some parameters?
Or is DNNClassifier unstable with TF v0.9?
I give the same answer as here. You might experience that because you used the steps parameter instead of max_steps. It was just steps on TensorFlowDNNClassifier that in reality did max_steps. Now you can decide if you really want that in your case 50000 steps or auto abort earlier.
I conducted a feature selection using lasso method as well as a covariance test using covTest::covTest to retrieve the p.values. I borrow an example from covTest such that:
require(lars)
require(covTest)
set.seed(1234)
x=matrix(rnorm(100*10),ncol=10)
x=scale(x,TRUE,TRUE)/sqrt(99)
beta=c(4,rep(0,9))
y=x%*%beta+.4*rnorm(100)
a=lars(x,y)
covTest(a,x,y)
$results
Predictor_Number Drop_in_covariance P-value
1 105.7307 0.0000
6 0.9377 0.3953
10 0.2270 0.7974
3 0.0689 0.9334
7 0.1144 0.8921
2 0.0509 0.9504
9 0.0508 0.9505
8 0.0006 0.9994
4 0.1190 0.8880
5 0.0013 0.9987
$sigma
[1] 0.3705
$null.dist
[1] "F(2,90)
The covTest's results showed the p-values of the top hit features. My question is how to retrieve the coefficient of these features such as that of the predictor 1 as well as its Std.err and 95%CI. I'd to compare these estimates with the counterparts from glm.
I'm trying to pull data stored as $24. I want to convert it from character to numeric. The input(variable-name,comma24.) function is not working for me. A sample of the data is given below.
5.35
5.78
413,000
3,280,000
5.97
6.72
5
6.53
6
4.59
4.25
5
6.38
6.41
4.1
6.56
5.45
6.07
4.28
5.54
5.87
3.88
5.53
5.65
6.47
207,000
4,935,000
4,400,000
6,765,000
2,856,000
53,690,000
You don't show your code, but for some reason I could get it work when the reading and conversion were in different data steps, but not when it was the same data step.
The following works just fine:
DATA one;
INPUT y: $24. ##;
DATALINES;
5.35 5.78 413,000 3,280,000 5.97
RUN;
DATA one;
SET one;
z = INPUT(y, comma24.);
RUN;
However if I put the calculation of z in the first data step, I was getting missing values without any error message. I have no explanation for this behavior, but hopefully the workaround will work for you as well.