Gurobi Solver and Convergence - optimization
I have an ILP which is ok with small problems. Gurobi easily converged and returned correct answers for these small problems. But when it comes to a little larger problems, it does not converge after even 2 days. I have changed many parameters like "MIPFocus", "ImproveStartGap", "Cuts", "ImproveStartTime" and even "Heuristics"., but nothing happens.
Could you please help me with this issue? Is there any way to reach convergence sooner at the cost of loosing optimally? what's the problem?
Best, Amir
FYI, this ILP has10135 integer variables (most of them, 10044, are binary ). the below is log when I stop the program:
Academic license - for non-commercial use only
Optimize a model with 131848 rows, 20748 columns and 577874 nonzeros
Variable types: 0 continuous, 20748 integer (20657 binary)
Coefficient statistics:
Matrix range [1e+00, 1e+05]
Objective range [4e+01, 8e+01]
Bounds range [1e+00, 1e+00]
RHS range [1e+00, 3e+05]
Presolve removed 23245 rows and 10613 columns
Presolve time: 1.67s
Presolved: 108603 rows, 10135 columns, 526215 nonzeros
Variable types: 0 continuous, 10135 integer (10044 binary)
Presolved: 10135 rows, 118738 columns, 536350 nonzeros
Root relaxation: objective 9.360000e+03, 10205 iterations, 0.79 seconds
Total elapsed time = 5.06s
Nodes | Current Node | Objective Bounds | Work
Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time
0 0 9360.00000 0 299 - 9360.00000 - - 5s
0 0 9360.00000 0 223 - 9360.00000 - - 8s
0 2 9360.00000 0 150 - 9360.00000 - - 23s
30 31 9360.00000 9 188 - 9360.00000 - 64.8 25s
207 208 9360.00000 64 263 - 9360.00000 - 14.4 31s
400 399 9360.00000 109 298 - 9360.00000 - 10.3 36s
587 584 9360.00000 156 319 - 9360.00000 - 9.5 41s
804 794 9360.00000 209 363 - 9360.00000 - 8.8 47s
918 905 9360.00000 238 303 - 9360.00000 - 8.6 50s
1159 1133 9360.00000 294 288 - 9360.00000 - 8.3 56s
1281 1247 9360.00000 319 371 - 9360.00000 - 8.2 60s
1534 1471 9360.00000 23 208 - 9360.00000 - 8.1 66s
1809 1736 9360.00000 237 223 - 9360.00000 - 8.0 71s
1811 1737 9360.00000 89 670 - 9360.00000 - 8.0 87s
1812 1738 9360.00000 192 572 - 9360.00000 - 8.0 96s
1813 1739 9360.00000 93 572 - 9360.00000 - 8.0 109s
1814 1742 9360.00000 11 371 - 9360.00000 - 9.3 117s
1865 1775 9360.00000 19 399 - 9360.00000 - 9.5 120s
1967 1840 9360.00000 31 395 - 9360.00000 - 9.1 125s
2180 1984 9360.00000 56 435 - 9360.00000 - 9.1 130s
2383 2121 9360.00000 84 408 - 9360.00000 - 9.4 136s
2495 2197 9360.00000 97 403 - 9360.00000 - 9.5 140s
2712 2337 9360.00000 124 425 - 9360.00000 - 9.6 147s
2829 2416 9360.00000 137 448 - 9360.00000 - 9.6 151s
2957 2505 9360.00000 153 421 - 9360.00000 - 9.6 155s
3196 2660 9360.00000 183 389 - 9360.00000 - 9.6 164s
3291 2721 9360.00000 195 412 - 9360.00000 - 9.6 168s
3383 2789 9360.00000 208 424 - 9360.00000 - 9.8 173s
3475 2850 9360.00000 221 427 - 9360.00000 - 10.0 177s
3590 2909 9360.00000 235 433 - 9360.00000 - 10.0 182s
3716 2990 9360.00000 250 424 - 9360.00000 - 10.1 187s
3830 3054 9360.00000 266 398 - 9360.00000 - 10.2 192s
3987 3151 9360.00000 285 400 - 9360.00000 - 10.2 197s
4079 3212 9360.00000 299 409 - 9360.00000 - 10.2 203s
4294 3385 infeasible 327 - 9360.00000 - 10.2 208s
4489 3316 9360.00000 61 404 - 9360.00000 - 10.3 213s
4688 3528 9360.00000 104 410 - 9360.00000 - 10.4 219s
4953 3715 9360.00000 144 409 - 9360.00000 - 10.2 225s
5175 3864 9360.00000 187 413 - 9360.00000 - 10.2 232s
5455 4022 9360.00000 221 427 - 9360.00000 - 10.0 239s
5683 4172 9360.00000 264 397 - 9360.00000 - 10.0 246s
5891 4318 9360.00000 300 395 - 9360.00000 - 10.0 254s
6211 4448 9360.00000 58 421 - 9360.00000 - 9.9 261s
6508 4642 9360.00000 110 472 - 9360.00000 - 9.9 268s
6856 4855 9360.00000 183 452 - 9360.00000 - 9.8 276s
7127 5068 9360.00000 231 397 - 9360.00000 - 9.9 285s
7508 5455 9360.00000 299 450 - 9360.00000 - 9.8 294s
7906 5732 9360.00000 350 400 - 9360.00000 - 9.7 303s
8105 5876 9360.00000 353 405 - 9360.00000 - 9.9 313s
8347 6093 9360.00000 362 366 - 9360.00000 - 10.1 323s
8657 6374 9360.00000 75 424 - 9360.00000 - 10.2 334s
8962 6663 9360.00000 135 506 - 9360.00000 - 10.3 345s
9380 7037 9360.00000 209 463 - 9360.00000 - 10.3 356s
9722 7363 9360.00000 302 445 - 9360.00000 - 10.4 368s
10215 7802 9360.00000 48 392 - 9360.00000 - 10.3 381s
10594 8171 9360.00000 128 488 - 9360.00000 - 10.3 394s
11142 8706 9360.00000 231 488 - 9360.00000 - 10.2 408s
11727 9203 infeasible 348 - 9360.00000 - 10.1 421s
12126 9573 9360.00000 112 489 - 9360.00000 - 10.2 435s
12631 10058 9360.00000 235 471 - 9360.00000 - 10.2 448s
13057 10467 9360.00000 313 509 - 9360.00000 - 10.3 461s
13442 10831 9360.00000 361 428 - 9360.00000 - 10.4 475s
14060 11357 9360.00000 62 399 - 9360.00000 - 10.3 489s
14714 11805 9360.00000 149 428 - 9360.00000 - 10.2 502s
15229 12295 9360.00000 258 458 - 9360.00000 - 10.1 516s
15794 12838 9360.00000 355 420 - 9360.00000 - 10.1 530s
16395 13384 infeasible 434 - 9360.00000 - 10.0 542s
16849 13726 9360.00000 124 497 - 9360.00000 - 10.1 555s
17364 14277 9360.00000 233 457 - 9360.00000 - 10.0 568s
17855 14758 9360.00000 327 432 - 9360.00000 - 10.0 582s
18446 15223 9360.00000 62 403 - 9360.00000 - 9.9 595s
19030 15662 9360.00000 152 434 - 9360.00000 - 9.9 608s
19502 16142 9360.00000 239 453 - 9360.00000 - 9.8 620s
20069 16702 9360.00000 355 432 - 9360.00000 - 9.8 633s
20643 17143 9360.06655 434 415 - 9360.00000 - 9.7 646s
21219 17545 9360.00000 89 493 - 9360.00000 - 9.7 658s
21694 17994 9360.00000 183 526 - 9360.00000 - 9.7 671s
22237 18517 9360.00000 302 462 - 9360.00000 - 9.6 683s
22822 18976 infeasible 383 - 9360.00000 - 9.5 695s
23246 19366 9360.00000 117 503 - 9360.00000 - 9.6 707s
23765 19879 9360.00000 212 484 - 9360.00000 - 9.5 720s
24139 20275 9360.00000 283 415 - 9360.00000 - 9.6 732s
24747 20695 infeasible 330 - 9360.00000 - 9.5 743s
25278 21165 9360.00000 90 434 - 9360.00000 - 9.5 755s
25714 21591 9360.00000 173 462 - 9360.00000 - 9.5 767s
26243 22075 9360.00000 296 396 - 9360.00000 - 9.4 779s
26830 22569 9360.00000 96 413 - 9360.00000 - 9.4 791s
27303 22968 9360.00000 188 438 - 9360.00000 - 9.4 802s
27692 23352 9360.00000 287 440 - 9360.00000 - 9.4 815s
28208 23839 9360.00000 50 370 - 9360.00000 - 9.4 826s
28753 24256 9360.00000 131 464 - 9360.00000 - 9.4 838s
29199 24630 9360.00000 71 408 - 9360.00000 - 9.4 850s
29586 25000 9360.00000 157 475 - 9360.00000 - 9.4 862s
30104 25497 9360.00000 247 428 - 9360.00000 - 9.3 874s
30660 25890 9600.00000 302 375 - 9360.00000 - 9.3 886s
30986 26191 9600.00000 309 399 - 9360.00000 - 9.5 899s
31374 26569 9600.00000 324 314 - 9360.00000 - 9.5 911s
31748 26902 9600.00000 324 346 - 9360.00000 - 9.5 923s
32341 27292 9360.00000 80 495 - 9360.00000 - 9.4 934s
32762 27690 9360.00000 159 528 - 9360.00000 - 9.5 947s
33288 28176 9360.00000 283 472 - 9360.00000 - 9.4 959s
33816 28698 infeasible 375 - 9360.00000 - 9.4 971s
34019 28872 9360.00000 382 355 - 9360.00000 - 9.5 982s
34249 29030 9360.00000 384 340 - 9360.00000 - 9.6 994s
34477 29168 9420.00000 407 370 - 9360.00000 - 9.7 1006s
34799 29473 infeasible 419 - 9360.00000 - 9.7 1018s
35163 29804 9360.00000 89 435 - 9360.00000 - 9.7 1031s
35749 30322 9360.00000 198 452 - 9360.00000 - 9.7 1044s
36357 30790 infeasible 295 - 9360.00000 - 9.6 1057s
36844 31266 9360.00000 147 473 - 9360.00000 - 9.6 1070s
37359 31755 9360.00000 226 463 - 9360.00000 - 9.6 1082s
37761 32178 9360.00000 328 494 - 9360.00000 - 9.6 1096s
38309 32576 9362.39601 376 479 - 9360.00000 - 9.6 1108s
38922 33011 infeasible 402 - 9360.00000 - 9.6 1120s
39313 33349 9360.00000 123 434 - 9360.00000 - 9.6 1132s
39891 33867 9360.00000 245 461 - 9360.00000 - 9.6 1145s
40260 34232 9360.00000 321 487 - 9360.00000 - 9.6 1157s
40817 34704 9360.00000 87 444 - 9360.00000 - 9.6 1169s
41151 35031 9360.00000 121 521 - 9360.00000 - 9.6 1182s
41732 35534 9360.00000 231 492 - 9360.00000 - 9.6 1196s
42304 35973 infeasible 321 - 9360.00000 - 9.6 1200s
Explored 42427 nodes (435518 simplex iterations) in 1200.24 seconds
Thread count was 8 (of 8 available processors)
Solution count 0
update:
even with a very small problem and continuous variables instead of integer variables,(469 binary and 15 continuous variables ), gurobi stuck in searching for feasible solutions. I think there must be something to do inorder to prevent this problem and make gurobi converge! the log for small problem:
Academic license - for non-commercial use only
Optimize a model with 2342 rows, 1206 columns and 8898 nonzeros
Variable types: 15 continuous, 1191 integer (1191 binary)
Coefficient statistics:
Matrix range [1e+00, 1e+05]
Objective range [1e+00, 8e+02]
Bounds range [1e+00, 2e+01]
RHS range [1e+00, 3e+05]
Presolve removed 849 rows and 722 columns
Presolve time: 0.01s
Presolved: 1493 rows, 484 columns, 6414 nonzeros
Variable types: 15 continuous, 469 integer (469 binary)
Root relaxation: objective 2.160000e+03, 168 iterations, 0.00 seconds
Nodes | Current Node | Objective Bounds | Work
Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time
0 0 2160.00000 0 19 - 2160.00000 - - 0s
0 0 2160.00000 0 44 - 2160.00000 - - 0s
0 0 2160.00000 0 57 - 2160.00000 - - 0s
0 0 2160.00000 0 46 - 2160.00000 - - 0s
0 0 2160.00000 0 31 - 2160.00000 - - 0s
0 0 2160.00000 0 28 - 2160.00000 - - 0s
0 0 2160.00000 0 49 - 2160.00000 - - 0s
0 0 2160.00000 0 42 - 2160.00000 - - 0s
0 0 2160.00000 0 59 - 2160.00000 - - 0s
0 0 2160.00000 0 41 - 2160.00000 - - 0s
0 2 2160.00000 0 41 - 2160.00000 - - 0s
10135 1630 2160.00000 36 12 - 2160.00000 - 8.7 5s
22631 2543 infeasible 40 - 2160.00000 - 12.8 10s
34157 2675 2160.00000 40 43 - 2160.00000 - 14.5 15s
47547 2906 infeasible 43 - 2160.00000 - 15.1 20s
61008 3057 2160.00000 36 30 - 2160.00000 - 15.3 25s
70483 3488 2160.00000 42 20 - 2160.00000 - 15.7 30s
81159 4625 2160.00000 34 23 - 2160.00000 - 15.8 35s
94894 6051 infeasible 43 - 2160.00000 - 16.0 40s
106278 6567 2160.00000 47 32 - 2160.00000 - 16.1 45s
118415 7154 2160.00000 40 9 - 2160.00000 - 16.4 50s
130278 7148 2160.00000 40 14 - 2160.00000 - 16.7 55s
141753 8045 2160.00000 38 21 - 2160.00000 - 16.8 60s
153321 8861 2160.00000 37 17 - 2160.00000 - 17.0 65s
163315 9327 infeasible 37 - 2160.00000 - 17.2 70s
174712 9284 2160.00000 43 22 - 2160.00000 - 17.4 75s
186989 9750 2160.00000 40 14 - 2160.00000 - 17.3 80s
199567 9980 2160.00000 47 17 - 2160.00000 - 17.3 85s
213163 10894 2160.00000 41 11 - 2160.00000 - 17.0 90s
225271 11352 infeasible 34 - 2160.00000 - 16.9 95s
237961 11732 2160.00000 45 7 - 2160.00000 - 16.8 100s
250844 11855 infeasible 46 - 2160.00000 - 16.7 105s
265028 13816 infeasible 51 - 2160.00000 - 16.8 110s
278712 14912 2160.00000 41 14 - 2160.00000 - 16.8 115s
290532 15964 2160.00000 43 27 - 2160.00000 - 16.8 120s
302974 17402 infeasible 44 - 2160.00000 - 16.8 125s
315002 18302 infeasible 42 - 2160.00000 - 16.8 130s
327409 19249 2160.00000 37 23 - 2160.00000 - 16.8 135s
339414 20128 2160.00000 50 16 - 2160.00000 - 16.8 140s
352135 20727 infeasible 45 - 2160.00000 - 16.8 145s
367381 21309 2160.00000 38 13 - 2160.00000 - 16.8 150s
7660856 348402 infeasible 49 - 2160.00000 - 15.9 3205s
7672378 348678 2160.00000 33 21 - 2160.00000 - 15.9 3210s
7685454 348828 2160.00000 37 18 - 2160.00000 - 15.9 3215s
7697794 348947 2160.00000 49 2 - 2160.00000 - 15.9 3220s
7707262 349326 2160.92308 40 31 - 2160.00000 - 15.9 3225s
7718583 349877 infeasible 40 - 2160.00000 - 15.9 3230s
7729574 350121 infeasible 40 - 2160.00000 - 15.9 3235s
7741901 350412 infeasible 44 - 2160.00000 - 15.9 3240s
7751253 350381 2160.00000 49 32 - 2160.00000 - 15.9 3245s
7763103 350489 2160.00000 37 26 - 2160.00000 - 15.9 3250s
7773839 350681 2160.00000 38 27 - 2160.00000 - 15.9 3255s
7786222 351217 infeasible 45 - 2160.00000 - 15.9 3260s
7797384 351803 infeasible 46 - 2160.00000 - 15.9 3265s
7808953 352474 2160.00000 51 12 - 2160.00000 - 15.9 3270s
7820291 353040 2160.00000 49 8 - 2160.00000 - 15.8 3275s
7831847 353412 2160.00000 54 2 - 2160.00000 - 15.8 3280s
7842631 354132 infeasible 50 - 2160.00000 - 15.8 3285s
7852436 354657 infeasible 47 - 2160.00000 - 15.8 3290s
7861503 354637 2160.00000 39 24 - 2160.00000 - 15.8 3295s
7874356 354907 2160.00000 41 9 - 2160.00000 - 15.8 3300s
Interrupt request received
Cutting planes:
Learned: 7
Gomory: 11
Cover: 18
Implied bound: 2
Clique: 10
MIR: 99
StrongCG: 6
Flow cover: 243
Inf proof: 6
Explored 7885073 nodes (124881023 simplex iterations) in 3303.69 seconds
Thread count was 8 (of 8 available processors)
Solution count 0
Solve interrupted
Best objective -, best bound 2.159999999845e+03, gap -
Have you tried to to turn off the presolve?
set presolve to 2 and try. Sometimes the presolve step prunes some feasible region in order to thighten the formulation and as consequence only a few feasible solutions remain in the model.
In many instances turning off the presolver helps.
Bests
Related
When does URL to .csv open or download file?
I'm learning some Python pandas and the course uses https://gist.githubusercontent.com/sh7ata/e075ff35b51ebb0d2d577fbe1d19ebc9/raw/b966d02c7c26bcca60703acb1390e938a65a35cb/drinks.csv Clicking this link opens the actual .csv file contents in my browser and I can read the data into pandas straight away. However, this doesn't work for https://www.spss-tutorials.com/downloads/browsers.csv. If I click this link, Google Chrome downloads the file rather than show its contents. Why is this and what can I do about it? I mean, they're both URLs for .csv files, right?
You can use requests module with custom HTTP header to download it. For example:
import requests
import pandas as pd
from io import StringIO
url = "https://www.spss-tutorials.com/downloads/browsers.csv"
headers = {
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0"
}
req = requests.get(url, headers=headers)
df = pd.read_csv(StringIO(req.text))
print(df.to_markdown())
Prints:
screen_resolution
sessions
perc_new_sessions
new_users
bounce_rate
pages_session
avg_session_duration
goal_conversion_rate
goal_completions
goal_value
0
1366x768
2,284
79.60%
1,818
69.40%
1.93
00:02:14
0.00%
0
€0.00
1
1920x1080
2,013
72.28%
1,455
71.93%
2.02
00:02:18
0.00%
0
€0.00
2
1280x1024
1,217
72.14%
878
74.53%
1.9
00:02:05
0.00%
0
€0.00
3
1680x1050
1,052
68.16%
717
74.62%
1.93
00:01:46
0.00%
0
€0.00
4
1440x900
921
77.85%
717
74.05%
1.73
00:01:45
0.00%
0
€0.00
5
1280x800
865
80.00%
692
71.91%
1.76
00:01:37
0.00%
0
€0.00
6
1600x900
737
76.39%
563
72.86%
1.8
00:02:02
0.00%
0
€0.00
7
1920x1200
441
64.85%
286
73.92%
1.87
00:01:55
0.00%
0
€0.00
8
1024x768
192
88.02%
169
73.96%
2.07
00:01:32
0.00%
0
€0.00
9
2560x1440
137
67.15%
92
61.31%
1.86
00:02:02
0.00%
0
€0.00
10
1280x720
134
82.84%
111
66.42%
2.16
00:01:15
0.00%
0
€0.00
11
1536x864
118
78.81%
93
72.03%
1.78
00:01:45
0.00%
0
€0.00
12
320x568
104
84.62%
88
75.00%
1.89
00:01:18
0.00%
0
€0.00
13
768x1024
91
83.52%
76
67.03%
2.66
00:02:11
0.00%
0
€0.00
14
1360x768
70
77.14%
54
74.29%
1.69
00:01:08
0.00%
0
€0.00
15
360x640
70
71.43%
50
77.14%
2.06
00:02:06
0.00%
0
€0.00
16
1600x1200
62
80.65%
50
82.26%
1.32
00:01:22
0.00%
0
€0.00
17
1344x840
56
44.64%
25
53.57%
3.11
00:04:39
0.00%
0
€0.00
18
320x480
51
80.39%
41
72.55%
1.61
00:00:55
0.00%
0
€0.00
19
1093x614
41
80.49%
33
78.05%
1.76
00:01:42
0.00%
0
€0.00
20
1280x768
38
60.53%
23
68.42%
2.63
00:02:41
0.00%
0
€0.00
21
1024x600
35
94.29%
33
85.71%
1.37
00:01:23
0.00%
0
€0.00
...and so on
How to call the same index in each of 2 dimensions in a GAMS variable?
I am running a model in GAMS with a number of variables with 2 dimensions, calling the indexes i and j respectively. I have a few constraints relating to just the "diagonal" entries, so I coded them something like "d(i,i)", but GAMS gives error 171 when I do this, for example the first error comes on equation e3. How do I express this in GAMS? Code follows: Set i 'Origin' / 1*20 / j 'Destination' / 1*20 /; Table A(i,j) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 1 0 4 2 574 481 440 408 633 573 1066 486 1926 1537 183 334 374 107 509 378 499 2 4 0 2 573 480 438 412 632 572 1064 484 1924 1535 182 332 372 111 508 377 498 3 3 2 0 572 479 437 411 631 571 1063 484 1923 1534 181 332 371 109 507 376 497 4 574 572 572 0 93 249 135 1188 1128 1620 1040 2480 2091 737 888 928 683 498 367 488 5 481 480 479 93 0 156 228 1095 1035 1527 948 2387 1998 645 796 836 591 406 275 396 6 440 438 437 249 156 0 384 1053 993 1485 906 2345 1956 603 754 794 549 364 233 354 7 409 412 411 135 228 384 0 1044 984 1476 897 2336 1947 594 745 785 302 633 502 623 8 633 632 631 1188 1095 1053 1044 0 532 1024 623 1246 792 450 300 403 742 560 992 668 9 573 572 571 1128 1035 993 984 532 0 494 326 879 424 392 385 207 682 1063 932 1053 10 1066 1064 1063 1620 1527 1485 1476 1024 494 0 820 1979 823 885 878 699 1175 1555 1424 1545 11 486 484 484 1040 948 906 897 623 326 820 0 724 270 359 477 223 595 975 844 966 12 1926 1924 1923 2480 2387 2345 2336 1246 878 1979 724 0 454 1943 1099 846 2035 2415 2284 2405 13 1537 1535 1534 2091 1998 1956 1947 792 424 823 270 454 0 1554 645 392 1646 2026 1895 2016 14 183 182 181 738 645 603 594 450 392 884 359 1943 1554 0 151 192 292 672 542 663 15 334 332 332 888 796 754 745 300 385 878 477 1099 645 151 0 257 443 261 692 369 16 373 372 371 928 835 793 784 403 207 699 223 846 392 192 256 0 482 863 732 853 17 107 111 109 683 591 549 302 742 683 1175 595 2035 1646 292 443 483 0 618 487 608 18 509 507 507 498 406 364 633 560 1063 1555 975 2415 2026 672 261 863 618 0 134 180 19 378 377 376 367 275 233 502 992 932 1424 844 2284 1895 542 692 732 487 134 0 122 20 500 498 497 489 396 354 624 668 1053 1545 966 2405 2016 663 369 854 609 180 122 0 ; Positive variable c(i,j) 'cost direct'; Positive variable cstar(i,j) 'cost routing'; Positive variable z 'objective'; Binary variable d(i,j) 'decision direct'; Binary variable dstar(i,j) 'decision routing'; Binary variable p(i,j) 'd(i,i)*dstar(i,j)'; Binary variable q(i,j) 'd(i,j)*p(i,j)'; Binary variable x(i,j) 'd(j,j)*dstar(i,j)'; Binary variable y(i,j) 'd(i,j)*x(i,j)'; Binary variable f(i,j) 'd(i,i)*d(j,j)'; Binary variable g(i,j) 'f(i,j)*dstar(i,j)'; Integer variable h(i) 'd(i,i)*u(i)'; Integer variable k(i,j) 'd(i,i)*u(j)'; Integer variable m(i,j) 'f(i,j)*u(i)'; Integer variable u(i) 'ordering of tours'; Equation e1 e2(i,j) e3 e4(j) e5(i,j) e6(i,j) e7(i) e8(j) e9(i) e10(i,j) e11(i,j) e12(i,j) e13(i,j) e14(i,j) e15(i,j) e16(i,j) e17(i,j) e18(i,j) e19(i,j) e20(i,j) e21(i,j) e22(i,j) e23(i,j) e24(i,j) e25(i,j) e26(i,j) e27(i,j) e28(i) e29(i) e30(i) e31(i,j) e32(i,j) e33(i,j) e34(i,j) e35(i,j) e36(i,j) e37(i,j) e38(i,j) e39(i) e40(i) e41(i) e42(i) e43(i,j) e44(i,j) e45(i,j) e46(i,j) e47(i,j) e48(i,j) e49(i,j) e50(i,j) e51(i,j); e1 .. z =e= sum((i,j),c(i,j))+sum((i,j),cstar(i,j)); e2(i,j) .. c(i,j) =e= A(i,j)*d(i,j); e3 .. sum((i),d(i,i)) =e= 2; e4(j) .. sum((i),d(i,j)) =e= 1; e5(i,j) .. d(i,j)-d(i,i) =l= 0; e6(i,j) .. cstar(i,j) =e= A(i,j)*dstar(i,j); e7(i) .. dstar(i,i) =e= 0; e8(j) .. sum((i),dstar(i,j)) =e= 1; e9(i) .. sum((j),dstar(i,j)) =e= 1; e10(i,j) .. p(i,j) =l= d(i,i); e11(i,j) .. p(i,j) =l= dstar(i,j); e12(i,j) .. p(i,j) =g= d(i,i)+dstar(i,j)-1; e13(i,j) .. p(i,j) =g= 0; e14(i,j) .. q(i,j) =l= d(i,j); e15(i,j) .. q(i,j) =l= p(i,j); e16(i,j) .. q(i,j) =g= d(i,j)+p(i,j)-1; e17(i,j) .. q(i,j) =g= 0; e18(i,j) .. p(i,j)-q(i,j) =e= 0; e19(i,j) .. x(i,j) =l= d(j,j); e20(i,j) .. x(i,j) =l= dstar(i,j); e21(i,j) .. x(i,j) =g= d(j,j)+dstar(i,j)-1; e22(i,j) .. x(i,j) =g= 0; e23(i,j) .. y(i,j) =l= d(i,j); e24(i,j) .. y(i,j) =l= x(i,j); e25(i,j) .. y(i,j) =g= d(i,j)+x(i,j)-1; e26(i,j) .. y(i,j) =g= 0; e27(i,j) .. x(i,j)-y(i,j) =e= 0; e28(i) .. u(i) =l= 20; e29(i) .. 1000*(1-d(i,i)) =g= u(i)-1; e30(i) .. 2-d(i,i) =l= u(i); e31(i,j) .. f(i,j) =l= d(i,i); e32(i,j) .. f(i,j) =l= dstar(j,j); e33(i,j) .. f(i,j) =g= d(i,i)+dstar(j,j)-1; e34(i,j) .. f(i,j) =g= 0; e35(i,j) .. g(i,j) =l= f(i,j); e36(i,j) .. g(i,j) =l= dstar(i,j); e37(i,j) .. g(i,j) =g= f(i,j)+dtar(i,j)-1; e38(i,j) .. g(i,j) =g= 0; e39(i) .. 0 =l= h(i); e40(i) .. h(i) =l= 1000*d(i,i); e41(i) .. u(i)-1000*(1-d(i,i)) =l= h(i); e42(i) .. h(i) =l= u(i); e43(i,j) .. 0 =l= k(i,j); e44(i,j) .. k(i,j) =l= 1000*d(i,i); e45(i,j) .. u(j)-1000*(1-d(i,i)) =l= k(i,j); e46(i,j) .. k(i,j) =l= u(j); e47(i,j) .. 0 =l= m(i,j); e48(i,j) .. m(i,j) =l= 1000*f(i,j); e49(i,j) .. u(i)-1000*(1-f(i,j)) =l= m(i,j); e50(i,j) .. m(i,j) =l= u(i); e51(i,j) .. u(i)-u(j)+1-h(i)+k(i,j)-d(i,i)-k(j,i)+h(j)-d(j,j)-m(i,j)+m(j,i)-f(i,j) =l= 19*(1-d(i,i)-d(j,j)-f(i,j)-dstar(i,j)+p(i,j)+x(i,j)+g(i,j)); Model transport / all /; solve transport using mip minimizing z; display d.l, dstar.l;
The problem is, that you declared d (and other symbols) with domain (i,j) but try to access it as (i,i). Therefore, you get a domain violation since you declared i and j as different sets. Is that really wanted? Or are they really the same (i.e., is each origin also a destination)? Looking at table A that seems to be the case. So, instead of declaring j as a new set, you should define it as Alias to j. The start of your model would look like this then: Set i 'Origin' / 1*20 /; Alias(i,j); The rest stays the same.
Using beautifulsoup to extract information within pre, turning it into a table to csv
I am trying to extract the text from the below site shown within the code. While I can print the list fine, I can't seem to turn it into a pandas dataframe, and print it out as a csv. This is a site that only has the pre info. Please let me know if there is way to do this. import requests from bs4 import BeautifulSoup #url list for the new stations url1="https://www.kyoshin.bosai.go.jp/cgi-bin/kyoshin/db/sitedat.cgi?1+NIG010+knet" tt1="C:/temp/" page = requests.get(url1) soup = BeautifulSoup(page.content, 'html.parser') print(soup) N-Value P,S-Velocity Density Soil Column (m/s) (g/cm^3) ---------------------------------------------------------------------------------- 1m 13 1351 93 1.43 0m - 1m Fl 2m 9 1351 105 1.77 1m - 7.75m S 3m 11 1389 102 1.86 7.75m - 15.15m S 4m 7 1408 104 1.83 15.15m - 16.75m S 5m 20 1429 120 1.74 16.75m - 19.3m SF 6m 20 1481 121 1.89 19.3m - 22.75m SF 7m 24 1538 143 1.97 22.75m - 25.7m M 8m 53 1550 189 1.87 25.7m - 33.44m S 9m 52 1550 233 1.85 10m 47 1504 222 1.93 11m 43 1493 206 1.9 12m 38 1504 222 1.89 13m 27 1492 213 1.84 14m 44 1492 213 1.9 15m 62 1527 235 1.89 16m 46 1504 189 1.92 17m 22 1481 165 1.87 18m 26 1471 147 1.86 19m 24 1493 202 1.82 20m 21 1493 198 1.87
Not the most robust, but you can iterate line by line to parse the data you need:
import requests
import pandas as pd
from io import StringIO
#url list for the new stations
url1="https://www.kyoshin.bosai.go.jp/cgi-bin/kyoshin/db/sitedat.cgi?1+NIG010+knet"
tt1="C:/temp/"
page = requests.get(url1)
s=str(page.content,'utf-8')
df = pd.DataFrame()
for lineNum, line in enumerate(s.splitlines()):
if lineNum == 0:
headers = line.split()
elif lineNum == 1:
unit1, unit2 = line.split()
elif lineNum == 2:
continue
else:
row = line.split()
idx = row[0]
nval = row[1]
ps = row[2] + ' ' + row[3]
den = row[4]
try:
soil = '%s %s %s' %(row[5], row[6], row[7])
col = row[8]
except Exception as e:
print(e)
soil = ''
col = ''
temp_df = pd.DataFrame([[idx, nval, ps, den, soil, col]],
columns = ['Index',headers[0], headers[1] + ' ' + unit1, headers[2] + ' ' + unit2, headers[3], headers[4]])
df = df.append(temp_df, sort=False).reset_index(drop=True)
df.to_csv('file.csv',index=False)
Output:
print (df)
Index N-Value P,S-Velocity (m/s) Density (g/cm^3) Soil Column
0 1m 13 1351 93 1.43 0m - 1m Fl
1 2m 9 1351 105 1.77 1m - 7.75m S
2 3m 11 1389 102 1.86 7.75m - 15.15m S
3 4m 7 1408 104 1.83 15.15m - 16.75m S
4 5m 20 1429 120 1.74 16.75m - 19.3m SF
5 6m 20 1481 121 1.89 19.3m - 22.75m SF
6 7m 24 1538 143 1.97 22.75m - 25.7m M
7 8m 53 1550 189 1.87 25.7m - 33.44m S
8 9m 52 1550 233 1.85
9 10m 47 1504 222 1.93
10 11m 43 1493 206 1.9
11 12m 38 1504 222 1.89
12 13m 27 1492 213 1.84
13 14m 44 1492 213 1.9
14 15m 62 1527 235 1.89
15 16m 46 1504 189 1.92
16 17m 22 1481 165 1.87
17 18m 26 1471 147 1.86
18 19m 24 1493 202 1.82
19 20m 21 1493 198 1.87
How to query using an array of columns on SQL Server 2008
Can you please help on this, Im trying to write a query which retrieves a total amount from an array of columns, I dont know if there is a way to do this, I retrieve the array of columns I need from this query: USE Facebook_Global GO SELECT c.name AS column_name FROM sys.tables AS t INNER JOIN sys.columns AS c ON t.OBJECT_ID = c.OBJECT_ID WHERE t.name LIKE '%Lifetime Likes by Gender and###$%' and c.name like '%m%' Which gives me this table column_name M#13-17 M#18-24 M#25-34 M#35-44 M#45-54 M#55-64 M#65+ So I need a query that gives me a TotalAmount of those columns listed in that table. Can this be possible? Just to clarify a little: I have this table Date F#13-17 F#18-24 F#25-34 F#35-44 F#45-54 F#55-64 F#65+ M#13-17 M#18-24 M#25-34 M#35-44 M#45-54 M#55-64 M#65+ 2015-09-06 00:00:00.000 257 3303 1871 572 235 116 71 128 1420 824 251 62 32 30 2015-09-07 00:00:00.000 257 3302 1876 571 234 116 72 128 1419 827 251 62 32 30 2015-09-08 00:00:00.000 257 3304 1877 572 234 116 73 128 1421 825 253 62 32 30 2015-09-09 00:00:00.000 257 3314 1891 575 236 120 73 128 1438 828 254 62 33 30 2015-09-10 00:00:00.000 259 3329 1912 584 245 131 76 128 1460 847 259 66 37 31 2015-09-11 00:00:00.000 259 3358 1930 605 248 136 79 128 1475 856 261 67 39 31 2015-09-12 00:00:00.000 259 3397 1953 621 255 139 79 128 1486 864 264 68 41 31 2015-09-13 00:00:00.000 259 3426 1984 642 257 144 80 129 1499 883 277 74 42 32 And I need a column with a SUM of all the columns containing the word F and other containig the word M, instead of using something like this: F#13-17+F#18-24+F#25-34+F#35-44+F#45-54+etc. Is this possible?
Try something like this: with derivedTable as (sql from your question goes here) select column_name from derivedTable union select cast(count(*) as varchar (10) + 'records' from derivedTable
Group clause in SQL command
I have 3 tables: Deliveries, IssuedWarehouse, ReturnedStock. Deliveries: ID, OrderNumber, Material, Width, Gauge, DelKG IssuedWarehouse: OrderNumber, IssuedKG ReturnedStock: OrderNumber, IssuedKG What I'd like to do is group all the orders by Material, Width and Gauge and then sum the amount delivered, issued to the warehouse and issued back to stock. This is the SQL that is really quite close: SELECT DELIVERIES.Material, DELIVERIES.Width, DELIVERIES.Gauge, Count(DELIVERIES.OrderNo) AS [Orders Placed], Sum(DELIVERIES.DeldQtyKilos) AS [KG Delivered], Sum(IssuedWarehouse.[Qty Issued]) AS [Film Issued], Sum([Film Retns].[Qty Issued]) AS [Film Returned], [KG Delivered]-[Film Issued]+[Film Returned] AS [Qty Remaining] FROM (DELIVERIES INNER JOIN IssuedWarehouse ON DELIVERIES.OrderNo = IssuedWarehouse.[Order No From]) INNER JOIN [Film Retns] ON DELIVERIES.OrderNo = [Film Retns].[Order No From] GROUP BY Material, Width, Gauge, ActDelDate HAVING ActDelDate Between [start date] And [end date] ORDER BY DELIVERIES.Material; This groups the products almost perfectly. However if you take a look at the results: Material Width Gauge Orders Placed Delivered Qnty Kilos Film Issued Film Returned Qty Remaining COEX-GLOSS 590 75 1 534 500 124 158 COEX-MATT 1080 80 1 4226 4226 52 52 CPP 660 38 8 6720 2768 1384 5336 CPP 666 47 1 5677 5716 536 497 CPP 690 65 2 1232 717 202 717 CPP 760 38 3 3444 1318 510 2636 CPP 770 38 4 4316 3318 2592 3590 CPP 786 38 2 672 442 212 442 CPP 800 47 1 1122 1122 116 116 CPP 810 47 1 1127 1134 69 62 CPP 810 47 2 2250 1285 320 1285 CPP 1460 38 12 6540 4704 2442 4278 LD 975 75 1 502 502 182 182 LDPE 450 50 1 252 252 50 50 LDPE 520 70 1 250 250 95 95 LDPE 570 65 2 504 295 86 295 LDPE 570 65 2 508 278 48 278 LDPE 620 50 1 252 252 67 67 LDPE 660 50 1 256 256 62 62 LDPE 670 75 1 248 248 80 80 LDPE 690 47 1 476 476 390 390 LDPE 790 38 2 2104 1122 140 1122 LDPE 790 50 1 286 286 134 134 LDPE 790 50 1 250 250 125 125 LDPE 810 30 1 4062 4062 100 100 LDPE 843 33 1 408 408 835 835 LDPE 850 80 1 412 412 34 34 LDPE 855 30 1 740 740 83 83 LDPE 880 60 1 304 304 130 130 LDPE 900 70 2 1000 650 500 850 LDPE 1017 60 1 1056 1056 174 174 OPP 25 1100 1 381 381 95 95 OPP 1000 30 2 1358 1112 300 546 OPP 1000 30 1 1492 1491 100 101 OPP 1200 20 1 418 417 461 462 PET 760 12 3 1227 1876 132 -517 You'll see that there are some materials that have the same width and gauge yet they are not grouped. I think this is because the delivered qty is different on the orders. For example: Material Width Gauge Orders Placed Delivered Qnty Kilos Film Issued Film Returned Qty Remaining LDPE 620 50 1 252 252 67 67 LDPE 660 50 1 256 256 62 62 I would like these two rows to be grouped. They have the same material, width and gauge but the delivered qty is different therefore it hasn't grouped it. Can anyone help me group these strange rows?
Your "problem" is that the deliveries occurred on different dates, and you're grouping by ActDelDate so the data splits, but because you haven't selected the ActDelDate column, this isn't obvious. The fix is: Remove ActDelDate from the group by list You should also remove the unnecessary brackets around the first join, and change HAVING ActDelDate Between [start date] And [end date] to WHERE ActDelDate Between [start date] And [end date] and have it before the GROUP BY
You are grouping by the delivery date, which is causing the rows to be split. Either omit the delivery date from the results and group by, or take the min/max of the delivery date.