Skip to content

Commit d9173bd

Browse files
dandohdandoh
authored
Merge pull request #18 from McMasterU/auto_download_solver
Auto download solver
2 parents 3651e9b + b1eb1f5 commit d9173bd

38 files changed

+5930
-937
lines changed

‎Contributing.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,3 +40,15 @@ The expression is now a hash table with a root node ID.
4040
Or explicitly:
4141
![Hash explicit](examples/Hash_Explicit.png)
4242

43+
## Generating Haddock (with Docker)
44+
Build the docker image located in docs (it's important you do this from the root of the repo), with
45+
```terminal
46+
docker build -t hashed-docker -f docs/Dockerfile .
47+
```
48+
-Then run the docker container to generate the haddock documentation (NOTE: every time you alter the
49+
-code base you'll have to rebuild the image)
50+
```terminal
51+
-docker run -v /some/path:/home/HashedExpression/docs hashed-docker
52+
```
53+
-this will generate all the haddock documentation (in html) into */some/path* on your local system
54+

‎Dockerfile‎

Lines changed: 15 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -1,33 +1,15 @@
1-
# FROM ubuntu:18.04
2-
3-
# RUN apt-get update
4-
# RUN apt-get install -y tzdata
5-
# RUN apt-get install -y haskell-stack
6-
# RUN apt-get install -y coinor-libipopt1v5
7-
# RUN apt-get install -y coinor-libipopt-dev
8-
# RUN apt-get install -y netbase
9-
# RUN apt-get install -y zlib1g-dev
10-
11-
# RUN stack upgrade
12-
# ENV PATH="/root/.local/bin:${PATH}"
13-
14-
# WORKDIR /home/HashedExpresion
15-
# COPY . ./
16-
17-
# # FIXME stack build fails first time, but succeeds second? (needs to download keys??)
18-
# RUN $(stack build || stack build) && stack install
19-
FROM haskell:8.8.3
20-
21-
WORKDIR /home/HashedExpression
22-
COPY . ./
23-
24-
# RUN stack upgrade
25-
26-
27-
# Docker will cache this command as a layer, freeing us up to
28-
# modify source code without re-installing dependencies
29-
# (unless the .cabal file changes!)
30-
RUN stack install
31-
RUN mkdir /target
32-
33-
ENTRYPOINT cd /target && symphony *.sp
1+
# Setup the environment needed for Symphony
2+
FROM ubuntu:18.04
3+
4+
RUN apt-get update
5+
RUN apt-get install -y tzdata
6+
RUN apt-get install -y pkg-config
7+
RUN apt-get install -y coinor-libipopt1v5
8+
RUN apt-get install -y coinor-libipopt-dev
9+
RUN apt-get install -y libhdf5-dev
10+
RUN apt-get install -y libfftw3-dev
11+
RUN apt-get install -y build-essential
12+
RUN apt-get install -y curl
13+
14+
ENV CPATH /usr/include/coin:/usr/include/hdf5/serial
15+
ENV LIBRARY_PATH /usr/lib/x86_64-linux-gnu/hdf5/serial

‎HashedExpression.cabal‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ cabal-version: 1.12
44
--
55
-- see: https://github.com/sol/hpack
66
--
7-
-- hash: 158d5ed7ffd77d8520dc6eedabb8d47f1240293abc2485d2358bc8fa6a262c85
7+
-- hash: 8cd28a087f508490f3891da8fc8a23620f27f29dc9d0598d42e970b328291f71
88

99
name: HashedExpression
1010
version: 0.1.0.0
@@ -22,7 +22,6 @@ extra-source-files:
2222
symphony/ComposOp.hs
2323
symphony/ErrM.hs
2424
symphony/LayoutHashedLang.hs
25-
symphony/LexHashedLang.hs
2625
symphony/Main.hs
2726
symphony/ParHashedLang.hs
2827
symphony/PrintHashedLang.hs
@@ -44,7 +43,7 @@ library
4443
HashedExpression.Differentiation.Exterior.Derivative
4544
HashedExpression.Differentiation.Reverse
4645
HashedExpression.Differentiation.Reverse.State
47-
HashedExpression.Embed.FFTW
46+
HashedExpression.Embed
4847
HashedExpression.Internal
4948
HashedExpression.Internal.Collision
5049
HashedExpression.Internal.Expression
@@ -79,6 +78,7 @@ library
7978
, deepseq
8079
, easyplot
8180
, extra
81+
, file-embed
8282
, filepath
8383
, mtl
8484
, process
@@ -111,6 +111,7 @@ executable HashedExpression-exe
111111
, deepseq
112112
, easyplot
113113
, extra
114+
, file-embed
114115
, filepath
115116
, hspec
116117
, mtl
@@ -130,14 +131,14 @@ executable symphony
130131
ErrM
131132
LayoutHashedLang
132133
LexHashedLang
133-
LexHashedLang
134134
ParHashedLang
135135
ParHashedLang
136136
PrintHashedLang
137137
SkelHashedLang
138138
Symphony.Common
139139
Symphony.Entry
140140
Symphony.Exp
141+
Symphony.Solver
141142
Symphony.Symphony
142143
Paths_HashedExpression
143144
hs-source-dirs:
@@ -159,6 +160,7 @@ executable symphony
159160
, deepseq
160161
, easyplot
161162
, extra
163+
, file-embed
162164
, filepath
163165
, mtl
164166
, optparse-applicative
@@ -205,6 +207,7 @@ test-suite HashedExpression-test
205207
, deepseq
206208
, easyplot
207209
, extra
210+
, file-embed
208211
, filepath
209212
, hspec
210213
, mtl

‎README.md‎

Lines changed: 53 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -12,98 +12,81 @@ $ stack install --ghc-options -O2
1212
```
1313

1414
### Usage
15-
Below is an example of an optimization problem to reconstruct image from loss MRI signal, details about the problem can be found at
16-
[MRI-Image-Reconstruction](examples/MRI-Image-Reconstruction.pdf).
15+
Consider minimizing the negative entropy function ![image](https://latex.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Chuge%20f%28p%29%20%3D%20%5Csum_%7Bi%20%3D%201%7D%5En%20p_i%20%5Clog%28p_i%29)
1716

17+
Create `entropy.sp`:
1818
```haskell
1919
variables:
20-
x[128][128] = 0
20+
p[10][10]
2121

22-
constants:
23-
im[128][128] = Dataset("data.h5", "im")
24-
re[128][128] = Dataset("data.h5", "re")
25-
signal[128][128] = Dataset("data.h5", "signal")
26-
xLowerBound[128][128] = Dataset("data.h5", "x_lb")
27-
xUpperBound[128][128] = Dataset("data.h5", "x_ub")
22+
constraints:
23+
p >= 0.1
2824

29-
constraints:
30-
x >= xLowerBound, x <= xUpperBound
25+
minimize:
26+
p <.> log (p)
3127

32-
let:
33-
smootherX = rotate (0, 1) x + rotate (0, -1) x - 2 *. x
34-
smootherY = rotate (1, 0) x + rotate (-1, 0) x - 2 *. x
35-
regularization = huberNorm 2 smootherX + huberNorm 2 smootherY
28+
solver: lbfgs-b
29+
```
30+
(<.> is dot product)
3631

37-
minimize:
38-
norm2square ((signal +: 0) * (ft x - (re +: im))) + 3000 *. regularization
32+
Run symphony:
33+
```terminal
34+
$ symphony entropy.sp
3935
```
40-
Running:
4136

37+
Symphony will generate code and download the solver (L-BFGS-B in this case) to current working directory.
38+
Then simply compile the code and run:
4239
```terminal
43-
$ symphony mri.sp
40+
$ make
4441
```
45-
Will generate `problem.c` with the following interface:
42+
(Note that we use [HDF5](https://www.hdfgroup.org/solutions/hdf5/) for reading and writing dataset)
4643

47-
```c++
48-
#define NUM_VARIABLES 1
49-
#define NUM_ACTUAL_VARIABLES 16384
50-
#define MEM_SIZE 671774
44+
```terminal
45+
$ ./lbfgs-b
46+
....
47+
....
48+
F = final function value
49+
* * *
50+
N Tit Tnf Tnint Skip Nact Projg F
51+
100 7 8 45 0 0 4.36e-06 -3.67879e+01
52+
21
53+
Total User time 2.470e-04 seconds.
54+
Writing p to p_out.h5...
55+
```
56+
Which is what we expected, and the output value of `p` is written in HDF5 file `p_out.h5`.
5157

52-
// all the actual double variables are allocated
53-
// one after another, starts from here
54-
#define VARS_START_OFFSET 0
5558

59+
For more, checkout `examples/Brain/brain.sp` example which solves an optimization problem to reconstruct image from loss MRI signal, details about the problem can be found at [MRI-Image-Reconstruction](examples/MRI-Image-Reconstruction.pdf).
60+
```haskell
61+
variables:
62+
x[128][128] = 0
5663

57-
const char* var_name[NUM_VARIABLES] = {"x"};
58-
const int var_num_dim[NUM_VARIABLES] = {2};
59-
const int var_shape[NUM_VARIABLES][3] = {{128, 128, 1}};
60-
const int var_size[NUM_VARIABLES] = {16384};
61-
const int var_offset[NUM_VARIABLES] = {0};
62-
const int partial_derivative_offset[NUM_VARIABLES] = {65543};
63-
const int objective_offset = 81927;
64-
double ptr[MEM_SIZE];
64+
constants:
65+
im[128][128] = Dataset("kspace.h5", "im")
66+
re[128][128] = Dataset("kspace.h5", "re")
67+
mask[128][128] = Dataset("mask.h5", "mask")
68+
xLowerBound[128][128] = Dataset("bound.h5", "lb")
69+
xUpperBound[128][128] = Dataset("bound.h5", "ub")
6570

71+
constraints:
72+
x >= xLowerBound, x <= xUpperBound
6673

67-
const int bound_pos[NUM_VARIABLES] = {0};
68-
double lower_bound[NUM_ACTUAL_VARIABLES];
69-
double upper_bound[NUM_ACTUAL_VARIABLES];
74+
let:
75+
smootherX = rotate (0, 1) x + rotate (0, -1) x - 2 *. x
76+
smootherY = rotate (1, 0) x + rotate (-1, 0) x - 2 *. x
77+
regularization = norm2square smootherX + norm2square smootherY
7078

71-
...
7279

73-
void evaluate_partial_derivatives_and_objective() { .. } ;
74-
void evaluate_objective() { .. };
75-
void evaluate_partial_derivatives() { .. } ;
76-
...
80+
minimize:
81+
norm2square ((mask +: 0) * (ft x - (re +: im))) + 3000 *. regularization
82+
83+
solver: lbfgs-b
7784
```
78-
Which you can plug to your favorite optimization solver.
79-
We provide several optimization solvers (LBFGS, LBFGS-b, Ipopt) adapter in `algorithms` directory,
80-
e.g: [LBFGS-b](https://github.com/dalvescb/HashedExpression/blob/master/algorithms/lbfgs-b/lbfgs-b.c).
8185

82-
### Usage Docker
83-
To use symphony through docker (if you wish to avoid installing stack), build the docker image with
84-
```terminal
85-
$ docker build . -t symphony
86-
```
87-
Then run the docker container. In order to provide the input file to the container, you'll have to link
88-
the path (say */some/path/*) containing the the symphony file on your host machine to the */target* path
89-
in the container
90-
```terminal
91-
docker run -v /some/path:/target symphony
92-
```
93-
Copy the resulting */some/path/problem.c* into your chosen *algorithms* directory (i.e ipopt, lbfgs, etc),
94-
which contain their own Dockerfile's for executing the optimization problem (see respective README's)
9586

96-
## Generating Haddock (with Docker)
97-
Build the docker image located in docs (it's important you do this from the root of the repo), with
98-
```terminal
99-
docker build -t hashed-docker -f docs/Dockerfile .
100-
```
101-
-Then run the docker container to generate the haddock documentation (NOTE: every time you alter the
102-
-code base you'll have to rebuild the image)
103-
```terminal
104-
-docker run -v /some/path:/home/HashedExpression/docs hashed-docker
105-
```
106-
-this will generate all the haddock documentation (in html) into */some/path* on your local system
87+
### Docker
88+
We provide ready-to-use and up-to-date docker image [hashexpression/symphony](https://hub.docker.com/r/hashexpression/symphony) which has
89+
`symphony` and all prerequisites installed. All you need is pull and create a container.
10790

10891
## Contributing
10992
Please read `Contributing.md`. PRs are welcome.

‎algorithms/ipopt/Dockerfile‎

Lines changed: 0 additions & 19 deletions
This file was deleted.

‎algorithms/ipopt/README.md‎

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,14 @@
11
# Ipopt
22
## Prerequisite
3-
- FFT Library ( [http://www.fftw.org/]() )
43
- hdf5
54
- MacOS: `brew install hdf5`
65
- Ubuntu: ` sudo apt-get install libhdf5-serial-dev`
76
- Ipopt
7+
- FFT Library ( [http://www.fftw.org/]() ) if involves fourier transform
88
## Usage
99
- Drop in *problem.c* (generated by symphony)
1010
- Run
1111
```terminal
1212
$ make
1313
```
1414
- Which should generate an executable *ipopt*
15-
## Docker Usage
16-
- Drop in *problem.c* (generated by symphony)
17-
- Build docker image, for example
18-
```terminal
19-
docker build . -t ipopt
20-
```
21-
- then run in a container with
22-
```terminal
23-
docker run -it ipopt
24-
```

‎algorithms/lbfgs-b/README.md‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,14 @@
1-
# L-BFGS with Box constraint
1+
# L-BFGS with Box constraints
22
## Prerequisite
3-
- FFT Library ( [http://www.fftw.org/]() )
43
- hdf5
54
- MacOS: `brew install hdf5`
65
- Ubuntu: ` sudo apt-get install libhdf5-serial-dev`
7-
6+
- FFT Library ( [http://www.fftw.org/]() ) if involves fourier transform
7+
## Usage
8+
- Drop in *problem.c* (generated by symphony)
9+
- Run
10+
```terminal
11+
$ make
12+
```
13+
- Which should generate an executable *lbfgs-b*
14+

src/HashedExpression/Embed/FFTW.hs renamed to embed/fftw.c

Lines changed: 0 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,3 @@
1-
module HashedExpression.Embed.FFTW where
2-
3-
import Data.String.Interpolate
4-
5-
-- | For computing fourier transform
6-
fftUtils :: String
7-
fftUtils =
8-
[i|
91
/**
102
* ---------------------------FOURIER TRANSFORM HELPERS---------------------------------
113
* ---------------------------------------START-----------------------------------------
@@ -114,4 +106,3 @@ void im_dft_twice_2d(int ROW, int COLUMN, double *in, double *out) {
114106
/**
115107
* ---------------------------------------END-------------------------------------------
116108
**/
117-
|]

0 commit comments

Comments
 (0)