Amey Agrawal

Speeding up graphics intense websites

Amey Agrawal — Sat, 07 Jan 2017 00:00:00 GMT

College fests are irreplaceable delights of a college student's life. Each college tries to make its fest bigger yet different than others' and so is case with fest websites. These fest websites stand as expression of creativity of its makers which more often than not leads to a large page load time and poor user experience. While working on websites for <a target='_blank' href="http://bits-oasis.org/2015main/"> Oasis </a> and <a target='_blank' href="http://bits-apogee.org/2016/"> Apogee </a> at Department of Visual Media, I explored a few techniques to make the sites faster. Here are a few of them. Both graphics designers and front-end developers might find the blog helpful.

1. Choose the right file format

This is the most basic yet the most effective step of all. With improved browser rendering engines, SVG files have gained popularity in the web design domain. Being vectors they scale up with screen sizes without any depreciation in quality and they are lightweight. As a bonus they support animations and JavaScript interactions.

But sometimes designs contain textures which cannot be recreated in vectors, in such situations one has to choose between JPEG and PNG? This question does not have a generic answer as JPEG uses lossy data compression unlike PNG and hence is always smaller in size but the lossy compression has it's own implications. For example, text in JPEG images is blurred out on edges. PNG also supports the use of transparency which makes its use as a necessity at times.

Just choosing the right file format can save up of to 80% payload in images.

2. Select the right export settings

Many designers tend to ignore this part which can actually turn out to be very crucial while exporting to raster formats (JPEG and PNG). Typically printers use CMYK colors (32 bit) and hence a lot of design tools export using this format by default. This leads to a problem when rendering on the web as it only supports RGB colors (24 bit) although, a browser would display a CMYK image it cannot reproduce the exact colors. Apart from this, JPEG also supports grayscale colors (8 bit). Progressive and Optimized JPEGs can provide significant reduction in size. As a rule of thumb you might use baseline optimized JPEGs but I strongly suggest going through <a target='_blank' href="https://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiDpYncjLDRAhXFOo8KHVvLBs8QFggZMAA&url=https%3A%2F%2Fforums.adobe.com%2Fthread%2F1962929&usg=AFQjCNGcfh77OnWkOA99LnDRVfTQ7G0GWQ&sig2=5wkjdUoFPIhqeqCC66PFQg"> [1] </a> and <a target='_blank' href="http://superuser.com/questions/379404/what-is-the-difference-between-progressive-and-optimized-jpegs-in-photostop"> [2] </a>.

PNG provides with four color modes of which 24 bit RGB and 8 bit paletted are of interest to us. Typically you would use RGB colors, but if the image does not contain too many colors 8 bit paletted colors might turn out to be better. Converting the image to 8 bit color mode would entail choosing only the best 256 colors, thus drastically decreasing the size of the image. As a word of caution, using paletted color modes can sometimes increase the size of the image when the dimensions of image are small. <a target='_blank' href="http://help.adobe.com/en_US/creativesuite/cs/using/WSC7A1F924-DD38-49b4-B84B-EFF50416C860.html"> [3] </a> is a great read about the tweaks discussed above.

3. Compress your files

Though file compression is the most obvious step in decreasing page load time, it is still not performed right most of the times. For all your text files HTML, Js, CSS, JSON, etc. "minify" (whitespace removal) is the most popular compression technique available. Though it might appear to be elementary, its aggregate impact could be significant. Most modern text editors like Sublime Text and Atom have plugins for minification. You can also add this to your gulp/grunt build process.

Different image formats have different compression techniques and there exist great open source utilities like <a target='_blank' href="http://www.kokkonen.net/tjko/projects.html"> JPEGOptim </a> , <a target='_blank' href="http://www.advsys.net/ken/util/pngout.htm"> PNGOUT </a> and <a target='_blank' href="https://github.com/svg/svgol"> SVGO </a> which can decrease the file size by up to 70%. I personally prefer <a target='_blank' href="https://github.com/toy/image_optim"> image_optim </a> which combines all these packages, provides a handy shell interface which works for all image formats and integrates easily into build processes. But setting up image_optim on Windows is tricky, if you are on Windows you might want to install those utilities individually or you could alternatively use <a target='_blank' href="https://tinypng.com/"> TinyPNG </a> for rasters and <a target='_blank' href="https://jakearchibald.github.io/svgomg/"> SVGOMG </a> for SVGs.

Most compression techniques in the above mentioned utilities are lossy hence deal with them cautiously.

4. Use gzip compression

Browsers can decode gzip encoded responses from a server. gzip does not help much with raster images but manages to compress text files to about one tenth their original size. This provides massive savings with minimum efforts and gzip compression can be easily enabled on both apache and nginx. <a target='_blank' href="https://varvy.com/pagespeed/enable-compression.html"> [4] </a> explains this process in a simple manner.

Fun fact, SVG files are just XML documents because of which gzip works great on them as well. The complete isometric map of the miniature city on the landing page of Apogee 2016 website mentioned earlier is a single ginormous SVG file of about 2 MBs. SVGO compression decreased it to half and ultimately with gzip it is transferred as a payload of only about 200 KBs.

5. Avoid loading multiple fonts

Fonts make websites seem beautiful but they can also make them lethargic. Google Fonts is by and far the best option to load fonts on the web almost all the fonts (or their similar alternatives) are available on Google Fonts. As browsers cache fonts, many a times the Google Fonts font you are trying to load might already be in the user’s cache. Also, we might only need one or two font-weights for a given font which we can choose to load selectively in Google Fonts thus dramatically reducing the size.

If you need a font only for some titles, it might be a good idea to export those as SVGs instead. Though a bit tedious, this leads to big savings as it removes the barrier to your creative ideas.

5. Lazy load files

Loading all the content at once might make the loading times large specially in one page sites. Lazy loading files of less importance might turn out to be useful, <a target='_blank' href="http://sourcey.com/recliner/"> Recliner.js </a> is a great tool to achieve the same.

6. Decrease the number of requests

Decreasing the number of HTTP requests can make your page load significantly faster. <a target='_blank' href="https://www.giftofspeed.com/fewer-http-requests/"> [5] </a> describes different ways you can achieve the same. Apart from the ones described above, SVG sprites are useful in decreasing number of requests. <a target='_blank' href="https://github.com/jkphl/svg-sprite"> svg-sprite </a> is a useful tool for making SVG sprites.

HTTP/2 would completely change this landscape though, you can read more about it in <a target='_blank' href="https://blog.newrelic.com/2016/02/09/http2-best-practices-web-performance/"> [6]</a>.

7. Enable caching

Caching the less frequently changed files makes pages load super-fast for returning users. <a target='_blank' href="https://betterexplained.com/articles/how-to-optimize-your-site-with-http-caching/"> [7]</a> walks through enabling caching in apache, for nginx you might want to refer <a target='_blank' href="https://www.nginx.com/blog/nginx-caching-guide/"> [8]</a>.

I hope that this posts helps in making your sites faster. If you have any suggestions please let me know in the comments below and I will surely take them into consideration next time! :-) Thank you!

Jupyter In Classroom

Amey Agrawal — Thu, 29 Mar 2018 00:00:00 GMT

Prologue

In recent years the popularity of machine learning and related courses has skyrocketed in BITS Pilani. Though during my junior year seven machine learning and data science courses were offered in Pilani, none taught the programming nuances required to effectively use the learnings in practice. I solved assignments from Stanford's CS 231n and CS 224d in my free time to learn NumPy and TensorFlow however with six courses and three term project it was a strenuous task. Hence, Nikhil Verma and I approached our project adviser Prof. Bhanot with some ideas to introduce Python-based learning components in her course on Neural Networks and Fuzzy Logic. Over the summers we charted the plan for the course and decided to have three assignments each followed by a programming test. Along with three other senior year teaching assistants we introduced the updated course during fall 2017.

Original Setup

Jupyter notebooks were the obvious choice for creating assignments. But the students who opted for the course came from diverse backgrounds. About half of them had never programmed in Python, the majority of EE students had programming experience limited to C and Matlab. We did not want them to spend time setting up Python environment. Following are the tools we employed to conduct the programming components considering the aforementioned constraints,

Assignment Distribution: We used Azure Notebooks to distribute assignments and tutorials for the course. Students could directly clone the assignment libraries and click to launch the notebooks.

Evaluating Submissions: An awesome open source project nbgrader turned out to be super helpful in creating and evaluating assignments in Jupyter notebooks.

Test Portal: We needed a way to collect the submissions during the lab tests. nbgrader integrates with JupyterLab but we could not cater to 150 users on the spare machine we had borrowed from one the general computer science labs to host the service. None of the competitive coding platforms provided on option to host Jupyter notebooks up to our knowledge. So we decided to write a simple Express web app which allowed students to upload the solutions as zips. We installed Anaconda 3.6 along with nbopen on each of the lab systems.

The snags

Though Azure notebooks were handy, they were slow, combined with the slow internet connections in hostels they lead to a terrible user experience. During the tests students' uploaded zips, hardly ever followed the instructions. They had different directory structures, multiple copies of same notebook and renamed files. I wrote this shell script to sanitize the submissions by comparing the notebooks to the original problem notebooks. The poor internet connection also made it difficult to download Anaconda and to install TensorFlow, Pytorch and other deep learning libraries which were crucial for the term project.

Enters Callisto

Due to the increased popularity of the course, it was decided to run the course during both the semester. I along with Shrikant Sharda decided to fix the issues we had faced in the previous offering of the course. We need to eliminate the need to hit the internet altogether and make all the resources available on the intranet. We decided to build an electron app, Callisto which would be bundled with Anaconda. We decided to keep most of the back-end components intact form our original express server which used ejs templates. Following are some of the neat features this app facilitated,

Cross-Platform Zero-Setup Jupyter Notebooks: We bundled Anaconda 3.6 and archives for some deep learning libraries along with the app. With some hacks (read installing anaconda on Windows via command line) we were able to ensure that regardless of which platform you are on, opening the app for the first time automatically installs an isolated Anaconda environment with all the necessary libraries with zero efforts.

Azure Notebooks/Google Colab Like Experience But Faster: The electron app communicates with the express server to fetch the assignments and orchestrates Jupyter notebook server. Hence, the user gets the same click to launch notebook experience we loved about Azure notebooks but faster, since the notebook server runs locally on the user's system.

Hassle Free Submission: With the new app, the workflow for students is identical whether they are solving assignments in their hostels or appearing for the test in labs. To submit a notebook the user just has to press the submit button. This also ensures the sanity of submissions.

Bundled Docs: We included documentations for Python, NumPy, Matplotlib along with several tutorials and cheat sheets in the app so that students do not have to navigate out to look for reference resources.

Improved Performance On Web Server: We still render most views displayed within the app on our back-end web server which enables us to push front-end updates without requiring a user-side update of the app. However, all the static resources are already bundled on the client app which dramatically improved the performance of our server; especially during the first few minutes of the test, when all the users would typically download the archives of Python documentation in the old workflow.

<br>

The YouTube video above provides a tour of all the features for students. The source code for the app and server is available here on GitHub. Apart from the app, we are also running a Jekyll site for the listing of suggested term-project topics here. We used old-school google forms with choice eliminator 2 to allocate the project topics on a first come first serve basis.

Some interesting ideas

Though the app's performance was functionally good during the two lab tests we conducted so far there are quite a few rough edges which need to be fixed. But following are some of the more interesting ideas we think could be incorporated in the app in future.

Remote Storage of Notebooks: Currently the notebooks are stored locally and any changes students make in their assignment notebooks do not reflect when logged in from another machine. Syncing the files to the remote server could fix this problem.

Consensus Protocols For Real-Time Grading: One of the biggest downsides of our current setup is that students cannot check if their submission passed the hidden test cases and receive the scores only after we manually run nbgrader on their submissions. One possible way to overcome the limitation put forward by the lack of server-side computer resources could be to use all the client systems with consensus protocol.

Future of The Project

I would graduate from BITS next month and would be focusing more on my research projects here on. If anyone finds this project useful please help us maintain and improve it.

A Data Driven Analysis of Indian Engineering Colleges

Amey Agrawal — Sat, 12 May 2018 00:00:00 GMT

During the recent protests against the fee hike at BITS Pilani, certain students claimed that the quality of education at the institute had fallen in the recent years. The rankings released under the National Institute Ranking Framework (NIRF) 2018 were cited to corroborate the claim. BITS is ranked at 17th overall and 26th by research in the MHRD approved rankings, hence I decided to investigate the ranking parameters. NIRF provides a rough outline of the methodology used in this document. The data provided by the institutes is also available for download in PDFs. However some of the critical details are missed out in the documentation, prohibiting the re-implementation of the ranking scores. Nirant Kasliwal provided an insightful analysis of some of the aspects of the ranking in his quora answer. I would only focus on certain aspects. Some of the following results also provide hints to why Indian academic research ecosystem fails to perform at par with it's international counterparts.

Understanding the research score criterion

NIRF defines Research and Professional Practice (RP) score which reflects the publications and patents published along with sponsored research and consultancy projects undertaken in past three years. RP is a combination of four individual matrices defined as,

$$ RP = PU + QP + IPR + FPPP $$

Combined metric for publications (PU)

$$ PU = 35 × f(P/F_{RQ}) $$

Here, $$P$$ is the total number of publications and $$F_{RQ}$$ is the nominal number of faculty members. $$f$$ is some function whose value lies between $$[0, 1]$$, however the exact definition is missing.

Combined metric for quality of publications (QP)

$$ QP = 20 × f (CC/P) + 20× f (TOP25P/P) $$

Here, $$CC$$ is Total Citation Count over previous three years and $$TOP25P$$ is the number of citations in top 25 percentile averaged over the previous three years.

Patents published and granted (IPR)

$$ IPR = 10× f (PG) + 5 × f (PP)$$

$$PG$$ is the number of patents granted while $$PP$$ is the number of patents published over the previous three years.

Footprint of projects and professional practice (FPPP)

$$ FPPP = 7.5 × f (RF) + 2.5 × f (CF) $$

Where, $$RF$$ is average annual research funding earnings and $$CF$$ annual consultancy amount per faculty at institute level in previous three years.

Factors which decide the rankings

The largest contributor to the final score is the quality index, $$QP$$ which caries forty-percent weightage. The following graph represents the average number of citations per paper.

One can notice that the variation in the average number of citations is small and mostly lies between two and four citations. This implies that there is only a small variation in the quality of an average publication among the top 100 colleges. Hence, we would expect $$QP$$ to not cause as much of difference. The small weight for $$IPR$$ and $$FPPP$$ makes them insignificant while analyzing the overall trend of RP rankings. $$PU$$ or the average publications per faculty seems to be the single biggest factor in deciding the RP rankings. Since, the function $$f$$ is not defined in the document I assume it to be the min-max scaling function for rest of this article.

$$ f(X) = \frac{X - X_{min}}{X_{max} - X_{min}} $$

At this point one might be tempted to think why are we considering the publication per faculty and not the total number of publications. This is probably to accommodate for the different sizes of colleges. The following plot shows the number of variation of publication per faculty with change in number of faculty and change in number of PhD students. The color of marker represents the RP score.

We notice that as per our assumption higher publication per faculty leads to higher RP scores. Also, the metric does not show any direct trend with the number of faculty members as per intended. But a clear trend is observable when compared by the total number of PhD students. Higher the number of PhD scholars leads to higher per faculty publication and hence higher RP. Since PhD students are directly responsible for publications this seems logical to compare publications per PhD student instead of faculty.

The PhD student crisis

PhD students drive the research outcome at any university. Only five colleges among NIRF's list of top hundred engineering colleges have a healthy average of more than four PhD students per faculty. 37 colleges have less than a single PhD student per faculty.

For reference MIT's school of engineering admitted 2105 PhD students in 2017-2018 alone [1], while it has 378 faculty members [2]. Similarly, UC Berkeley engineering has 229 faculty [3] and enrolled 1332 PhD students in Fall 2017 [4].

Notably, a new UGC guidelines which restricts the number a professor, an associate professor and an assistant professor can guide to eight, six and four respectively lead to a strike in JNU [5] last year.

BITS Pilani is one of the worst hit universities with a mere PhD to faculty ratio of 0.41. Probably some of the readers know more about why BITS has so few PhD scholars. The following plot shows the top 20 colleges ranked by publication per PhD student over last three years.

Accommodating for the skew

To equate for this high imbalance in the number of PhD students I defined a function which quantifies the productivity of a faculty member as a function of number of PhD students.

We can use the above stated function to define 'Effective Faculty Units' ($$EFU$$) as the product of total number of faculty and productivity function. Also, the original $$IPR$$ score does not factor for the number of faculty so I decided to update the metric to factor in for $$EFU$$ as well.

$$RP_{EFU} = PU_{EFU} + QP + IPR_{EFU} + FPPP_{EFU}$$

$$PU_{EFU} = 35 × f(P/EFU)$$

$$IPR_{EFU} = 10× f(PG/EFU) + 5 × f(PP/EFU)$$

$$FPPP_{EFU} = 7.5 × f (\frac{RF * F}{EFU}) + 2.5 × f (\frac{CF * F}{EFU})$$

The following plot shows that unlike $$RF$$, $$RF_{EFU}$$ is not biased towards colleges with large number of PhD students.

Note: Here on for most of the analysis we only consider the colleges which have more than two hundred faculty members and hundred PhD students.

Does our metric capture the quality of research?

One concern I had while adding more importance to publications per PhD student was, what if the pressure of publishing more on a PhD student adversely affect the quality of publications. As it turns out $$RP_{EFU}$$ captures the quality of publication pretty well.

The above plot shows that the citations per paper increase with increasing $$RP_{EFU}$$ score. The plot also shows that a lower PhD student to faculty ratio (denoted by color) does not imply lower publication quality.

Following are the top twenty colleges based on $$RP_{EFU}$$.

Please note that some of the institutes listed are barely above the threshold of size specified earlier.

Graduate outcome

The graduate outcome metric puts twenty percent weight on average number of PhD scholars graduated in past three years. Which makes quite biased towards institutes with large number of PhD students. It also involves a term considering the number of students who opted higher studies in past years, however the document does not provide sufficient information on how this number is calculated. However, we do get some interesting placement statistics.

The above plot shows the top 20 colleges based on the median salary of placed students. However, these numbers should be taken with a grain of salt as they do not carry sector-wise figures.

When plotted against the median salary the popularly known 'premier' institutes of India become distinctly recognizable.

Perception score

Perception score only counts 10% towards the final ranking but is probably the most biased component of the rankings. The methodology for calculating perception score is not documented. Commonly Indian engineering colleges are perceived in tiers, the tier one institutes consisting of old IITs, tier two involving NITs and so on. However, the perception score is modeled as an exponentially decreasing function, which might lead to larger changes in overall ranking despite it's small weight. The last plot also depicts the bias in perception score.

Scholarships

Scholarships provides for welfare of underprivileged students. But it also serves as a means to attract students. Surprisingly, the top ranking colleges are not the ones receiving the largest government scholarships.

Epilogue

BITS is home to some of the professors and students I deeply admire. The significantly lower research funding and lack of PhD students makes it much harder for professors to do research. But still we do perform fairly well when compared in fair conditions. Maintaining the quality of education and research is a continuous effort and requires a combined effort from administration, faculty members and students. Those who feel that the quality of education at BITS has declined should find ways contributing towards it's improvement, you would discover some of the most supportive teachers.

I have added many more interesting plots and tables here. The code for this project is available on GitHub. Although I only presented analysis for engineering colleges here, the parsers works for all categories in the ranking framework.

With <span style="color:red; font-size:24px">♥</span> for BITS, Signing off 2014A7PS0148P.

Geometric Transformations with Autoencoders

Amey Agrawal — Mon, 27 Aug 2018 00:00:00 GMT

During my last semester at BITS I was working with Prof. Tiwari on indexing iris images. Unfortunately, given the short time frame, we could not complete the project, however here I would share some experiments I enjoyed performing.

The traditional pre-processing pipeline for iris images includes two stages, segmentation and normalization with Daugman's rubber sheet model.

<figure> <img style="width:70%" src="/images/posts/transforms/daugman_1.png" /> <figsource>Credits: <a target="_blank" href="https://www.worldscientific.com/doi/abs/10.1142/S0218001415560169">Al-Zubi et al.</a></figsource> <figcaption><b>Figure 1:</b> Daugman's rubber sheet model</figcaption> </figure>

<figure> <img src="/images/posts/transforms/daugman_2.jpeg" /> <figsource>Credits: <a target="_blank" href="https://ieeexplore.ieee.org/document/5304768/">Han et al.</a></figsource> <figcaption><b>Figure 2:</b> Normalized iris image</figcaption> </figure>

What makes dealing with normalized iris images difficult is the horizontal translation which exists between the normalized samples of the same iris. Hom et al. use Fully Convolutional Network (FCN) to obtain a 2-D feature map and accounts for the translation at the matching time. But this approach has an O(n) time complexity making it less suitable for matching indexing of large database. We wanted to use dense features with a staged regression model as described by Kraska et al. for a possibility of O(1) lookup.

<figure> <img style="width:58%" src="/images/posts/transforms/horizontal-translation-iris.png" /> <figsource>Credits: <a target="_blank" href="http://openaccess.thecvf.com/content_ICCV_2017/papers/Zhao_Towards_More_Accurate_ICCV_2017_paper.pdf"> Hom et al.</a></figsource> <figcaption><b>Figure 3:</b> Horizontal translation in normalized iris images</figcaption> </figure>

In their paper on robotic grasping Levin et al. add a motor command vector, a one-hot vector to an intermediate layer in CNN by tiling it to the same dimensions. I decided to use this cool trick to train an autoencoder like architecture to perform translation and hopefully learn some translation invariant features.

<figure> <img style="width:50%" src="/images/posts/transforms/vector-addtion-block.png" /> <figcaption><b>Figure 4:</b> Vector addition block</figcaption> </figure>

Before trying with the iris images I decided to verify if the technique could work on simpler dataset. So I did a couple of experiments with MNIST and Fashion-MNIST. We train the encoder in the standard way, however we pass an extra 14-D one-hot vector to the decoder which represents the number of pixels the image is supposed to be translated by. In case of MNIST since every image is 28x28 pixels, we allow shift in quantums of 2 pixels hence creating 14 different possible outcomes.

<figure> <img src="/images/posts/transforms/mnist-translation.png" /> <figcaption><b>Figure 5:</b> Schematic representation of MNIST translation network</figcaption> </figure>

To prepare the training data we create five translated images with random corresponding every image in the MNIST train split.

def translate_data(X, number_of_samples=5):
    """
    Takes images from MNIST dataset as input
    and transforms them to create new to datset.

    :param X: (n, 28, 28) array containing original MNIST images
    :param number_of_samples: Number of samples to be generated from
                              each input image.
    :return: tuple of ((new_x, translation_vectors), new_y)
            new_x: (n * number_of_samples, 28, 28) array containing
                         images from the original datset.
            translation_vectors: (14,) one-hot array containing the amount
                                of translation applied to each corresponding
                                image in new_y
            new_y: (n * number_of_samples, 28, 28) array containing
                         translated images.
    """
    new_x = []
    new_y = []
    translation_vectors = []

    for i in range(X.shape[0]):
        for _ in range(number_of_samples):
            translation = np.random.randint(0, 14) * 2
            # Copy image to new_x as it is
            new_x.append(X[i])
            # Perform translation and add image to new_y
            new_y.append(
                np.hstack([X[i, :, translation:],
                           X[i, :, :translation]]))
            translation_vector = np.zeros(14)
            translation_vector[translation // 2] = 1
            translation_vectors.append(translation_vector)

    new_x, translation_vectors, new_y = np.asarray(new_x), \
                                        np.asarray(translation_vectors), \
                                        np.asarray(new_y)

    return ((new_x, translation_vectors), new_y)

We then define the encoder network as a simple CNN,

def get_encoded(input_img):
    """
    Defines the encoder network.
    :param input_image: A tensor containing input image
    :return: Tensor representing the encoded image
    """
    x = Conv2D(16, (3, 3), activation='relu', padding='same')(input_img)
    x = BatchNormalization()(x)
    x = MaxPooling2D((2, 2), padding='same')(x)
    x = Conv2D(14, (3, 3), activation='relu', padding='same')(x)
    x = MaxPooling2D((2, 2), padding='same')(x)
    x = Conv2D(14, (3, 3), activation='relu', padding='same')(x)
    x = BatchNormalization()(x)
    x = MaxPooling2D((2, 2), padding='same')(x)
    x = Conv2D(14, (3, 3), activation='relu', padding='same')(x)
    encoded = MaxPooling2D((2, 2), padding='same')(x)

    return encoded

Since defining the decoder all at once could be a mouthful we create a helper method which returns us the convolution and addition block.

def convolve_and_add(x, translation_vector_reshaped, image_dim, num_filters=14):
    """
    Defines the convolution and add block.
    :param x: Input tensor
    :param translation_vector_reshaped: Shape input reshaped as (1, 1, 14) tensor
    :param image_dim: The dimension of input tensor, assuming the
                      image hight and width are same
    :param num_filters: Number of filters in each convolution layer
    :return: Output tensor
    """
    #Sun
    get_tiling_lambda = lambda: Lambda(lambda x: K.tile(x, [1, image_dim, image_dim, 1]))
    
    x = Conv2D(num_filters, (3, 3), activation='relu', padding='same')(x)
    translation_vector_tiled = get_tiling_lambda()(translation_vector_reshaped)
    x = Add()([x, translation_vector_tiled])
    x = Conv2D(num_filters, (3, 3), activation='relu', padding='same')(x)
    translation_vector_tiled = get_tiling_lambda()(translation_vector_reshaped)
    x = Add()([x, translation_vector_tiled])
    x = Conv2D(num_filters, (3, 3), activation='relu', padding='same')(x)
    translation_vector_tiled = get_tiling_lambda()(translation_vector_reshaped)
    x = Add()([x, translation_vector_tiled])
    x = BatchNormalization()(x)

    return x

Then we define decoder network using convolve_and_add as,

def get_decoded(encoded, translation_vector):
    """
    Defines the decoder network.
    :param encoded: Tensor representing the encoded image
    :param translation_vector: One-hot vector representing the translation amount
    :return: Decoded image
    """
    translation_vector_reshaped = Reshape((1, 1, 14))(translation_vector)

    #### Block 1 ####
    # Convolution
    x = Conv2D(36, (3, 3), activation='relu', padding='same')(encoded)
    x = Conv2D(36, (3, 3), activation='relu', padding='same')(x)
    x = Conv2D(36, (3, 3), activation='relu', padding='same')(x)
    x = BatchNormalization()(x)
    # Upsample
    x = Conv2D(14, (3, 3), activation='relu', padding='same')(x)
    x = UpSampling2D((2, 2))(x)

    #### Block 2 ####
    # Convolve and Add
    x = convolve_and_add(x, translation_vector_reshaped, 4)
    # Upsample
    x = Conv2D(14, (3, 3), activation='relu', padding='same')(x)
    x = UpSampling2D((2, 2))(x)

    #### Block 3 ####
    # Convolve and Add
    x = convolve_and_add(x, translation_vector_reshaped, 8)
    # Upsample
    x = Conv2D(14, (3, 3), activation='relu', padding='same')(x)
    x = UpSampling2D((2, 2))(x)

    #### Block 4 ####
    # Convolve and Add
    x = convolve_and_add(x, translation_vector_reshaped, 16)
    # Upsample
    x = Conv2D(32, (3, 3), activation='relu')(x)
    x = UpSampling2D((2, 2))(x)

    #### Block 5 ####
    # Convolution
    x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)
    x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)
    x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)

    decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same', name='last_conv')(x)

    return decoded

This gives in surprisingly neat results.

<figure> <img style="width:80%" src="/images/posts/transforms/mnist-translation-output.png" /> <figcaption><b>Figure 6:</b> Result of the autoencoder on translation task</figcaption> </figure>

Then I decided to repeat the experiment but this time trying to rotate the digits.

<figure> <img style="width:80%" src="/images/posts/transforms/mnist-rotation-output.png" /> <figcaption><b>Figure 7:</b> Result of the autoencoder on rotation task</figcaption> </figure>

Then as a harder benchmark I tried to train it on Fashion-MNIST. On the translation task we lose some details but it gets the outlines right for the most part. As our primary objective was translation, I did not repeat the experiment on rotation task.

<figure> <img style="width:80%" src="/images/posts/transforms/fmnist-translation-output.png" /> <figcaption><b>Figure 8:</b> Result of the autoencoder on Fashion-MNIST translation task</figcaption> </figure>

The Google Colab notebook for the MNIST translation task is available here, where you can train the network with cloud GPUs. Though, ultimately I was unable to use the method on iris images, I thoroughly enjoyed performing these experiments.