Academia.eduAcademia.edu

Outline

Title
Abstract
Figures
Introduction
Related Work
Dataset
Methodology
Canonical Correlation Analysis
Results and Discussion
Conclusions
References
All Topics
Computer Science
Computer Vision

Fashion Meets Computer Vision and NLP at e-Commerce Search

Profile image of Susana ZoghbiSusana ZoghbiProfile image of Juan Carlos GomezJuan Carlos Gomez
https://doi.org/10.17706/IJCEE.2016.8.1.31-43
visibility

…

description

13 pages

link

1 file

Abstract

In this paper, we focus on cross-modal (visual and textual) e-commerce search within the fashion domain. Particularly, we investigate two tasks: 1) given a query image, we retrieve textual descriptions that correspond to the visual attributes in the query; and 2) given a textual query that may express an interest in specific visual product characteristics, we retrieve relevant images that exhibit the required visual attributes. To this end, we introduce a new dataset that consists of 53,689 images coupled with textual descriptions. The images contain fashion garments that display a great variety of visual attributes, such as different shapes, colors and textures in natural language. Unlike previous datasets, the text provides a rough and noisy description of the item in the image. We extensively analyze this dataset in the context of cross-modal e-commerce search. We investigate two state-of-the-art latent variable models to bridge between textual and visual data: bilingual latent Dirichlet allocation and canonical correlation analysis. We use state-of-the-art visual and textual features and report promising results.

Figures (9)
Fig. 1. Our system performs two cross-modal tasks: image to text and text to image. Cross-modal e-commerce fashion search has not received much attention in the literature. Existing works 1]-[3] mostly focus on classifying fashion visual content into a set of predefined categories. One exception ; [4], where the authors focus on automatically annotating images of shoes and bags, but fashion garments nd the Txt2Img task are not investigated.
Fig. 1. Our system performs two cross-modal tasks: image to text and text to image. Cross-modal e-commerce fashion search has not received much attention in the literature. Existing works 1]-[3] mostly focus on classifying fashion visual content into a set of predefined categories. One exception ; [4], where the authors focus on automatically annotating images of shoes and bags, but fashion garments nd the Txt2Img task are not investigated.
Table 1. Statistics on the Training Datasets after Projection on POS- and Zappos-Based Vocabularies
Table 1. Statistics on the Training Datasets after Projection on POS- and Zappos-Based Vocabularies
EDIE II II OIE OI II II II EI IIE NIE EE IEE Regarding visual features, the overall best performer is the convolutional neural network (CNN). For. iven fixed triad of (model, vocabulary, K), the CNN feature always outperforms the SIFT feature in bot recision and recall. For example, for the POS vocabulary, at K=5, BiLDA-SIFT achieves 19.97% precisio! nd 10.34% recall; whereas BiLDA-CNN achieves 24.04% in precision (20% increase) and 11.96% in recal 15% increase). It is remarkable that CNNs perform so well compared to the SIFT counterparts becaus: 1ey were not trained for this particular task. Instead, as previously described, the model that generate 1em was trained on a large image classification task [22]. In the future, we will explore fine-tuning th NNs to improve further performance.
EDIE II II OIE OI II II II EI IIE NIE EE IEE Regarding visual features, the overall best performer is the convolutional neural network (CNN). For. iven fixed triad of (model, vocabulary, K), the CNN feature always outperforms the SIFT feature in bot recision and recall. For example, for the POS vocabulary, at K=5, BiLDA-SIFT achieves 19.97% precisio! nd 10.34% recall; whereas BiLDA-CNN achieves 24.04% in precision (20% increase) and 11.96% in recal 15% increase). It is remarkable that CNNs perform so well compared to the SIFT counterparts becaus: 1ey were not trained for this particular task. Instead, as previously described, the model that generate 1em was trained on a large image classification task [22]. In the future, we will explore fine-tuning th NNs to improve further performance.
Regarding the preprocessing, we observe that for this task the Zappos vocabulary performs much better than the POS-based counterpart. This makes sense, since the Zappos vocabulary is much more limited (around 200 unique tokens) than the POS-based vocabulary (over 9000 unique tokens). Having a limited, yet meaningful and complete vocabulary is beneficial for this task. In particular, the Zappos vocabulary is quite interesting because it contains the actual categories that a real-life online shop uses to manually categorize its apparel dress garments.
Regarding the preprocessing, we observe that for this task the Zappos vocabulary performs much better than the POS-based counterpart. This makes sense, since the Zappos vocabulary is much more limited (around 200 unique tokens) than the POS-based vocabulary (over 9000 unique tokens). Having a limited, yet meaningful and complete vocabulary is beneficial for this task. In particular, the Zappos vocabulary is quite interesting because it contains the actual categories that a real-life online shop uses to manually categorize its apparel dress garments.
‘I'7o Increase) and 11.96'/1n recall (65 Increase). simUar behaviors are observed as K Increases. Furthermore, when we compare our best system (BiLDA-CNN-Zappos) with the setup ir 6](BiLDA-Sift-POS) we obtain remarkable improvements. For example, at K=5, [16] obtains 24.04% ‘ecision and 11.97% recall. In contrast our system obtains 48.75% precision (102% increase) and 43.66% call (364% increase). Similar observations can be made for other values of K, as shown in Fig. 2 and Fig. 3. Of course, our best system benefits from a more targeted vocabulary (Zapos-based). The natural questio1 which of these two vocabularies to use if we were to deploy this application. In general, there is a tradeof tween the size of the vocabulary and expressiveness. The larger the vocabulary, the more likely we are tc pture the nuances in visual attributes. For example, we might have many more distinct words tc fferentiate varied shades of the same color, as opposed to one single word that encompasses the whole yectrum. We might also be able to differentiate in a more detailed way types of textures and shapes owever, as the vocabulary increases, the task becomes more difficult because we might not have enougt ita to instantiate and consequently learn all these nuances. This explains the large differences ir srformance between the two vocabularies. However, regardless of the choice of the vocabulary, the clea p choice of visual features is the CNN and not SIFT as previously used in [16].
‘I'7o Increase) and 11.96'/1n recall (65 Increase). simUar behaviors are observed as K Increases. Furthermore, when we compare our best system (BiLDA-CNN-Zappos) with the setup ir 6](BiLDA-Sift-POS) we obtain remarkable improvements. For example, at K=5, [16] obtains 24.04% ‘ecision and 11.97% recall. In contrast our system obtains 48.75% precision (102% increase) and 43.66% call (364% increase). Similar observations can be made for other values of K, as shown in Fig. 2 and Fig. 3. Of course, our best system benefits from a more targeted vocabulary (Zapos-based). The natural questio1 which of these two vocabularies to use if we were to deploy this application. In general, there is a tradeof tween the size of the vocabulary and expressiveness. The larger the vocabulary, the more likely we are tc pture the nuances in visual attributes. For example, we might have many more distinct words tc fferentiate varied shades of the same color, as opposed to one single word that encompasses the whole yectrum. We might also be able to differentiate in a more detailed way types of textures and shapes owever, as the vocabulary increases, the task becomes more difficult because we might not have enougt ita to instantiate and consequently learn all these nuances. This explains the large differences ir srformance between the two vocabularies. However, regardless of the choice of the vocabulary, the clea p choice of visual features is the CNN and not SIFT as previously used in [16].
Fig. 5. Txt2Img: Recall@K for POS-based vocabulary (left) and Zappos-based vocabulary (right). dl Txt2Img Results. Fig. 5 presents recall@K for all conditions. In all instances, our models perform mucl etter than random. This suggests that we have captured meaningful, useful aspects of the data. Regardins visual features, just as in the previous task, the best performer is the convolutional neural network (CNN) ‘or any fixed (model, vocabulary, K) combination, the CNN feature always outperforms the SIFT feature <egarding the choice of model, the BiLDA model performs roughly as well as the CCA model when using the appos vocabularies. However, CCA outperforms BiLDA with the POS-vocabulary. Regarding the choice o vocabulary, we see that performance is generally higher using the POS-based vocabulary. This makes sens« yecause the POS-based vocabulary is much larger (over 9,000 unique tokens) than the Zappos-basec around 200 tokens). Having a larger vocabulary is beneficial to the Txt2Img task because it allows for more “xpressiveness in the required visual attributes. Fig. 6 presents qualitative results, where given a textual query, we show the top 4 images retrieved. We see very interesting results. The query ‘little black dress black polyester jersey lace’ actually finds little black dresses. It can be argued that the retrieved items also display some jersey-type characteristics, especially the first and third items. For the attribute polyester, it is not clear from the image what the fabrics of the garments are, as this is not a particularly visual word. Also, we do not see any laces on the dresses, so there is room for improvement. The query ‘wedding gown sleeveless scalloped ruffles’ retrieved wedding gowns in all four items. Two of them are sleeveless and three of them contain ‘scalloped ruffles’. The query ‘casual sleeveless floral print’ retrieves garments with floral patterns on them. They are sleeveless and casual. The query ‘long cocktail wedding gown strapless yellow ruched’ retrieves yellow items in all cases. Three of them are long and ruched. Two of them are strapless. In this case, the query might be slightly misleading, since it mentions wedding gown, and it is often assumed that that corresponds to an actual white wedding dress. However, the word wedding is often used to describe the occasion that the dress may be worn to. It is similar to the word ‘cocktail’ as it also refers to the occasion. The occasion to which a particular garment is appropriate to wear is of course highly subjective. Overall, these results are highly impressive given the difficulty of the task. A demonstration and workshop Regarding the choice of model, the BiLDA model performs roughly as well as the CCA model when using the
Fig. 5. Txt2Img: Recall@K for POS-based vocabulary (left) and Zappos-based vocabulary (right). dl Txt2Img Results. Fig. 5 presents recall@K for all conditions. In all instances, our models perform mucl etter than random. This suggests that we have captured meaningful, useful aspects of the data. Regardins visual features, just as in the previous task, the best performer is the convolutional neural network (CNN) ‘or any fixed (model, vocabulary, K) combination, the CNN feature always outperforms the SIFT feature <egarding the choice of model, the BiLDA model performs roughly as well as the CCA model when using the appos vocabularies. However, CCA outperforms BiLDA with the POS-vocabulary. Regarding the choice o vocabulary, we see that performance is generally higher using the POS-based vocabulary. This makes sens« yecause the POS-based vocabulary is much larger (over 9,000 unique tokens) than the Zappos-basec around 200 tokens). Having a larger vocabulary is beneficial to the Txt2Img task because it allows for more “xpressiveness in the required visual attributes. Fig. 6 presents qualitative results, where given a textual query, we show the top 4 images retrieved. We see very interesting results. The query ‘little black dress black polyester jersey lace’ actually finds little black dresses. It can be argued that the retrieved items also display some jersey-type characteristics, especially the first and third items. For the attribute polyester, it is not clear from the image what the fabrics of the garments are, as this is not a particularly visual word. Also, we do not see any laces on the dresses, so there is room for improvement. The query ‘wedding gown sleeveless scalloped ruffles’ retrieved wedding gowns in all four items. Two of them are sleeveless and three of them contain ‘scalloped ruffles’. The query ‘casual sleeveless floral print’ retrieves garments with floral patterns on them. They are sleeveless and casual. The query ‘long cocktail wedding gown strapless yellow ruched’ retrieves yellow items in all cases. Three of them are long and ruched. Two of them are strapless. In this case, the query might be slightly misleading, since it mentions wedding gown, and it is often assumed that that corresponds to an actual white wedding dress. However, the word wedding is often used to describe the occasion that the dress may be worn to. It is similar to the word ‘cocktail’ as it also refers to the occasion. The occasion to which a particular garment is appropriate to wear is of course highly subjective. Overall, these results are highly impressive given the difficulty of the task. A demonstration and workshop Regarding the choice of model, the BiLDA model performs roughly as well as the CCA model when using the
Fig. 6. Txt2Img: Example results. Given a textual query, we show the top retrieved images. 7. Conclusions We investigated cross-modal search of fashion items. Given a textual query composed of visual attribute of dresses, our system retrieves relevant images of dresses, and given a picture of a dress as query, thé system describes the attributes of the dress in natural language terms. We implemented and compare¢ several algebraic and probabilistic graphical models to learn latent components that bridge the visual anc textual features. We have experimented with different types of visual and textual features. Our system wa. trained on real Web data found at Amazon.com composed of fashion products and their textual description. and was evaluated on an additional set of Amazon data. Our best approach uses CNN-based visual feature: and a controlled, commonly used fashion vocabulary. It obtained a remarkable performance whel compared to the state-of-the-art setting of [16], which uses SIFT-based features and a vocabulary based o1 part-of-speech. For example, at K=5, the previous setting obtains 24.04% precision and 11.97% recall. Ii contrast our best system obtains 48.75% precision (102% increase) and 43.66% recall (364% increase).W find a similar behaviour for other values of K. Additionally, by visually inspecting the annotations ou system generates, we find reasonable descriptions that capture different garment lengths, colors an¢ textures.
Fig. 6. Txt2Img: Example results. Given a textual query, we show the top retrieved images. 7. Conclusions We investigated cross-modal search of fashion items. Given a textual query composed of visual attribute of dresses, our system retrieves relevant images of dresses, and given a picture of a dress as query, thé system describes the attributes of the dress in natural language terms. We implemented and compare¢ several algebraic and probabilistic graphical models to learn latent components that bridge the visual anc textual features. We have experimented with different types of visual and textual features. Our system wa. trained on real Web data found at Amazon.com composed of fashion products and their textual description. and was evaluated on an additional set of Amazon data. Our best approach uses CNN-based visual feature: and a controlled, commonly used fashion vocabulary. It obtained a remarkable performance whel compared to the state-of-the-art setting of [16], which uses SIFT-based features and a vocabulary based o1 part-of-speech. For example, at K=5, the previous setting obtains 24.04% precision and 11.97% recall. Ii contrast our best system obtains 48.75% precision (102% increase) and 43.66% recall (364% increase).W find a similar behaviour for other values of K. Additionally, by visually inspecting the annotations ou system generates, we find reasonable descriptions that capture different garment lengths, colors an¢ textures.

Related papers

Using Images to Improve Machine-Translating E-Commerce Product Listings
Pintu Lohar

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers

In this paper we study the impact of using images to machine-translate user-generated ecommerce product listings. We study how a multi-modal Neural Machine Translation (NMT) model compares to two text-only approaches: a conventional state-of-the-art attentional NMT and a Statistical Machine Translation (SMT) model. User-generated product listings often do not constitute grammatical or well-formed sentences. More often than not, they consist of the juxtaposition of short phrases or keywords. We train our models end-to-end as well as use text-only and multimodal NMT models for re-ranking n-best lists generated by an SMT model. We qualitatively evaluate our user-generated training data also analyse how adding synthetic data impacts the results. We evaluate our models quantitatively using BLEU and TER and find that (i) additional synthetic data has a general positive impact on text-only and multi-modal NMT models, and that (ii) using a multi-modal NMT model for re-ranking n-best lists improves TER significantly across different nbest list sizes.

View PDFchevron_right
Multimodal deep neural networks for attribute prediction and applications to e-commerce catalogs enhancement
Thales Vieira

Multim. Tools Appl., 2021

Compiling and managing huge e-commerce catalogs is a hard and time-consuming task for a retailer. In particular, deriving standardized and structured descriptions from unstructured data modalities, such as texts and images, is crucial to the performance of search engines and the general organization of virtual store databases. In this paper, we propose methodologies and strategies based on Deep Learning classifiers to structure, update, and inspect large e-commerce catalogs. To this purpose, we exploit multimodal representations combining data from images and unstructured textual descriptions to identify relevant labels for e-commerce applications. Such modalities of data are employed to train deep neural network architectures, which are then able to automatically recognize attributes. Three classes of architecture were investigated: variations of the VGG architecture for recognition from images; architectures combining embedding, convolutional and recurrent layers for text recognit...

View PDFchevron_right
Extending CLIP for Category-to-image Retrieval in E-commerce
Svitlana Vakulenko

ArXiv, 2021

E-commerce provides rich multimodal data that is barely leveraged in practice. One aspect of this data is a category tree that is being used in search and recommendation. However, in practice, during a user’s session there is often a mismatch between a textual and a visual representation of a given category. Motivated by the problem, we introduce the task of category-to-image retrieval in e-commerce and propose a model for the task, CLIP-ITA. The model leverages information from multiple modalities (textual, visual, and attribute modality) to create product representations. We explore how adding information from multiple modalities (textual, visual, and attribute modality) impacts the model’s performance. In particular, we observe that CLIP-ITA significantly outperforms a comparable model that leverages only the visual modality and a comparable model that leverages the visual and attribute modality.

View PDFchevron_right
Powering Robust Fashion Retrieval With Information Rich Feature Embeddings
Kumar Ayush

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Visual content-based product retrieval has become increasingly important for e-commerce. Fashion retrieval, in particular, is a challenging problem owing to a wide range of visual distortions in their product images. In this paper, we propose a Grid Search Network (GSN) for learning feature embeddings for fashion retrieval. The proposed approach posits the training procedure as a search problem, focused on locating matches for a reference query image in a grid containing both positive and negative images w.r.t the query. The proposed framework significantly outperforms existing state-of-the-art methods on benchmark fashion datasets. We also utilize a reinforcement learning based strategy to learn a specialized transformation function which further improves retrieval performance when applied over the feature embeddings. We also extend the reinforcement learning based strategy to learn custom kernel functions for SVM based classification over FashionM-NIST and MNIST datasets, showing improved performance. We highlight the generalization capabilities of this search strategy by showing performance improvement in domains beyond fashion.

View PDFchevron_right
GrokNet: Unified Computer Vision Model Trunk and Embeddings For Commerce
Edward Pizzi

Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020

In this paper, we present GrokNet, a deployed image recognition system for commerce applications. GrokNet leverages a multi-task learning approach to train a single computer vision trunk. We achieve a 2.1x improvement in exact product match accuracy when compared to the previous state-of-the-art Facebook product recognition system. We achieve this by training on 7 datasets across several commerce verticals, using 80 categorical loss functions and 3 embedding losses. We share our experience of combining diverse sources with wide-ranging label semantics and image statistics, including learning from human annotations, user-generated tags, and noisy search engine interaction data. GrokNet has demonstrated gains in production applications and operates at Facebook scale.

View PDFchevron_right
SIR: Similar Image Retrieval for Product Search in E-Commerce
nihar vanjara

2020

We present a similar image retrieval (SIR) platform that is used to quickly discover visually similar products in a catalog of millions. Given the size, diversity, and dynamism of our catalog, product search poses many challenges. It can be addressed by building supervised models to tagging product images with labels representing themes and later retrieving them by labels. This approach suffices for common and perennial themes like &quot;white shirt&quot; or &quot;lifestyle image of TV&quot;. It does not work for new themes such as &quot;e-cigarettes&quot;, hard-to-define ones such as &quot;image with a promotional badge&quot;, or the ones with short relevance span such as &quot;Halloween costumes&quot;. SIR is ideal for such cases because it allows us to search by an example, not a pre-defined theme. We describe the steps - embedding computation, encoding, and indexing - that power the approximate nearest neighbor search back-end. We also highlight two applications of SIR. The firs...

View PDFchevron_right
Discriminative Learning of Apparel Features
Marko Ristin

Fashion is a major segment in e-commerce with growing importance and a steadily increasing number of products. Since manual annotation of apparel items is very tedious, the product databases need to be organized automatically, e.g. by image classification. Common image classification approaches are based on features engineered for general purposes which perform poorly on specific images of apparel. We therefore propose to learn discriminative features based on a small set of annotated images. We experimentally evaluate our method on a dataset with 30,000 images containing apparel items, and compare it to other engineered and learned sets of features. The classification accuracy of our features is significantly superior to designed HOG and SIFT features (43.7% and 16.1% relative improvement, respectively). Our method allows for fast feature extraction and training, is easy to implement and, unlike deep convolutional networks, does not require powerful dedicated hardware. 1 eMarketer 2014 2 Amazon Associates 1. Input Images 2. Feature Learning 3. Categorization

View PDFchevron_right
CONTENT2VEC: Specializing Joint Representations of Product Images and Text for the Task of Product Recommendation
Flavian Vasile

2017

We propose a unified product embedded representation that is optimized for the task of retrieval-based product recommendation. We generate this representation using Content2Vec, a new deep architecture that merges product content information such as text and image, and we analyze its performance on hard recommendation setups such as cold-start and cross-category recommendations. In the case of a normal recommendation regime where collaborative information signal is available, we merge the product co-occurrence information and propose a second architecture Content2vec+ and show its lift in performance versus non-hybrid approaches in both cold start and normal recommendation regimes.

View PDFchevron_right
Discovering Style Trends Through Deep Visually Aware Latent Item Embeddings
Adair Kovac

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018

In this paper, we explore Latent Dirichlet Allocation (LDA) [1] and Polylingual Latent Dirichlet Allocation (PolyLDA) [6], as a means to discover trending styles in Overstock 1 from deep visual semantic features transferred from a pretrained convolutional neural network and textbased item attributes. To utilize deep visual semantic features in conjunction with LDA, we develop a method for creating a bag of words representation of unrolled image vectors. By viewing the channels within the convolutional layers of a Resnet-50 [2] as being representative of a word, we can index these activations to create visual documents. We then train LDA over these documents to discover the latent style in the images. We also incorporate text-based data with PolyLDA, where each representation is viewed as an independent language attempting to describe the same style. The resulting topics are shown to be excellent indicators of visual style across our platform.

View PDFchevron_right
A Multimodal Recommender System for Large-scale Assortment Generation in e-Commerce
Adair Kovac

ArXiv, 2018

E-commerce platforms surface interesting products largely through product recommendations that capture users&#39; styles and aesthetic preferences. Curating recommendations as a complete complementary set, or assortment, is critical for a successful e-commerce experience, especially for product categories such as furniture, where items are selected together with the overall theme, style or ambiance of a space in mind. In this paper, we propose two visually-aware recommender systems that can automatically curate an assortment of living room furniture around a couple of pre-selected seed pieces for the room. The first system aims to maximize the visual-based style compatibility of the entire selection by making use of transfer learning and topic modeling. The second system extends the first by incorporating text data and applying polylingual topic modeling to infer style over both modalities. We review the production pipeline for surfacing these visually-aware recommender systems and ...

View PDFchevron_right

References (33)

  1. Chen, H., Gallagher, A., & Girod, B. (2012). Describing clothing by semantic attributes. Proceedings of the 12th European Conference on Computer Vision. Berlin, Heidelberg: Springer-Verlag.
  2. Yamaguchi, K., Kiapour, M. H., & Berg, T. L. (2013). Paper doll parsing: Retrieving similar styles to parse clothing items. Proceedings of the IEEE International Conference on Computer Vision.
  3. Bossard, L., Dantone, M., Leistner, C., Wengert, C., Quack, T., & Van Gool, L. (2013). Apparel classification with style. Proceedings of the 11th Asian Conference on Computer Vision (pp. 321-335). Springer-Verlag.
  4. Mason, R., & Charniak, E. (2014). Domain-specific image captioning. Proceedings of the Eighteenth Conference on Computational Natural Language Learning (pp. 11-20). Ann Arbor, Michigan: ACL.
  5. Choi, T-M., Hui, C-L., & Yu, Y. (2013). Intelligent Fashion Forecasting Systems: Models and Applications. Springer Publishing Company, Incorporated.
  6. Chen, Q., Wang, G., & Tan, C. L. (2013). Modeling fashion. Proceedings of IEEE International Conference on Multimedia and Expo.
  7. Mori, Y., Takahashi, H., & Oka, R. (1999). Image-to-word transformation based on dividing and vector quantizing images with words. Proceedings of First International Workshop on Multimedia Intelligent Storage and Retrieval Management.
  8. Hodosh, M., Young, P., & Hockenmaier, J. (2013). Framing image description as a ranking task: Data, models and evaluation metrics. J. Artif Intell Res.
  9. Socher, R., Karpathy, A., Le, Q. V., et al. (2014). Grounded Compositional Semantics for Finding and Describing Images with Sentences. TACL.
  10. Karpathy, A., & Fei-Fei, L. (2015). Deep visual-semantic alignments for generating image descriptions. Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition (pp. 3128-3137).
  11. Vulić, I., Zoghbi, S., & Moens, M.-F. (2014). Learning to bridge colloquial and formal language applied to linking and search of e-commerce data. Proceedings of the 37th International ACM SIGIR Conference on Research; Development in Information Retrieval (pp. 1195-1198). New York, NY, USA: ACM.
  12. Yu, J., Mohan, S., Putthividhya, D., & Wong, W.-K. (2014). Latent dirichlet allocation based diversified retrieval for e-commerce search. Proceedings of the 7th ACM International Conference on Web Search and Data Mining (pp. 463-472). New York, NY, USA: ACM.
  13. Lin K., Yang H-F., Liu K-H., Hsiao J-H., & Chen C-S. (2015). Rapid Clothing Retrieval via Deep Learning of Binary Codes and Hierarchical Search. Proceedings of the 5th ACM on International Conference on Multimedia Retrieval (pp. 499-502). New York, NY, USA: ACM.
  14. Lin, T-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., et al. (2014). Microsoft (COCO) common objects in context. Proceedings of European Conference on Computer Vision (ECCV).
  15. Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014). From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Trans Assoc Comput Linguist, 2, 67-78.
  16. Mason, R., & Charniak, E. (2013). Annotation of online shopping images without labeled training examples. North American Chapter of the ACL Human Language Technologies.
  17. Schmid, H. (1994). Probabilistic part-of-speech tagging using decision trees. Proceedings of the International Conference on New Methods in Language Processing (pp. 44-49).
  18. Schmid, H. (1995). Improvements in part-of-speech tagging with an application to German. Proceedings of the ACL SIGDAT-Workshop (pp. 47-50).
  19. Lowe, D. G. (Nov. 2004). Distinctive image features from scale-invariant keypoints. Int. J. Comput Vis. 60(2), 91-110.
  20. Vedaldi, A., & Fulkerson, B. (2010). VLFeat -An open and portable library of computer vision Volume 8, Number 1, February 2016 algorithms. Proceedings of ACM International Conference on Multimedia (pp. 1469-1472).
  21. Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE Conference (pp. 2278-2324).
  22. Deng, J., Dong, W., Socher, R., Li, L-J., Li, K., & Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. CVPR09.
  23. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, & K. Q. Weinberger (Eds.), Advances in Neural Information Processing Systems (pp. 1097-1105). Curran Associates, Inc..
  24. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv abs/1409.1556.
  25. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R. B., et al. (2014). Caffe: Convolutional architecture for fast feature embedding. Proceedings of the 22Nd ACM International Conference on Multimedia (pp. 675-678).
  26. Rasiwasia, N., Pereira, J. C., Coviello, E., Doyle, G., Lanckriet, G. R. G., Levy, R., et al. (2010). A new approach to cross-modal multimedia retrieval. Proceedings of the 18th International ACM Conference on Multimedia (pp. 251-260).
  27. Hardoon, D. R., Szedmák, S., & Shawe-Taylor, J. (2004). Canonical correlation analysis: An overview with application to learning methods. Neural Comput., 16(12), 2639-2664.
  28. Faruqui, M., & Dyer, C. (2014). Improving vector space word representations using multilingual correlation. Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (pp. 462-471).
  29. De Smet, W., & Moens, M.-F. (2009). Cross-language linking of news stories on the Web using interlingual topic modeling. Proceedings of the CIKM 2009 Workshop on Social Web Search and Mining.
  30. De Smet, W., Tang, J., & Moens, M.-F. (2011). Knowledge transfer across multilingual corpora via latent topics. Proceedings of the 15th Pacific-Asia Conference on Knowledge Discovery and Data Mining (pp. 549-560).
  31. Zoghbi, S., Heyman, G., Carranza, J. C. G., & Moens, M.-F. (2015). Cross-modal fashion search. Proceedings of the 22nd International Conference on Multimedia Modelling.
  32. Zoghbi, S., Heyman, G., Carranza, J. C. G., & Moens, M-F. (2015). Cross-modal attribute recognition in fashion. Proceedings of NIPS Multimodal Machine Learning Workshop.
  33. Susana Zoghbi is a PhD student in computer science at the KU Leuven. She obtained a masters degree from the University of British Columbia in 2011. Her research interests lie at the boundary of computer vision and natural language processing, and include deep learning, topic modeling and graphical models. Geert Heyman is a doctoral researcher in the Department of Computer Science, KU Leuven, Belgium. He completed his undergraduate studies and his master thesis at the Faculty of Engineering Science at KU Leuven in July 2014. His research interests are statistical models (such as neural networks and graphical models) for natural language processing, in particular for language modeling and machine translation.

Related papers

Cross-Modal Fashion Search
Geert Heyman, Susana Zoghbi, Juan Carlos Gomez

In this demo we focus on cross-modal (visual and textual) e-commerce search within the fashion domain. Particularly, we demonstrate two tasks: 1) given a query image (without any accompanying text), we retrieve textual descriptions that correspond to the visual attributes in the visual query; and 2) given a textual query that may express an interest in specific visual characteristics, we retrieve relevant images (without leveraging textual meta-data) that exhibit the required visual attributes. The first task is especially useful to manage image collections by online stores who might want to automatically organize and mine predominantly visual items according to their attributes without human input. The second task renders useful for users to find items with specific visual characteristics, in the case where there is no text available describing the target image. We use a state-of-the-art visual and tex-tual features, as well as a state-of-the-art latent variable model to bridge between textual and visual data: bilingual latent Dirichlet allocation. Unlike traditional search engines, we demonstrate a truly cross-modal system, where we can directly bridge between visual and textual content without relying on pre-annotated meta-data.

View PDFchevron_right
Cross-Modal Attribute Recognition in Fashion
Juan Carlos Gomez

In this paper we focus on cross-modal (visual and textual) attribute recognition within the fashion domain. Particularly, we investigate two tasks: 1) given a query image, we retrieve textual descriptions that correspond to the visual attributes in the query; and 2) given a textual query that may express visual characteristics, we retrieve relevant images that exhibit the required visual attributes. To this end, we collected a dataset that consists of 53,689 images coupled with textual descriptions in natural language. The images contain fashion garments that display a great variety of visual attributes. Examples of visual attributes in fashion include colors (e., the text provides a rough and noisy description of the item in the image. We extensively analyze this dataset in the context of cross-modal attribute recognition. We investigate two latent variable models to bridge between textual and visual data: bilingual latent Dirichlet allocation and canonical correlation analysis. We use visual and textual features and report promising results 1 .

View PDFchevron_right
DeepStyle: Multimodal Search Engine for Fashion and Interior Design
Aleksander Skorupa

In this paper, we propose a multimodal search engine that combines visual and textual cues to retrieve items from a multimedia database aesthetically similar to the query. The goal of our engine is to enable intuitive retrieval of fashion merchandise such as clothes or furniture. Existing search engines treat textual input only as an additional source of information about the query image and do not correspond to the real-life scenario where the user looks for "the same shirt but of denim". Our novel method, dubbed DeepStyle, mitigates those shortcomings by using a joint neural network architecture to model contextual dependencies between features of different modalities. We prove the robustness of this approach on two different challenging datasets of fashion items and furniture where our DeepStyle engine outperforms baseline methods by more than 20% on tested datasets. Our search engine is commercially deployed and available through a Web-based application.

View PDFchevron_right
AdsCVLR: Commercial Visual-Linguistic Representation Modeling in Sponsored Search
Si Ying Li, Kay

Proceedings of the 30th ACM International Conference on Multimedia

Sponsored search advertisements (ads) appear next to search results when consumers look for products and services on search engines. As the fundamental basis of search ads, relevance modeling has attracted increasing attention due to the significant research challenges and tremendous practical value. In this paper, we address the problem of multi-modal modeling in sponsored search, which models the relevance between user query and commercial ads with multi-modal structured information. To solve this problem, we propose a transformer architecture with Ads data on Commercial Visual-Linguistic Representation (AdsCVLR) with contrastive learning that naturally extends the transformer encoder with the complementary multi-modal inputs, serving as a strong aggregator of image-text features. We also make a public advertising dataset, which includes 480K labeled query-ad pairwise data with structured information of image, title, seller, description, and so on. Empirically, we evaluate the AdsCVLR model over the large industry dataset, and the experimental results of online/offline tests show the superiority of our method.

View PDFchevron_right
CurlingNet: Compositional Learning between Images and Text for Fashion IQ Data
YoungJae Yu

ArXiv, 2020

We present an approach named CurlingNet that can measure the semantic distance of composition of image-text embedding. In order to learn an effective image-text composition for the data in the fashion domain, our model proposes two key components as follows. First, the Delivery makes the transition of a source image in an embedding space. Second, the Sweeping emphasizes query-related components of fashion images in the embedding space. We utilize a channel-wise gating mechanism to make it possible. Our single model outperforms previous state-of-the-art image-text composition models including TIRG and FiLM. We participate in the first fashion-IQ challenge in ICCV 2019, for which ensemble of our model achieves one of the best performances.

View PDFchevron_right

Related topics

  • Information Retrieval
  • Image Processing
  • Natural Language Processing
  • Machine Learning
  • Content-Based Information Retrie...
  • Object Recognition (Pattern Reco...
  • Content-Based Image Retrieval
  • Topic Models
  • Multimedia information retrieval
  • Content based image retrieval
  • Image Retrieval
  • Machine Learning and Pattern Rec...
  • Latent Dirichlet Allocation
  • Natural Language Processing(NLP)
  • E-Commerce
  • Content based Retrieval
  • Multi Modality
  • Electronic Commerce and E ­‐busi...
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved