Academia.eduAcademia.edu

Outline

Title
Abstract
FAQs
Figures
Introduction
Background
Related Work
Definition of Research Question
Results and Discussion
Conclusions
References
All Topics
Computer Science
Human-Computer Interaction

Data Mining applied on Web Robots Detection: A Systematic Mapping

Profile image of Ramon S AbilioRamon S Abilio

2021, Anais do 15. Congresso Brasileiro de Inteligência Computacional

https://doi.org/10.21528/CBIC2021-60
visibility

…

description

8 pages

link

1 file

Abstract

Browsing on Internet is part of the world population’s daily routine. The number of web pages is increasing and so is the amount of published content (news, tutorials, images, videos) provided by them. Search engines use web robots to index web contents and to offer better results to their users. However, web robots have also been used for exploiting vulnerabilities in web pages. Thus, monitoring and detecting web robots’ accesses is important in order to keep the web server as safe as possible. Data Mining methods have been applied to web server logs (used as data source) in order to detect web robots. Then, the main objective of this work was to observe evidences of definition or use of web robots detection by analyzing web server-side logs using Data Mining methods. Thus, we conducted a systematic Literature mapping, analyzing papers published between 2013 and 2020. In the systematic mapping, we analyzed 34 studies and they allowed us to better understand the area of web robots d...

FAQs

sparkles

AI

What are the most commonly used algorithms for detecting web robots?add

The study identifies 33 machine learning algorithms, with SVM featured in 9 papers and Decision Trees in 8.

How does feature selection impact web robot detection performance?add

Different studies selected varied feature sets ranging from 2 to 50, affecting detection accuracy and efficacy.

What methodologies were employed in the systematic mapping of relevant studies?add

The systematic mapping included selecting studies based on inclusion/exclusion criteria, retrieving 34 relevant papers from 2150.

What challenges exist in labeling sessions for web robot identification?add

Manual labeling is impractical due to data volume, and heuristics may fail to detect all robotic behavior.

When did the significance of detecting web robots gain research attention?add

Research focused on detecting web robots increased significantly post-2013, as their presence escalated.

Figures (4)
Fig. 1. Number of Publications per Publication Venue
Fig. 1. Number of Publications per Publication Venue
TABLE IV TABLE V MACHINE LEARNING ALGORITHMS EXTRACTED FROM THE SELECTED WORKS AND THEIR RESPECTIVE FREQUENCIES (10 MOST FREQUENT) FEATURES EXTRACTED FROM THE SELECTED WORKS AND THEIR RESPECTIVE FREQUENCIES (10 MOST FREQUENT)
TABLE IV TABLE V MACHINE LEARNING ALGORITHMS EXTRACTED FROM THE SELECTED WORKS AND THEIR RESPECTIVE FREQUENCIES (10 MOST FREQUENT) FEATURES EXTRACTED FROM THE SELECTED WORKS AND THEIR RESPECTIVE FREQUENCIES (10 MOST FREQUENT)
A study published in 2015 [27] developed a system to collect data from the client-side and server-side. The authors used, for example, hidden links and a web form in that system to identify web robots with other 6 features such as: number of requests in the session, a boolean feature that indicates if all the requests have empty referrer, and a boolean feature that indicates changes from one session to another subsequent session. After the feature selection, 5 features were selected: a boolean feature that indicates if the client has filled or not the
A study published in 2015 [27] developed a system to collect data from the client-side and server-side. The authors used, for example, hidden links and a web form in that system to identify web robots with other 6 features such as: number of requests in the session, a boolean feature that indicates if all the requests have empty referrer, and a boolean feature that indicates changes from one session to another subsequent session. After the feature selection, 5 features were selected: a boolean feature that indicates if the client has filled or not the

Related papers

A density based clustering approach for web robot detection
Javad Hamidzadeh

2014 4th International Conference on Computer and Knowledge Engineering (ICCKE), 2014

Distinction between humans and web robots, in terms of computer network security, has led to the robot detection problem. An exact solution for this issue can preserve web sites from the intrusion of malicious robots and increase the performance of web servers by prioritizing human users. In this article, we propose a density based method called DBC_WRD (Density Based Clustering for Web Robot Detection) to discover the traffic of web robots on two large real data sets. So, we assume the visitors as the spatial instances and introduce two new features to describe and distinguish them. These attributes are based on the behavioral patterns of web visitors and remain invariant over time. By focusing on one of the disadvantages of DBSCAN as the density based clustering algorithm used in this paper, we just utilize 4 features to reduce the dimensions. According to the supervised evaluations, DBC_WRD can have the 96% of Jaccard metric and produce two clusters which have the entropy and purity rates of 0.0215 and 0.97, respectively. Furthermore, the comparisons show that from the standpoint of clustering quality and accuracy, DBC_WRD performs better than state-of-the-art algorithms. Finally, it can be concluded that some non-malicious popular web robots, through imitating the human's behavior, make it difficult to be identified.

View PDFchevron_right
Content-aware web robot detection
Athanasios Lagopoulos

Applied Intelligence, 2020

Web crawlers account for more than a third of the total web traffic and they are threatening the security, privacy and veracity of web applications and their users. Businesses in finance, ticketing and publishing, as well as websites with rich and unique content, are the ones mostly affected by their actions. To deal with this problem, we present a novel web robot detection approach that takes advantage of the content of a website based on the assumption that human web users are interested in specific topics, while web robots crawl the web randomly. Our approach extends the typical user session representation of log-based features with a novel set of features that capture the semantics of the content of the requested resources. In addition, we contribute a new real-world dataset, which we make publicly available, towards alleviating the scarcity of open data in this field. Empirical results on this dataset validate our assumption and show that our approach outranks state-of-the-art methods for web robot detection.

View PDFchevron_right
Web Robot Detection in Academic Publishing
Athanasios Lagopoulos

ArXiv, 2017

Recent industry reports assure the rise of web robots which comprise more than half of the total web traffic. They not only threaten the security, privacy and efficiency of the web but they also distort analytics and metrics, doubting the veracity of the information being promoted. In the academic publishing domain, this can cause articles to be faulty presented as prominent and influential. In this paper, we present our approach on detecting web robots in academic publishing websites. We use different supervised learning algorithms with a variety of characteristics deriving from both the log files of the server and the content served by the website. Our approach relies on the assumption that human users will be interested in specific domains or articles, while web robots crawl a web library incoherently. We experiment with features adopted in previous studies with the addition of novel semantic characteristics which derive after performing a semantic analysis using the Latent Diric...

View PDFchevron_right
A Density Based Clustering Approach to Distinguish Between Web Robot and Human Requests to a Web Server
Javad Hamidzadeh

Today world's dependence on the Internet and the emerging of Web 2.0 applications significantly increased the requirement of web robots crawling the sites to support services and technologies. Regardless of the advantages of robots, they may occupy the bandwidth and reduce the performance of web servers. Despite a variety of researches, there is no accurate method for classifying huge data sets of web visitors in a reasonable amount of time. Moreover, this technique should be insensitive to the ordering of instances and produce deterministic accurate results. Therefore, this paper presents a density-based clustering approach using Density-Based Spatial Clustering of Applications with Noises (DBSCAN), to classify web visitors of two real large data sets. We propose two new features based on the behavioral patterns of visitors to describe them. What's more, we consider 12 common features and use the significance of the difference test (T-test) to reduce the dimensions and over...

View PDFchevron_right
Detection and Confirmation of Web Robot Requests for Cleaning the Voluminous Web Log Data
Tanvir Sardar
View PDFchevron_right
A Survey of HTTP BotNet Identification Using Data Mining Technique
Dr. Anilkumar Suthar

GEDRAG & ORGANISATIE REVIEW, 2021

From all the different malware attack the botnet is extensive and serious threat which occurs over all the time in today's Cyber attack. Botnet is a group of accord computer which are fortuitously controlled by the hacker to launch various type of attack like DDOS, Frauds like click fraud , Phishing attack. Botnet has become one of the popular and fecund tool behind many attacks the main characteristic of the botnet is there command and control channel by which the re-quip and handled. In recent trends botnet expands into HTTP botnet In this research we are proposed to Data Mining techniques allow us robotize detecting similarity from the huge amount of data which can be conventional heuristic and signature based method can not be apply.

View PDFchevron_right
Detection and confirmation of web robot requests
Tanvir Sardar, Zahid A Ansari
View PDFchevron_right
Detection of Advanced Web Bots by Combining Web Logs with Mouse Behavioural Biometrics
christos iliou

Digital Threats: Research and Practice

Web bots vary in sophistication based on their purpose, ranging from simple automated scripts to advanced web bots that have a browser fingerprint, support the main browser functionalities, and exhibit a humanlike behaviour. Advanced web bots are especially appealing to malicious web bot creators, due to their browserlike fingerprint and humanlike behaviour that reduce their detectability. This work proposes a web bot detection framework that comprises two detection modules: (i) a detection module that utilises web logs, and (ii) a detection module that leverages mouse movements. The framework combines the results of each module in a novel way to capture the different temporal characteristics of the web logs and the mouse movements, as well as the spatial characteristics of the mouse movements. We assess its effectiveness on web bots of two levels of evasiveness: (a) moderate web bots that have a browser fingerprint and (b) advanced web bots that have a browser fingerprint and also ...

View PDFchevron_right
Characterizing Crawler Behavior from Web Server Access Logs
Loizos Papageorgiou

2003

In this paper, we present a study of crawler behavior based on Web-server access logs. To this end, we use logs from five different academic sites in three countries. Based on these logs, we analyze the activity of different crawlers that belong to five Search Engines: Google, AltaVista, Inktomi, FastSearch and CiteSeer. We compare crawler behavior to the characteristics of the general World-Wide Web traffic, and to general characterization studies based on Web-server access logs. We analyze crawler requests to derive insights into the behavior and strategy of crawlers. Our results and observations provide useful insights into crawler behavior and serve as basis of our ongoing work on the automatic detection of WWW robots.

View PDFchevron_right
Fuzzy Inference for Intrusion Detection of Web Robots in Computer Networks
Javad Hamidzadeh

Distinction between humans and web robots, in terms of computer network security, has led to the robot detection problem. An exact solution for this issue can preserve web sites from the intrusion of malicious robots and increase the performance of web servers by prioritizing human users. In this article, we propose a novel method called FID (Fuzzy Intrusion Detection), for accurate intrusion detection of web robots, in computer networks. The proposed method relies on fuzzy inference systems and uses a decision tree to fuzzify the features which are used to describe web clients. The aim of FID is to overcome the curse of dimensionality, reduce the number of fuzzy rules and subsequently, facilitate the designing of fuzzy inference system. Our experiments show that the proposed algorithm has 0.13 false alarm ratios, in distinguishing humans from robots. Experimental results, on a real data set, demonstrate the effectiveness of our approach when compared to some of the state-of-the-art...

View PDFchevron_right

References (74)

  1. Identification and characterization of crawlers through analysis of web logs 2013 [17] Access patterns for robots and humans in web archives 2013 [18] Detecting Impolite Crawler by Using Time Series Analysis 2013
  2. A comparison of web robot and human requests 2013 [20] Detecting anomalous Web server usage through mining access logs 2013
  3. Mining web logs to identify search engine behaviour at websites 2013
  4. An integrated approach to defence against degrading application-layer DDoS attacks 2013
  5. Detection and confirmation of web robot requests for cleaning the voluminous web log data 2014 [24] Analysis of Aggregated Bot and Human Traffic on E-Commerce Site 2014
  6. A Supplementary Method for Malicious Detection Based on HTTP-Activity Similarity Features 2014
  7. A density based clustering approach to distinguish between web robot and human requests to a web server 2014 [27] Lino -An Intelligent System for Detecting Malicious Web-Robots 2015
  8. Optimized Distributed Association Mining (ODAM) Algorithm for detecting Web Robots 2015
  9. A Comparative Analysis of Browsing Behavior of Human Visitors and Automatic Software Agents 2015
  10. Agglomerative approach for identification and elimination of web robots from web server logs to extract knowledge about actual visitors 2015
  11. HTTP-sCAN: Detecting HTTP-flooding attack by modeling multi-features of web browsing behavior from noisy web-logs 2015
  12. An integrated method for real time and offline web robot detection 2016
  13. HTTP Flooding Attack Detection by Modeling Features of Web Browsing behavior from Web Log 2016 [34] Website Navigation Behavior Analysis for Bot Detection 2017
  14. A study of different web-crawler behaviour 2017
  15. Analysis of Robot Detection Approaches for Ethical and Unethical Robots on Web Server Log 2017
  16. A soft computing approach for benign and malicious web robot detection 2017
  17. Bot or Not? A Case Study on Bot Recognition from Web Session Logs 2018
  18. User behavior analytics-based classification of application layer HTTP-GET flood attacks 2018 [40] Performance Evaluation of Large Data Clustering Techniques on Web Robot Session Data 2018 [41] Categorization Performance of Unsupervised Learning Techniques for Web Robots Sessions 2018 [42] Performance Evaluation of Density-Based Clustering Methods for Categorizing Web Robot Sessions 2018
  19. A System Framework for Efficiently Recognizing Web Crawlers 2018
  20. Towards a framework for detecting advanced Web bots 2019
  21. A Hybrid Approach for Recognizing Web Crawlers 2019 [14] An Overview of Web Robots Detection Techniques 2020 [46] Determination of User Navigational Patterns from Server Log Files using Hadoop Techniques 2020
  22. Bot recognition in a web store: an approach based on unsupervised learning 2020
  23. Identifying legitimate Web users and bots with different traffic profiles-an Information Bottleneck approach 2020 REFERENCES
  24. N. Kandpal, R. Sinha, and M. Shekhawat, "A survey on web usage mining: Process, application and tools," Suresh Gyan Vihar University Journal of Engineering & Technology, vol. 3, no. 1, pp. 19-25, 2017.
  25. C. Bomhardt, W. Gaul, and L. Schmidt-Thieme, "Web robot detection -preprocessing web logfiles for robot detection," in New Developments in Classification and Data Analysis, H. Bock and et al, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005, pp. 113-124.
  26. D. Doran, "Detection, classification, and workload analysis of web robots," Ph.D. dissertation, University of Connecticut, Storrs, CT, 2014, accessed: 2021-06-16. [Online]. Available: https://opencommons.uconn.edu/dissertations/348
  27. F. Menczer, G. Pant, P. Srinivasan, and M. E. Ruiz, "Evaluating topic- driven web crawlers," in Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, 2001, pp. 241-249.
  28. Y. Sun, I. Councill, and C. Giles, "The ethicality of web crawlers," in Proceedings of International Conference on Web Intelligence and Intelligent Agent Technology (IEEE/WIC/ACM), 2010, pp. 668-675.
  29. G. Chang, M. Healey, J. McHugh, and J. Wang, Mining the World Wide Web. The Information Retrieval Series. Boston: Springer, 2001.
  30. U. Fayyad, G. Piatetsky-Shapiro, and P. Smyth, "From data mining to knowledge discovery in databases," AI magazine, vol. 17, no. 3, p. 37, 1996.
  31. J. Srivastava, R. Cooley, M. Deshpande, and P. Tan, "Web usage mining: discovery and applications of usage patterns from web data," ACM SIGKDD Explorations Newsletter, vol. 1, no. 2, pp. 12-23, 2000.
  32. M. Srivastava, A. Srivastava, and R. Garg, "Data preprocessing tech- niques in web usage mining: A literature review," in Proceedings of International Conference on Sustainable Computing in Science, Tech- nology and Management (SUSCOM), 2019.
  33. Apache Software Foundation. Apache HTTP server version 2.4 -log files. Accessed: 2021-06-23. [Online]. Available: https://httpd.apache.org/docs/2.4/logs.html
  34. M. Srivastava, R. Garg, and P. Mishra, "Preprocessing techniques in web usage mining: A survey," International Journal of Computer Applications, vol. 97, no. 18, pp. 1-9, 2014.
  35. R. Rao and J. Arora, "A survey on methods used in web usage mining," International Research Journal of Engineering and Technology IRJET, vol. 4, no. 5, pp. 2627-2631, 2017.
  36. M. Mughal, "Data mining: Web data mining techniques, tools and algorithms: An overview," International Journal of Advanced Computer Science and Applications, vol. 9, no. 6, pp. 208-2015, 2018.
  37. H. Chen, H. He, and A. Starr, "An overview of web robots detection techniques," in Proceedings of the International Conference on Cyber Security and Protection of Digital Services (Cyber Security 2020), jun 2020, pp. 1-6.
  38. K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson, "Systematic map- ping studies in software engineering," in 12th International Conference on Evaluation and Assessment in Software Engineering (EASE) 12, 2008, pp. 1-10.
  39. N. Algiriyage, S. Jayasena, G. Dias, A. Perera, and K. Dayananda, "Identification and characterization of crawlers through analysis of web logs," in Proceedings of the 8th International Conference on Industrial and Information Systems, dec 2013, pp. 150-155.
  40. Y. A. AlNoamany, M. C. Weigle, and M. L. Nelson, "Access patterns for robots and humans in web archives," in Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries -JCDL '13, 2013.
  41. Z. Chen and W. Feng, "Detecting impolite crawler by using time series analysis," in Proceedings of the 25th International Conference on Tools with Artificial Intelligence, nov 2013.
  42. D. Doran, K. Morillo, and S. S. Gokhale, "A comparison of web robot and human requests," in Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, aug 2013.
  43. T. Gržinic, T. Kišasondi, and J. Šaban, "Detecting anomalous web server usage through mining access logs," Central European Conference on Information and Intelligent Systems, 2013.
  44. J. Jose and P. S. Lal, "Mining web logs to identify search engine behaviour at websites," Informatica, vol. 37, no. 2013, pp. 381-386, 2013.
  45. D. Stevanovic and N. Vlajic, "An integrated approach to defence against degrading application-layer ddos attacks," in Proceedings of the 12th International Conference on Security and Management, 2013.
  46. T. H. Sardar and Z. Ansari, "Detection and confirmation of web robot requests for cleaning the voluminous web log data," in Proceedings of the International Conference on the IMpact of E-Technology on US (IMPETUS 2014), jan 2014.
  47. G. Suchacka, "Analysis of aggregated bot and human traffic on e- commerce site," in Proceedings of the 2014 Federated Conference on Computer Science and Information Systems, sep 2014.
  48. M. Tran and Y. Nakamura, "A supplementary method for malicious detection based on http-activity similarity features," Journal of Commu- nications, vol. 9, no. 12, 2014.
  49. M. Zabihi, M. V. Jahan, and J. Hamidzadeh, "A density based clustering approach to distinguish between web robot and human requests to a web server," The ISC International Journal of Information Security (ISeCure), vol. 6, no. 1, pp. 1-13, 2014.
  50. T. Gržinić, L. Mršić, and J. Šaban, "Lino -an intelligent system for detecting malicious web-robots," in Asian Conference on Intelligent Information and Database Systems. Springer International Publishing, 2015, pp. 559-568.
  51. A. D. Jagtap and V. Kadroli, "Optimized distributed association mining (odam) algorithm for detecting web robots," International Journal of Engineering and Computer Science (IJECS), vol. 4, no. 7, pp. 13 196- 13 200, 2015.
  52. D. Sisodia, S. Verma, and O. Vyas, "A comparative analysis of browsing behavior ofhuman visitors and automatic software agents," American Journal of Systems and Software, 2015.
  53. D. S. Sisodia, S. Verma, and O. P. Vyas, "Agglomerative approach for identification and elimination of web robots from web server logs to extract knowledge about actual visitors," Journal of Data Analysis and Information Processing, vol. 03, no. 01, pp. 1-10, 2015.
  54. J. Wang, M. Zhang, X. Yang, K. Long, and J. Xu, "HTTP-sCAN: Detect- ing HTTP-flooding attack by modeling multi-features of web browsing behavior from noisy web-logs," China Communications, 12, no. 2, pp. 118-128, feb 2015.
  55. D. Doran and S. S. Gokhale, "An integrated method for real time and offline web robot detection," Expert Systems, vol. 33, no. 6, pp. 592-606, nov 2016.
  56. A. Verma and D. Xaxa, "Http flooding attack detection by modeling features of web browsing behavior from web log," International Journal of Innovations & Advancement in Computer Science (IJIACS), vol. 5, no. 6, pp. 154-159, 2016.
  57. R. Haidar and S. Elbassuoni, "Website navigation behavior analysis for bot detection," in Proceedings of the International Conference on Data Science and Advanced Analytics (DSAA 2017), oct 2017, pp. 60-68.
  58. A. Menshchikov, A. Komarova, Y. Gatchin, A. Korobeynikov, and N. Tishukova, "A study of different web-crawler behaviour," in Proceed- ings of the 20th Conference of Open Innovations Association (FRUCT), apr 2017, pp. 268-274.
  59. M. Srivastava, A. kumar Srivastava, R. Garg, and P. K. Mishra, "Analysis of robot detection approaches for ethical and unethical robots on web server log," International Journal of Advanced Research in Computer Science (IJARCS), vol. 8, no. 5, pp. 1132-1134, May 2017.
  60. M. Zabihimayvan, R. Sadeghi, H. N. Rude, and D. Doran, "A soft computing approach for benign and malicious web robot detection," Expert Systems with Applications, vol. 87, pp. 129-140, nov 2017.
  61. S. Rovetta, A. Cabri, F. Masulli, and G. Suchacka, "Bot or not? a case study on bot recognition from web session logs," Quantifying and Processing Biomedical and Behavioral Signals, pp. 197-206, aug 2018.
  62. K. Singh, P. Singh, and K. Kumar, "User behavior analytics-based classification of application layer HTTP-GET flood attacks," Journal of Network and Computer Applications, vol. 112, pp. 97-114, jun 2018.
  63. D. S. Sisodia, R. Borkar, and H. Shrawgi, "Performance evaluation of large data clustering techniques on web robot session data," in Machine Intelligence and Signal Analysis. Springer, aug 2018, vol. 748, pp. 545-553.
  64. D. S. Sisodia, R. Khandelwal, and A. Anuragi, "Categorization perfor- mance of unsupervised learning techniques for web robots sessions," in Proceedings of the International Conference on Inventive Research in Computing Applications (ICIRCA 2018), jul 2018, pp. 1370-1374.
  65. D. S. Sisodia and N. Verma, "Performance evaluation of density-based clustering methods for categorizing web robot sessions," in Proceedings of the 2018 International Conference on Advanced Computation and Telecommunication (ICACAT), dec 2018, pp. 1-5.
  66. W. Zhu, J. Qin, R. Kong, H. Lin, and Z. He, "A system framework for efficiently recognizing web crawlers," in Proceedings of the 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innova- tion (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), oct 2018, pp. 1130-1133.
  67. C. Iliou, T. Kostoulas, T. Tsikrika, V. Katos, S. Vrochidis, and Y. Kom- patsiaris, "Towards a framework for detecting advanced web bots," in Proceedings of the 14th International Conference on Availability, Reliability and Security (ARES'19), aug 2019, pp. 1-10.
  68. W. Zhu, H. Gao, Z. He, J. Qin, and B. Han, "A hybrid approach for recognizing web crawlers," in Proceedings of the 14th International Conference on Wireless Algorithms, Systems, and Applications (WASA 2019), 2019, pp. 507-519.
  69. R. Patil and P. Trivedi, "Determination of user navigational patterns from server log files using hadoop techniques," International Journal for Research in Applied Science & Engineering Technology (IJRASET), vol. 8, no. 6, pp. 1864-1870, jun 2020.
  70. S. Rovetta, G. Suchacka, and F. Masulli, "Bot recognition in a web store: an approach based on unsupervised learning," Journal of Network and Computer Applications, vol. 157, pp. 1-15, may 2020.
  71. G. Suchacka and J. Iwanski, "Identifying legitimate web users and bots with different traffic profiles-an information bottleneck approach," Knowledge-Based Systems, vol. 197, pp. 1-18, jun 2020.
  72. J. P. Barddal, H. M. Gomes, and F. Enembreck, "Sfnclassifier: A scale- free social network method to handle concept drift," in Proceedings of the 29th Annual ACM Symposium on Applied Computing, 2014, pp. 786-791.
  73. I. Škrjanc, J. A. Iglesias, A. Sanchis, D. Leite, E. Lughofer, and F. Gomide, "Evolving fuzzy and neuro-fuzzy approaches in clustering, regression, identification, and classification: a survey," Information Sci- ences, vol. 490, pp. 344-368, 2019.
  74. C. Garcia, D. Leite, and I. Škrjanc, "Incremental missing-data imputation for evolving fuzzy granular prediction," IEEE Transactions on Fuzzy Systems, vol. 28, no. 10, pp. 2348-2362, 2019.

Related papers

Web Robot Detection - Preprocessing Web Logfiles for Robot Detection
Lars Schmidt-Thieme

Studies in Classification, Data Analysis, and Knowledge Organization, 2000

Web usage mining has to face the problem that parts of the underlying logfiles are created by robots. While cooperative robots identify themselves and obey to the instructions of server owners not to access parts or all of the pages on the server, malignant robots may camouflage themselves and have to be detected by web robot scanning devices. We describe the methodology of robot detection and show that highly accurate tools can be applied to decide whether session data was generated by a robot or a human user.

View PDFchevron_right
Agglomerative Approach for Identification and Elimination of Web Robots from Web Server Logs to Extract Knowledge about Actual Visitors
Shrish Verma

Journal of Data Analysis and Information Processing, 2015

In this paper we investigate the effectiveness of ensemble-based learners for web robot session identification from web server logs. We also perform multi fold robot session labeling to improve the performance of learner. We conduct a comparative study for various ensemble methods (Bagging, Boosting, and Voting) with simple classifiers in perspective of classification. We also evaluate the effectiveness of these classifiers (both ensemble and simple) on five different data sets of varying session length. Presently the results of web server log analyzers are not very much reliable because the input log files are highly inflated by sessions of automated web traverse software's, known as web robots. Presence of web robots access traffic entries in web server log repositories imposes a great challenge to extract any actionable and usable knowledge about browsing behavior of actual visitors. So web robots sessions need accurate and fast detection from web server log repositories to extract knowledge about genuine visitors and to produce correct results of log analyzers.

View PDFchevron_right
A Methodology for Detecting Web Robot Requests from Voluminous Web Log File
Zahid A Ansari

Web robots are software programs that run automated tasks over the internet. They pass through the hyperlink structure of the World Wide Web so that they can retrieve information. There are many reasons to differentiate web robot requests and user requests. Some tasks of web robots can be harmful to the web. Firstly, Web robots are employed to assemble business intelligence at e-commerce sites. In such a state of affairs, the e-commerce site may need to detect robots. Secondly, many e-commerce sites carry out a Web traffic inspection to deduce the way their customers have accessed the site. Unfortunately, such scrutiny can be erroneous by the presence of Web robots. Thirdly, Web robots often consume considerable network bandwidth and server resources at the expense of other users. A web log file is a web server file automatically created and maintained by a web server to check the activity performed by it. It maintains a history of page requests on its site. In this paper, we have used four methods together to detect and finally confirm requests as a robot request. Experiments have been performed on the log file generated from the server of an operational web site named vtulife.com which contains data of march-2013. In our research results of web robot detection using various techniques has been implemented. In this work, the provision of selecting any of the methods, selecting all of the methods and selecting any combination of the desired methods are available. Also, we can compare and integrate different methods. Keywords-web usage mining, web robot detection, web log file.

View PDFchevron_right
Survey on web Robot Request Detection Techniques on Web Server Log
IJRISE Journal

Web robots are software programs which automatically traverse through hyperlink structure of Web to retrieve Web resources. Robots can be used for variety of tasks such as crawling and indexing information for search engines, offline browsing, shopping comparison and email collectors. Apart from that robots can also be used for some malicious purposes like sending spam mails, stealing business intelligence etc. It is necessary to detect robots due to privacy, security and performance of server related issues. Several well-known techniques to detect robots are : robots.txt check, known robot's IP address, User agent mapping, keywords matching in User agent field, browsing speed, unassigned referrer etc. In this paper we have discussed as well as implemented various robot identification techniques on real server log data and compared their performance for a given dataset.

View PDFchevron_right
Feature evaluation for web crawler detection with data mining techniques
Aijun An

Expert Systems with Applications, 2012

Distributed Denial of Service (DDoS) is one of the most damaging attacks on the Internet security today. Recently, malicious web crawlers have been used to execute automated DDoS attacks on web sites across the WWW. In this study we examine the effect of applying seven well-established data mining classification algorithms on static web server access logs in order to: (1) classify user sessions as belonging to either automated web crawlers or human visitors and (2) identify which of the automated web crawlers sessions exhibit 'malicious' behavior and are potentially participants in a DDoS attack. The classification performance is evaluated in terms of classification accuracy, recall, precision and F 1 score. Seven out of nine vector (i.e. web-session) features employed in our work are borrowed from earlier studies on classification of user sessions as belonging to web crawlers. However, we also introduce two novel web-session features: the consecutive sequential request ratio and standard deviation of page request depth. The effectiveness of the new features is evaluated in terms of the information gain and gain ratio metrics. The experimental results demonstrate the potential of the new features to improve the accuracy of data mining classifiers in identifying malicious and well-behaved web crawler sessions.

View PDFchevron_right

Related topics

  • Computer Science
  • Web page design
  • Web Modeling
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved